> ## Content Index
> Fetch the complete content index at: https://www.ratneshmishra.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# 400,000 Columns and Nobody to Describe Them
- URL: https://www.ratneshmishra.com/400-000-columns-and-nobody-to-describe-them/
- Published: 2026-09-16T15:48:52.000Z
- Updated: 2026-09-16T16:42:27.000Z
- Author: Ratnesh Kumar Mishra

### AI can draft the descriptions. A confidence band is what makes them usable.

**The Enterprise Metadata Strategy · Part 2 of 5**

Proposed is not verified. If the band does not change what search and agents do, the score is a label.

Four hundred thousand columns. Four bands: Verified, Reviewed, Suggested, Unverified.

AI can fill the blanks. Only the band decides whether search, agents and people should believe the sentence.

*Last week: metadata should be logically central and physically federated, and a catalog nothing depends on will rot. This week: the coverage problem, what AI actually solves, and why a raw 73% is not a control.*

The rest of this piece is three moves. Build the evidence pack so the draft is grounded. Detect SDE and PII with methods that are allowed to disagree. Put every output in a band that changes system behaviour — or admit the score is decoration.

![](https://storage.ghost.io/c/bf/09/bf092b96-7c12-4ea4-b204-9fb1e04f4342/content/images/2026/09/fig3-confidence-pipeline-2.png)

## The arithmetic that stalls every program

A mid-sized enterprise has somewhere north of 400,000 columns. A steward, working carefully and including review, documents perhaps 200 a day. That is eight person-years of work — for a snapshot that starts decaying the moment it is finished.

Nobody funds eight person-years of documentation. Programs describe the important five percent, declare victory, and freeze below fifteen percent coverage. That is why so many catalogs are simultaneously expensive and empty.

Generative AI changes the economics of the first pass. It changes nothing about who is accountable for the result. The band is what closes that gap.

**A blank field is an honest unknown. A fluent, wrong description is a liability that looks like progress.**

A model that has never seen your business will explain that `acct_status_cd` indicates whether an account is open or closed, when in your company it encodes a collections stage. Nobody catches it, because it reads well. The difference surfaces six weeks later in a board deck.

## Generating business metadata at speed

Quality is decided before the model runs, by the evidence pack.

Teams that get disappointing output almost always sent a column name and a type and hoped. Assemble what the estate already knows:

| Evidence                                         | What it contributes                    |
| ------------------------------------------------ | -------------------------------------- |
| Column name, type, nullability, cardinality      | The skeleton                           |
| Sample values, sensitive content masked          | Format, units, domain                  |
| Table name and description                       | What the row is                        |
| Upstream transformation SQL                      | How the value is derived               |
| Query logs                                       | How people use it, what they filter on |
| Aliases in saved queries                         | The name analysts actually say         |
| dbt docs, Confluence, tickets                    | Prose somebody already wrote           |
| Dashboard and report titles                      | The business question it answers       |
| Lineage parents and their confirmed descriptions | Inherited meaning                      |

![](https://storage.ghost.io/c/bf/09/bf092b96-7c12-4ea4-b204-9fb1e04f4342/content/images/2026/09/eQhcH.jpg)

Your organization has already described this data. It did so in eleven places, none of them the catalog.

The single strongest signal is the transformation SQL. A column called `net_revenue` computed as gross minus returns, with an explicit intercompany exclusion, tells you what it means in a way no naming convention can.

With a full pack, generated descriptions beat an empty field and often beat a rushed human guess. The same pass is good at glossary linkage, lineage hypotheses from un-instrumented SQL, relationship inference, and change summaries people will actually read.

None of it should auto-verify a certified asset.

## Finding SDE and PII

A wrong description misleads someone. A missed classification leaks regulated data. Treat them as different problems.

Three layers, valuable **because they disagree**:

1. **Deterministic validators** — checksums, Luhn, IBAN, well-formed email and phone. When these fire, you are nearly certain.
2. **Statistical profiling** — cardinality near row count, entropy of encoded values, lengths that cluster like postcodes.
3. **Semantic models** — notes containing clinical information, complaints in an innocuous field, account numbers pasted into comments, category values that encode religious dietary preference.

That last category is where most sensitive data hides. Regex was never going to find it.

Three-of-three is as close to certain as classification gets. One-of-three is a question, not an answer.

### Fail closed on sensitivity. Fail open on meaning.

- **Sensitivity** — a false negative leaks; a false positive is friction. Tune toward recall. Let uncertainty trigger *provisional* enforcement: mask a suspected SDE while it awaits review, with a fast clear path and a fourteen-day expiry that escalates to the owner. The friction is the point. It is what gets the review done.
- **Descriptions** — a wrong definition propagates. Favour precision. Leave gaps visible. Publishing nothing is a legitimate outcome.

One threshold for both is malpractice.

![](https://storage.ghost.io/c/bf/09/bf092b96-7c12-4ea4-b204-9fb1e04f4342/content/images/2026/09/fig4-fail-closed-fail-open.png)

## The confidence band

Model self-reported confidence is poorly calibrated. Do not use it as the whole number. Do not put "73%" on the asset. Show a band.

Compose the underlying score from signals outside the generation:

- **Layer agreement** — how many independent methods concur
- **Evidence richness** — a column with no SQL, no queries and no docs should never score highly, however fluent the sentence
- **Estate consistency** — same name elsewhere, same confirmed meaning
- **Lineage support** — parent already human-confirmed
- **Domain track record** — where the model is historically weak

A score is a promise about how often you are right. If 80% does not mean right four times in five, it is decoration. Measure calibration against a gold set. Recalibrate on every model and prompt change.

| Band           | Meaning                               | What the system does                                                   |
| -------------- | ------------------------------------- | ---------------------------------------------------------------------- |
| **Verified**   | Confirmed by the accountable owner    | Auto-publishes. Agents may ground. Ranks first in search               |
| **Reviewed**   | Steward or several frequent users     | Publishes with attribution. Agents may ground                          |
| **Suggested**  | Generated, unreviewed, well-evidenced | Visible badge, one-click confirm. SDE candidates enforce provisionally |
| **Unverified** | Generated, thin evidence              | Stewards only. Never agents. Never search rank                         |

![](https://storage.ghost.io/c/bf/09/bf092b96-7c12-4ea4-b204-9fb1e04f4342/content/images/2026/09/zfHpZ.jpg)

If the band does not change system behaviour, you built a label, not a control.

Confidence should **decay on events**, not on a calendar: upstream schema change, transformation rewrite, sharp shift in value distribution, departure of the person who confirmed it. An asset that drops out of Verified should say so. Otherwise you have rebuilt the catalog that is confidently wrong.

## Why the band changes collaboration

A blank field is a chore. A draft is a two-second decision.

"Write a definition" costs fifteen minutes and gets ignored. "Does this Suggested description look right?" costs two seconds. Editing beats authoring by an order of magnitude. The band is what makes the ask legitimate — the system admits it is unsure.

**The best reviewer is not the steward.** It is the analyst who queries that column every Tuesday. Query logs already name them. Route low-confidence items to the top three users of the asset, in the tool they already use, not into a ticket queue.

**Weight the confirmation.** An owner moves an item to Verified alone. Three independent frequent users move it to Reviewed with no steward. That is the only mechanism that scales past the five-percent plateau.

**Disagreement is signal.** Two users editing the same text in opposite directions means the column is ambiguous. That is a glossary gap. Escalate it. Do not resolve it quietly.

## Why the band changes usability

- Search ranks by band, so the certified revenue table beats the abandoned copy.
- Bands travel into the BI tool, the SQL editor and the notebook — trust at the point of use, not in a portal nobody visits.
- Agents ground only on Verified and Reviewed. An LLM will happily build a confident answer on an unreviewed guess. Gating retrieval by band is the cheapest AI-safety control a data team has, and it falls out of work you were doing anyway.

This is Part 1 applied to generation: **put metadata on the critical path.** If search, access and agents ignore the band, the score will rot like every other descriptive field.

## The loop that compounds

- Steward edit → labelled example
- Search and click → business term linked to a physical asset
- Join in a query → relationship no schema declared
- Good answer in the help channel → glossary entry waiting to be written

The metadata layer should get better every time anyone uses the data.

## What good looks like

| Measure                                          | 12-month target |
| ------------------------------------------------ | --------------- |
| Description coverage on tier 0–1 assets          | \>90%           |
| Sensitivity recall on regulated data             | \>97%           |
| Sensitivity precision on regulated data          | \>90%           |
| Calibration error across bands                   | <5 points       |
| Published metadata with band + provenance        | 100%            |
| Items reaching Reviewed without a steward        | \>40%           |
| Median confirmation time                         | <15 seconds     |
| Provisional enforcements unresolved past 14 days | 0               |

**Two questions this week:**

1. If your classifier is wrong, how would you know? If the answer is "someone would complain eventually," that is not a measurement problem. It is an accountability gap wearing a measurement problem's clothes.
2. What does the system do differently at Verified versus Unverified? If nothing, the band is decoration.