Confident Numbers That Are Wrong

Fluent, Precise, and Wrong

A language model asked for a figure produces the most probable-looking figure rather than a retrieved one, because nothing in the mechanism distinguishes a number it has seen from a number that merely fits the shape of the sentence. That mechanism is treated properly in AI Deep Dive, and AI for Legal follows the same failure into invented citations; what this lesson is about is why finance is an unusually bad place for it to land. In most domains a fabricated detail looks odd. In finance it looks like work: a spread quoted in basis points, a leverage ratio to two decimals, a year-on-year change that sounds about right. The output carries exactly the signals the profession uses to indicate care — units, precision, a source-shaped attribution — supplied by a process that used none of them.

• The mechanism is general and covered elsewhere; what is finance-specific is how convincing the result looks
• Financial formatting — decimals, basis points, ratios — reads as evidence of care it did not involve
• Fabricated attributions travel with fabricated figures: a filing, a note, a quarter that does not say it
• Fluency in your own conventions is what makes it persuasive, so domain expertise is not protection here

Where Wrong Numbers Enter the Work

The high-risk surfaces are the ones where a model compresses a long document into a short claim: summarised filings and prospectuses, condensed earnings calls, credit memos assembled from a data room, market commentary, pitch material, and any spreadsheet formula a model has drafted. Within those, the recurring defects are specific rather than random. The right number from the wrong period. A restated figure presented as originally reported. Consolidated confused with segment. Currency and units silently converted, or silently not. A percentage change computed against the wrong base. Totals that do not tie to their components. None of these announces itself; each stays invisible until somebody opens the source and looks.

• Compression is where the risk sits: summarised filings, condensed calls, credit memos, drafted formulas
• The classic defects are period, restatement, consolidated-versus-segment, currency, units, and the base of a percentage
• Checking that totals tie to their components is the cheapest test and the one most often skipped
• A model will also attach a real figure to the wrong document, which survives a shallow check

Tie It Back to the Source

Verification only survives a deadline if it is cheap and specific. Treat every figure a model produces as unsourced until somebody has opened the filing, the pricing screen, or the system of record and seen it there. Require a locator — document, statement, period, page or note — rather than a source name, because a name is what the model is best at inventing. Recompute anything derived instead of accepting it, since ratios and growth rates are where small errors compound into confident ones. Never ask the model to confirm its own figure; asked to check, it produces confirmation and more invented detail. And check at the point where the number enters the institution's work, not where it leaves it.

• Treat every model-produced figure as unsourced until a person has opened the source and seen it
• Demand a locator — document, statement, period, page — not a source name a model can invent
• Recompute derived quantities, and never ask the model to verify its own output
• Check at the point of entry into the work: by committee stage a figure has been believed several times

Try It Yourself

This takes about fifteen minutes and it usually finds something. Run it once, on a document anybody can read.

Company and filing used (public document only):
Figure as stated by the tool:
Figure in the filing:
Period correct: yes / no
Basis correct — consolidated or segment, reported or restated: yes / no
Currency and units correct: yes / no
Ties to its components: yes / no
Attribution — did the named page or note actually contain it: yes / no