What's being tested
Every test claim gets sorted into one of five categories:
- Clean — no issue found.
- Complex-math — a real coverage calculation (a deductible or benefit-cap crossing), but mechanical, not a judgment call.
- Ambiguous — a genuine judgment call the system can't resolve from the claim alone.
- Missing-data — a required field is actually absent.
- Fraud — phantom billing, upcoding, unbundling, double billing, unnecessary care, or a documented provider volume spike.
That category, plus a confidence tier from a second, independent pass, together decide whether a claim resolves automatically or goes to a human — the mechanism the "silent miss" section below is about.
The accuracy arc
Six prompt iterations, same fixed claim set, same instructions on both sides wherever the test called for it. Each point is the mean of a 5-run batch, graded against the ground truth understood at the time — Claude leads every iteration shown here. A ground-truth correction and one further iteration since (see below) brought Kimi to 18.6 in its most recent batch, ahead of every point on this chart — that result isn't plotted here yet, since it isn't graded on the same basis as the batches that are.
What each iteration changed: "Base addendum" — a handful of worked examples added to the prompt for categories the model was getting wrong. "+ complex-math example" — one more worked example, targeting a specific miscategorized claim. "+ reasoning field" — the model must write a brief explicit justification before committing to a category, not just after. "+ reasoning, tightened" — a follow-up instruction telling it not to default to "clean" when uncertain. "Reasoning removed" — that field dropped again, since it helped Claude but not Kimi (see below).
Reliability: how often a run fails outright
Independent of accuracy — how often did a batch exhaust all retries and fail to complete at all. Measured across the three clean, uninterrupted same-day batches (a rate-limit-affected batch is excluded as an infrastructure confound, not a model signal). The pattern held in every later batch too: Kimi has completed 20 of 20 runs cleanly since; Claude has failed roughly 1 in 5, a real and repeated pattern tied to its own Confidence step, not a fluke.
Accuracy and safety are both the story — just different questions
ClaimsDock's pipeline is two model calls, not one: a first pass proposes a category, then a second, independent pass assigns a confidence tier without seeing the first call's reasoning. A deterministic rule then decides what happens next — a claim that's actually fraud or ambiguous always reaches a human, regardless of confidence, but a claim wrongly called "clean" with a confident tier resolves automatically, with no human ever reviewing it. Raw accuracy can't see that difference; it counts every wrong category the same. This section can — it's the one place a wrong-but-cautious answer and a wrong-and-confident one are told apart. Both charts matter: one says how often each model is right, this one says how dangerous it is when it isn't.
Kimi — 10 runs across two batches, own best configuration
Claude — 10 runs across two batches, own best configuration (2 failed outright, shown below)
Full run counts shown above, not just one representative batch — 10 for Kimi (0 failures), 10 for Claude (8 successful, 2 failed outright, shown as hatched bars rather than silently dropped). Kimi held at 13 silent misses across its 10 runs; Claude at 4 across its 8 successful ones — steady enough across two independent batches each that this wasn't a one-off draw on either side, at the time it was measured. Since then, one of Kimi's own ground-truth labels was found to be mislabeled (see below) — that single claim accounted for most of Kimi's silent misses above. Correcting it and adding one more targeted prompt example dropped Kimi's rate to 5 silent misses in its most recent 5-run batch, close to Claude's own rate rather than meaningfully above it. The bars above reflect the original measurement, not redrawn onto it, so the two eras of grading stay honestly distinguishable.
What the process itself surfaced
Some of the most useful findings weren't about either model — they were about the testing itself.
- !Our own test harness had a grading bug — twice.
Two claims' ground-truth labels described which authored scenario they belonged to, not the category the Pipeline should actually assign to that specific claim — silently under-scoring both providers for most of the session. The second one mattered more than expected: once corrected, most of what looked like a Kimi-specific safety gap turned out to be a mislabeled test claim, not a real difference in how dangerous each model's mistakes are.
- ×Kimi's native "thinking" mode breaks structured output.
Enabling it floods the response with near-blank filler tokens and never completes the JSON — confirmed reproducible twice, including after the prompt changed substantially. Disabling it is what actually works.
- $Both providers hit a real infrastructure ceiling.
Claude's prepaid balance ran out mid-session; Kimi's new-account tier hit its 1.5M-token daily cap purely from testing volume. Neither was a model problem — both were fixed with a small top-up.
- ✓Targeted prompt examples measurably helped — the ones that were tested against real weak spots.
Two fraud sub-types that Kimi systematically under-flagged both recovered after adding worked examples targeting them specifically. A complex-math example, added for a different claim, never fixed its target — evidence that this kind of tuning has to be verified, not assumed.