ClaimsDock

Architecture & build-process planning

project-spec.txt · CLAUDE.md

Before any code: read the full functional spec end to end and worked through what it left open. Settled the evaluation pipeline down to two batched API calls total per run (not one call per claim, not three per claim) by moving severity, status, and the category-to-action mapping out of the model's hands entirely and into deterministic code — the spec was already explicit that those are calculations, not judgment calls, so the pipeline redesign just followed that logic all the way through. Confirmed the router as a real tool-use judgment step, not string-matching. Chose real chunked-embedding retrieval over full-context stuffing for the three RAG corpora specifically so the project demonstrates actual retrieval mechanics, not a shortcut past them — scoped to an in-memory vector store rather than a hosted database, since the corpora are small by design. Resolved a real multi-user question along the way: pipeline output (evidence, confidence, recommendation) is cached per ISO week and shared, since it depends only on claim data; adjuster actions are private, client-side, per browser — which is also what guarantees a new visitor always sees the full, untouched queue.

All of this is now written into project-spec.txt (amended in place, with each decision dated inline) and a new CLAUDE.md that sets the stack, file structure, and the 14-phase build order. Two open items carried forward on purpose rather than forced to a premature decision: which embedding provider to use for retrieval, and the batching strategy once the claim set grows past 20.

Claims schema & seed generator

src/lib/claims/ · scripts/validate-claims.ts

Repo scaffolded (Next.js 16, TypeScript, App Router) and pushed to GitHub. On top of it: a full claim schema (types.ts) with genuinely separate CMS-1500 and UB-04 shapes, field names mirroring the real box numbers rather than generic ones, and 20 authored claim records covering all 15 scenarios from the spec's test-set design — exactly 4 clean, 3 ambiguous, 3 missing-data, 3 complex-coverage-math, and 7 fraudulent, including the linked professional/facility pairs and the documentation-mismatch pattern that only shows up by comparing two linked claims against each other.

Content was authored for domain plausibility, not just schema conformance: age- and sex-appropriate procedures, diagnosis/procedure pairings that read as a real encounter rather than a random field-filler, and a shared provider identity reused across three of the fraud claims to produce the volume-spike pattern — which needed its own small dataset (provider-history.json) stating a trailing 6-month average against a current-month count, since "3.2x this provider's average" isn't something a single claim can carry on its own.

Only submitted_date is generated rather than authored, per the spec's own instruction — a small seeded PRNG (mulberry32) keyed to the current ISO week produces a deterministic-but-varied spread of claim ages (fresh, mid, near-deadline, one deliberately past its SLA deadline) that regenerates consistently within a week and shifts the next, with no reset step required.

A validation script checks the set for internal consistency — reciprocal linked-claim pairs, valid enum values, line-item totals that actually sum to the stated charge, and the exact scenario distribution the spec calls for — deliberately scenario-aware, so a claim's own _testMeta.deliberately_missing_field tag exempts its one intentional gap from being flagged, without silencing checks anywhere else. One real type-safety catch along the way: an exhaustive form-type check's defensive fallback branch was quietly unreachable as far as TypeScript's own narrowing was concerned (never), which is correct given today's two form types — cast rather than suppressed, so it stays a real guard if a third form type is ever added.

Author the three RAG corpora

content/corpora/

Three condensed reference documents (~500–600 words each) plus a typed manifest for the retrieval layer to consume in Phase 4: Coverage & Adjudication Policy (fully synthetic, in-house — coverage percentages, deductible logic, network rates, prior-auth rules, benefit caps), Fraud-Indicator Reference, and Regulatory Deadline Reference (ERISA's tiered deadlines plus state prompt-pay timing).

Before writing the fraud reference, a secondhand relay from a companion planning thread claimed the five billing-fraud category names (phantom billing, upcoding, unbundling, double billing, unnecessary/substandard care) came directly from FinCEN's FIN-2026-A001 advisory. Rather than build a citable document on that relay alone, fetched the advisory and NHCAA's consumer page directly. The volume-spike red flag and NHCAA's misrepresentation framing held up; the five category names did not — FinCEN's actual advisory centers on financial red flags (shell companies, ownership-change reimbursement spikes), consistent with its real mandate as a financial-crimes regulator rather than a medical-coding one. The terms themselves are real, just grounded elsewhere (OIG enforcement, DOJ False Claims Act settlements, CMS's National Correct Coding Initiative for unbundling specifically) — I corrected the attribution in the corpus and in project-spec.txt at the time. The same wrong attribution had already made its way into the workflow-visualization diagram's own node copy, though, and needed a second, separate correction once that diagram was revisited later (Phase 10) — a sourcing error, once caught, is often already further along than the one place it was first noticed. ERISA's own tiered deadlines (72hr/15-day/30-day) were checked the same way and held up exactly as specified, against DOL/EBSA guidance and the eCFR text — with one real addition worth folding in: each non-urgent tier gets a single notified extension (up to 15 or 30 more days), which wasn't in the original spec draft.

Reference material

• Coverage & Adjudication Policy
• Fraud-Indicator Reference
• Regulatory Deadline Reference

Deterministic calculation layer

src/lib/rules/ · 31 tests, 6 files, all passing

Five pure functions, no API calls: severity.ts (dollar band × SLA-window escalation, medical-necessity bump, breach override), sla.ts (percent-of-deadline-window math), coverage.ts (deductible/network-aware coverage math, verified against the spec's own worked example — $1,140 billed → $912 covered, exactly), action-lookup.ts (the category+confidence → action table), and status.ts. Vitest installed as the test runner.

Planning this one surfaced a real design question worth recording: is "disputed medical necessity" — an input severity needs — something to extract from messy claim prose, or something the deterministic layer should just be handed as a clean field? Concluded it's neither exactly — it's a judgment the Pipeline's Analysis call (Phase 5) has to make by reasoning over the claim, the same way it judges the unnecessary/substandard-care fraud category. Rather than pre-bake that judgment as a boolean now, computeSeverity() takes it as an explicit parameter, sourced from _testMeta as a stand-in until Phase 5 exists.

Two structured reference datasets added alongside the rule modules — network-directory.json and member-accumulators.json — modeling something true of real claims processing: network status and deductible balance both live in separate systems (a payer's network directory, a member's benefit accumulator), never on the claim form itself, so looking them up rather than parsing them off the claim is the domain-accurate design, not a shortcut.

Separately, added CMS-1500's real Box 19 ("Additional Claim Information") and UB-04's Box 80 ("Remarks") — genuine free-text fields on both real forms — to eight of the twenty seed claims, with realistic clinical shorthand for Phase 5 to eventually read and reason over.

coverage-policy.md is now generated, not hand-maintained: coverage-constants.ts is the single source of truth, and scripts/generate-coverage-policy.ts renders the citable prose document's tables from it, so the numbers Anchor cites and the numbers the calculation actually uses can't quietly drift apart.

Status taxonomy correction

project-spec.txt · src/lib/rules/status.ts · workflow diagram

A review of the status taxonomy (project-spec.txt Section 7a) against the deterministic layer already built surfaced a real bug in the spec itself, not just an ambiguity: the original implementation note said initial status was "derived from whether Call 1's evidence is non-empty." That rule is wrong — a complex-math claim or a non-material-missing-data claim both produce non-empty evidence, yet both need to stay "Submitted, no flags," since Approve is still a reachable action for each. The correct rule is category-based: fraud, ambiguous, and material-missing-data start flagged; clean, complex-math, and non-material-missing-data start unflagged. status.ts was rewritten accordingly.

Working through the taxonomy also surfaced two genuine gaps: Denied was missing as its own status — the original 5-value list folded every closed outcome into "Resolved." Adding it back passes the same non-redundancy test that got a 6th "Overdue" status rejected earlier: denial carries real regulatory weight an approval doesn't. Additional Info Requested — a material missing-data hold was previously modeled as just a clock-pause with no visible status, implying eventual re-evaluation once the missing field arrived. This system has no real document-intake mechanism for that field to ever arrive, so calling it anything other than terminal would misrepresent what the system can do.

Status is now 7 values, up from 5 — each addition justified individually against the same non-redundancy test.

RAG retrieval layer

src/lib/rag/ · 54 tests passing · real end-to-end smoke test against live queries

Local, self-hosted embeddings — @huggingface/transformers running Xenova/all-MiniLM-L6-v2 — over Voyage AI, closing the one embedding-provider decision left open since the RAG mechanism was first designed. Chunking splits each corpus on its own ## headings; embeddings and a plain in-memory array replace any hosted vector store, appropriately scoped to a few dozen chunks total.

The interesting part of this phase was verifying retrieval actually works, not just that it runs. A real smoke test — genuine questions against the real corpora, not synthetic fixtures — surfaced retrieval quality problems worth working through:

• The chunk containing ERISA's actual tiered-deadline table ranked 5th of 6 for "what does ERISA require for pre-service claims" — the one question it was built to answer. Root cause: the chunk was built almost entirely from raw markdown table syntax, which a sentence-embedding model embeds poorly. Fixed by embedding a flattened, plain-language version of any table while still citing and displaying the original markdown.
• That fix barely moved the ranking. The real mechanism: mean-pooled embeddings dilute a long chunk's vector toward its average content, so a short, keyword-dense chunk can systematically out-rank a longer chunk that actually contains the answer. Compounded by an actual authoring mistake: every document's pre-heading "Overview" chunk was just its provenance line, already captured structurally elsewhere, yet short and keyword-dense enough to out-rank real content — dropped from the retrievable index entirely.
• The underlying dilution problem remained, so added hybrid scoring, blending embedding similarity (70%) with a lightweight keyword-overlap signal (30%). That recovered the ERISA tiers chunk to 2nd place. Default k bumped to 4 for margin, which costs nothing at this corpus size.

Left deliberately unresolved: retrieval ranking isn't perfect and won't be tuned further at this scale — the practical backstop is that Anchor feeds several top-k chunks to the model for actual answer synthesis, not just the single top-ranked one.

Evaluation pipeline

src/lib/pipeline/ · Sonnet, low effort · verified live against all 20 seed claims

The two batched calls specced back in Phase 1: Call 1 (Analysis) reads every claim — plus its linked claim for combo pairs, plus the billing provider's volume history for the spike pattern — and returns evidence and a proposed category, no tier, no recommendation. Call 2 (the isolated Confidence judge) is shown only that evidence and category, never Call 1's reasoning or the original claim again, and returns a confidence tier plus a recommendation narrative written to double as citation text for the Human Gate's mismatch-warning guardrail. Both calls use structured JSON output rather than hoping a prompt-shaped response parses cleanly, and both stuff all three RAG corpora into context whole rather than retrieving per query — the corpora are a few pages total, and real chunked retrieval is reserved for Anchor, where it's the actual thing being demonstrated.

A deterministic safety net sits between the two calls: whether a required field is actually null is a confirmed fact, not a judgment call, so missing-fields.ts checks every claim directly and overrides Call 1's category if it disagrees with a detected material gap.

The live smoke test surfaced a real reliability bug on the first two runs, not a hypothetical one: a batched call occasionally came back missing one or more claims from an otherwise well-formed, schema-valid response — structured outputs constrain each item's shape but can't express "exactly N items, covering this exact ID list," so the model can under-generate a batch without tripping any parse error. Root-caused by instrumenting both calls' raw stop_reason/usage and replaying the same inputs against a cached intermediate result. Fixed two ways: the user message now explicitly enumerates every expected claim_id and the exact count expected back, and both calls are wrapped in a completeness check that retries (up to 3 attempts) rather than trusting the first response.

End-to-end result against the real 20-claim set: 20/20 categories matched their authored ground truth, with sensible, claim-specific evidence and correctly computed severity/status/SLA for every claim — including one claim the model itself flagged as already past its SLA deadline.

Severity-on-terminal-status design

project-spec.txt Section 7b — caught reviewing this phase's own results

Reviewing the live Pipeline output surfaced a real gap: severity's deadline-proximity factor assumes a still-running decision clock, which stops being true the moment a claim is actually decided. Resolved by checking the actual regulation rather than assuming: ERISA's decision-deadline rule governs the determination only, not payment speed; state prompt-pay statutes do keep a payment clock running past approval, but none reset a fresh timer at the approval date. Since this system explicitly doesn't simulate reimbursement at all, there's no honest data to keep escalating severity against post-decision — so severity resets to Low on Resolved or Denied. Additional Info Requested is the deliberate exception: it was never actually decided, just stuck, so its severity stays frozen at the hold's start value.

Interactive router & Anchor

src/lib/router/ · Sonnet, low effort · verified live against an adversarial question set

Real Claude tool-use across the four tools from Section 1: Lookup, Analysis, Recommendation, Reference Lookup. How the Router handles an ambiguous, broad query like "show me problem claims" had sat in Open Items since Section 1 was first written — resolved by extending Lookup to accept a structured filter (status/severity/category) alongside a specific claim ID, plus a small house table mapping common ambiguous phrases to a default filter. The response pattern is deliberate: always show best-guess results now and state the interpretation plus a named alternative in the same answer — never a silent guess, never a blocking clarifying question before showing anything.

Analysis and Recommendation default to re-surfacing the already-computed Pipeline result for a claim — free, instant, code-only — rather than paying for a new call every time an adjuster asks about a claim the Pipeline already evaluated. An explicit "re-check" request reruns Call 1 + Call 2 fresh for just that one claim, reusing Phase 5's own functions directly.

A ten-question adversarial set — not happy path — probed the three failure modes Section 1 names explicitly, plus the ambiguous-query fallback and a genuinely out-of-scope request ("deny this claim"). One real bug surfaced on the first live run: a question that reasonably needed two tools in sequence had its second tool call silently dropped, because the round-trip only handled a single dispatch round. Fixed as a bounded loop (up to 4 rounds). Rerun clean: correct tool chosen in every case, including refusing the out-of-scope denial request while naming what it could actually do instead.

Also gave Phase 4's real chunked retrieval layer its first actual caller — the Pipeline deliberately uses full-context stuffing instead, so Reference Lookup is where retrieval finally gets exercised for real.

Human Gate guardrail design

project-spec.txt Section 4a — design decision, ahead of this phase's code

Speced two guardrail mechanisms for the moment a human's action is submitted: a deterministic recommendation-mismatch check (no model call — compares the human's chosen action against the Pipeline's own recommendation, surfacing a soft "Are you sure?"), and a small model call that judges the quality of a Deny action's required justification against the claim's evidence and the retrieval layer's policy corpora, drafting a policy-cited replacement when the human's own text is inadequate. Both mechanisms are deliberately soft: neither blocks a human from proceeding with their original choice — a system that could override a human's Deny or Approve decision would contradict the human-gate principle the whole platform rests on.

Human gate & persistence

src/lib/humangate/ · src/lib/persistence/ · real jsdom tests · verified live

Three scope questions settled before writing anything: the guardrail check needs a live model call, so does that mean standing up the first real app/api/ route? No — a plain server-side function now, tested headlessly, with the HTTP wrapper waiting for Phase 8. localStorage only exists in a browser — real jsdom-backed tests instead of a hand-rolled fake. And the real per-ISO-week Pipeline cache: build it now, since the Pipeline is provably nondeterministic run-to-run and the audit log needs a stable snapshot to log against.

Both Section 4a guardrails built: the deterministic mismatch check, and the RAG-grounded justification-quality check, genuinely retrieving against the corpora rather than full-context stuffing. Also corrected a real architectural inconsistency while touching this code: the SLA-clock-pause logic was reading the authoring answer-key field directly — the field explicitly meant to never touch real system logic — swapped for the deterministic missing-fields scanner instead.

Section 6's coverage math got its first real caller. That surfaced a genuine bug on the first pass: CMS-1500's charge field turned out to be a per-unit rate needing multiplication by units, while UB-04's total-charge field is already a line total — mixing them up understated one claim's total by $420, caught by cross-checking the computed total against each claim's own authored total across all 20 claims.

Live verification: a flippant justification ("Eat my shorts") was correctly rejected with a policy-cited replacement. More telling: a vague-but-serious-sounding justification ("found a discrepancy the automated evidence missed") was rejected too, for being unverifiable rather than for tone. And in one case where the chosen action actually matched the recommendation, the justification check still caught that the adjuster's stated reason named the wrong fraud category against the claim's real evidence.

Human Gate extension — structured denial, reversible actions, recoupment

src/lib/humangate/ · src/lib/rules/ · test suite extended, all green

Deny's justification split from a single free-text field into four structured ones — specific reason and the plan/policy provision cited (both required), plus the internal rule or clinical standard applied and what might reverse the decision (both optional) — so the justification-quality guardrail can point at exactly which field falls short. A genuinely new terminal status, Recoupment Requested, reached only from Resolved, modeled deliberately as forward-only, not a reversal — Approve stays permanent, since unwinding a real payment has actual legal weight this system doesn't simulate. Paired with three real reversible actions — undoing a request-for-info, an escalation, or a denial — each looking up its own matching entry in that claim's audit log and reverting to that entry's prior status.

Placeholder UI (8A)

app/

A plain, unstyled worklist and card view, verifying Phases 1–7 end to end — real severity/status/SLA logic, real category+confidence action lookups — before any visual design existed to wire against. Superseded by the real UI below rather than kept running in parallel, once the Figma work landed.

Visual design system

Figma · token-driven · three styles × light/dark

Five initial style directions narrowed to three, each carrying a genuinely different point of view rather than a palette swap: Ledger (dense, tabular, high information density), Clinical (serif, precision-first, reads like a clinical record), and Field (warmer, more approachable, for a less adversarial read of the same data). Each ships in light and dark — six combinations total — driven end to end by a token architecture rather than six hand-maintained copies.

Contrast held to WCAG AAA (7:1) throughout, checked computationally rather than eyeballed — caught and fixed three real failures (accent text against its own ink color dropping as low as 4.77:1 in Clinical light and 6.98:1 in Ledger dark) by retuning the underlying primitives, so the fix propagated everywhere at once.

The Claims Card carries a three-way toggle — Quick View, All Claim Fields, Audit Log — grounded directly in the real schema. Guardrail states from the Human Gate design are fully surfaced: a soft "[Action] anyway" confirmation naming the Pipeline's actual recommendation, and Deny's four-field justification form with a real accept/edit/submit-as-is/cancel flow.

Real UI (8B)

components/ · React + TypeScript + CSS Modules · verified in both dev and production builds

The complete Figma design wired into working React, across all six style/mode combinations. A token architecture (src/app/tokens.css) carries every color/font/radius decision through [data-style]/[data-theme] attributes.

A handful of real, non-cosmetic bugs came out of this pass, each root-caused rather than patched around:

• A shared fade-in/fade-out hook was starting its exit sequence from inside a useEffect, which only runs after React commits — so on the exact render a modal closed, it was already unmounted for one real paint before the effect could re-add it with a fade-out class. Fixed by catching the transition synchronously during render instead of in an effect.
• A status/severity legend popover was getting cut off at the bottom of a short table. Root cause: the table's own horizontal scroll region set overflow-x: auto without setting overflow-y explicitly, which CSS quietly resolves to auto too — clipping the popover along with it. Fixed by portaling the popover to document.body.
• Legend badges were overflowing their panel even after repeated width increases — the actual cause was the badge component's own white-space: nowrap, not the container.

Added after the rest of this phase was already marked done: the checkbox-driven bulk-action bar — selecting any row surfaces a persistent "N selected / Approve N claims / Escalate N claims / Cancel" bar. No bulk actions actually executed yet — this was a render/interaction pass only, deliberately ahead of Phase 13's human-gate wiring catching up to it.

Workflow visualization

standalone Artifact — built out of numeric order, on purpose

An interactive, draggable, zoomable D3 diagram of the full system — the evaluation pipeline, the five category branches and their distinct recommendation logic, the human gate, the interactive router's four tools, Anchor's RAG retrieval path, and the confidence/severity/status taxonomies as a separate reference cluster. One big overview plus three camera-preset detail views sharing the same underlying node set.

Colors reuse the build-log's own ink/teal/coral neutrals for continuity, but the three subsystem colors are a categorical triple run through an actual colorblind-safety validator rather than hand-picked. Two real interaction bugs surfaced and got fixed after review: a text-overflow bug traced to a CSS text-transform/letter-spacing pair the wrap-measurement code wasn't accounting for, and a click-required-twice bug that turned out to be d3's pan-to-zoom and each node's own drag behavior both reacting to the same mousedown — fixed by excluding node-originating gestures from the canvas-pan behavior entirely.

The workflow diagram is badly out of date (rewrite, late in the build)

public/workflow-diagram.html

Asked directly and checked rather than assumed: the standalone diagram is describing roughly a Phase 5–7 snapshot of the system, not what exists today. It still states "20 claims → 2 total calls per pipeline run" as a hardcoded fact (actual: 132 claims, 6 chunks, 12 calls); has no chunking architecture at all; no Kimi/provider-flexibility story anywhere; describes the Router as "one-shot per question, not multi-turn" (false); and still lists thumbs-up/down feedback on Anchor's answers, which was cut from scope and never built. Rewritten node-by-node against the current codebase rather than patched. Node labels stayed factually plain; only the click-through detail text picked up a more direct, ownership-first voice for the decisions behind each piece.

Reference material

• The current interactive workflow diagram

Pass A — claim-ID redaction

src/lib/claims/claim-number.ts · verified live across the Pipeline, Router, and Anchor

Planning the 100+ claim scale-up surfaced a real, previously-unnoticed problem: the internal claim_id authored for this test set (FRD-UPCODE-01, FRD-MISMATCH-01A, etc.) was being sent to the model unmodified in every Pipeline and Router prompt — and that ID spells out the exact fraud sub-type in its own text. Fixed with a single opaque, realistic-looking claim number (CLM-nnnn-nnnnnn) generated per claim and used for two jobs at once — the UI's display-facing number, and the only claim identifier ever sent to or returned by a model call.

Live re-verification surfaced a second, unrelated finding: the Confidence call was completing a full 20-claim batch reliably only about half the time. Three model-configuration hypotheses were tested and ruled out — a more visually-structured ID format, a doubled token budget, and a bump to medium reasoning effort (which made it measurably worse). The fix that actually worked came from a prompt rewrite: reframing the task explicitly as producing a set of paired judgement-and-recommendation results, rather than a flatter list of dual-purpose entries. Five consecutive live runs afterward: four clean on the first attempt, one recovered after a single retry, zero outright failures.

Pass B — Pipeline wired to a real route

app/api/pipeline/route.ts · dashboard-placeholder-data.ts deleted

The homepage now awaits a real, live two-call Evaluation Pipeline run instead of reading hand-authored placeholder data. Three real bugs surfaced immediately: severity was silently ignoring its own terminal-status reset rule; a Turbopack-specific path-resolution bug broke reading the RAG corpora from inside a real route handler (__dirname doesn't point where it used to once Turbopack relocates a compiled route's chunk — fixed with process.cwd()); and next build was quietly trying to prerender the homepage at build time, meaning every production build would trigger a real, costly Pipeline run. Fixed by marking both the homepage and the API route explicitly dynamic.

Live testing also surfaced a real, repeated misclassification problem: claims authored around deductible and inpatient-day-cap math kept landing as "clean" or "ambiguous" instead of "complex-math." Root-caused to two compounding causes — Call 1 had no visibility into a member's remaining deductible, network status, or annual inpatient-day usage, and separately the category definitions shared an example with the real distinguishing logic buried in a subordinate clause. Fixed both: the missing facts are now handed to Call 1 as raw inputs, and the category instructions gained a hard decision test plus a self-check.

Pass C — Router wired to a real route

app/api/anchor/route.ts (dispatch layer)

The four-tool dispatch built and adversarially tested back in Phase 6 got its first real route to live behind. Surfaced the same class of Turbopack path bug found in Pass B, in a different subsystem — the corpus loader had never actually run inside a bundled route handler before, since nothing had called it live until now.

Pass D — Anchor panel wired to a real route

components/AnchorPanel.tsx · real fetch/loading/error states · verified live

The Anchor panel's input, loading state, and answer rendering are now genuinely live. A long night of live, adversarial-by-accident testing surfaced a real, separate leak Pass A hadn't caught: a hardcoded old-format claim ID example baked directly into the Lookup tool's own schema description, sent to the model on every call regardless of any runtime redaction — fixed. It also surfaced a genuine, still-open list: no way to search by patient or provider name yet, filtered/list results don't get a citation card, answers render as literal markdown syntax, the panel wipes its previous answer on every new question, and — most interesting — a live example where an answer and its own citation card disagreed with each other, traced to the Pipeline's cache running independently per route rather than being shared.

Pass E — full prompt review

All four live model prompts reviewed end to end · Anchor's tools reworked · re-verified against an expanded adversarial set

Every live model-facing prompt read in full and revised against real, observed behavior rather than left as originally drafted. Call 1 was restructured into clearly delineated sections; a proposed addition asking it to justify its own category choice was deliberately dropped — that job belongs to Call 2's isolated recommendation, which never sees Call 1's reasoning and has to judge independently.

Anchor's system prompt changed the most. Lookup gained real filtering by patient name, provider name, dollar amount, and SLA-percent-remaining — closing a real, observed gap where an adjuster asking about a claim by patient name was told it didn't exist. Recommend-action was reworked from "draft a fresh recommendation" to "explain and draft supporting language for the one already computed," since a second, independently-recomputed one risked contradicting it. The tool-use loop gained a graceful fallback: reaching the round limit without a final answer used to throw an error; it now forces one last tools-off call summarizing what was found instead.

Re-verified against an expanded adversarial set — nine new cases layered onto the original ten. One incidentally proved the redesigned recommend-action tool works better than the test itself assumed — asked to draft a denial justification for a claim actually on record as Escalate, Anchor caught the mismatch on its own and offered the correct thing instead.

Kimi / provider flexibility

src/lib/pipeline/model-client.ts · scripts/compare-providers.ts · Pipeline routed to Kimi in production

Motivated by two real things this build surfaced on its own: Kimi's meaningfully lower cost, and the Pipeline's own observed run-to-run classification inconsistency, which raised a fair, testable question — does a different model handle the same batched-reasoning task more consistently, given the same complete information? Architecture first: model-client.ts consolidates the Pipeline's two calls behind one shared callModel(). Anchor's tool-use loop and the Deny justification-quality check were deliberately left out of this consolidation — their request shapes differ enough between providers that forcing a shared interface would be a leaky abstraction.

Getting Kimi actually running surfaced three real infrastructure bugs. K2.6's own native reasoning mode breaks structured JSON output outright — floods the response with a long run of near-blank whitespace and never completes the object, confirmed reproducible twice. Running with reasoning explicitly disabled is what actually works. Separately, the OpenAI SDK's own client-level timeout didn't reliably fire — one request ran 13+ minutes past its configured bound with no error at all — replaced with a manually enforced Promise.race timeout. And Moonshot's own Tier-0 new-account rate limit got hit purely from testing volume, resolved with a real top-up crossing the threshold that unlocks Tier 1's unlimited daily cap.

Comparison ran through a real multi-pass harness, not a single trusted run — the same fixed, ISO-week-seeded 20-claim set both providers see identically, repeated passes per configuration. That produced the phase's most useful methodological finding: raw accuracy treats every miscategorized claim as equally bad, but the deterministic layer doesn't. A wrong category paired with a cautious confidence tier still reaches a human; only a wrong "clean" call paired with a confident tier resolves automatically, with nobody ever reviewing it — a "silent miss," and tracking it specifically turned out more decision-relevant than the accuracy number on its own. The harness also caught a bug in its own grading: one claim's ground-truth label described which authored "combo" scenario it belonged to, not the category the Pipeline should assign to that specific claim — grading against the raw label had been silently under-scoring both providers by up to a full point for the entire testing history until this was caught and fixed.

Prompt tuning produced a real, mixed result. Targeted few-shot examples for two fraud sub-types Kimi was systematically softening toward clean/ambiguous measurably worked. A dedicated example built for a separate complex-math claim never fixed its target, confirming this kind of tuning has to be verified against repeated results, not assumed to work. An explicit reasoning field, forced ahead of the category decision, was a clean win for Claude but neutral-to-mildly-negative for Kimi, which has no private reasoning phase of its own — kept for Claude, dropped for Kimi.

Final numbers, confirmed across two independent matched-prompt batches each: Claude averaged 18.4/20 (92%) category accuracy, a 5% silent-miss rate, and completed cleanly on roughly 4 of 5 runs. Kimi averaged 16.3/20 (82%), a 13% silent-miss rate, and completed cleanly on every single run tested. Real, opposite trade-offs — Claude is more accurate and safer when it finishes; Kimi finishes more consistently. The Pipeline's real cache and Anchor's own recheck path are now both routed to Kimi in production, a cost-driven decision for a public portfolio prototype with no real patients, providers, or dollars on the line — documented explicitly as a different call than an actual production system would be permitted to make.

Reference material

• Kimi vs. Claude comparison — full data visualization

Pass A — Anchor wired live, Kimi first

src/lib/router/anchor.ts · src/lib/router/model-round.ts · verified against a 19-question adversarial set

I ran Kimi first, cost-driven: the open items left from Phase 11 — a cache/pending duplication bug, a missing recommended-action label, one-shot-only memory — aren't provider-specific, so shaking them out against the cheaper model first meant not paying to debug the same logic twice. The duplication bug was real and visible: an Anchor answer and its own citation card could disagree on a claim's status or severity, traced to the Pipeline's cache not actually being shared across Turbopack's separately bundled routes. Fixed at the root: dashboard rows now carry the Pipeline result they already compute, and Anchor's route builds its claim index from that same client-supplied data instead of an independent cache read.

The panel gained a real scrolling history, paired with exactly one prior turn of memory, distilled to plain question/answer text rather than raw tool output. Live testing caught a real prompt bug, not a data bug: Anchor's own system prompt hardcoded an assumption that fraud always resolves to Deny, contradicting the actual confidence-tiered mapping. Fixed by having the prompt always defer to the tool's own recommended-action field instead of asserting one.

Pass B — Claude wired live, provider toggle shipped

components/SettingsPanel.tsx · same shared per-round helper as Pass A, second provider

Claude went through the identical shared per-round helper Pass A built for Kimi, rather than its own hand-rolled loop — the real test of whether that consolidation was a genuine abstraction. It held: the same 19-question adversarial set ran clean against Claude with no debugging needed. One real, worth-documenting behavior difference: asked to analyze five claims, Claude self-limited to three and explicitly offered to continue — the exact behavior the system prompt asks for. Kimi had done all five in one turn, since OpenAI-style function calling can request several tool calls in a single turn rather than exhausting the budget the way Claude's more literal one-tool-per-turn behavior does.

Claim-card action logic & Anchor visual polish

components/ClaimsCard.tsx · components/ChromeIcons.tsx

Claim-card action buttons now order themselves around the claim's actual recommendation instead of always leading with Approve. Anchor's "thinking" state moved from a static line of text to an actual shimmer — animated bars plus the Anchor wordmark itself. The wordmark's shimmer lands on CSS mask-image rather than an SVG <mask> element, after the mask-based version proved visually unreliable once combined with a moving gradient.

Pass C — Human Gate wiring, real persistence

src/components/ActionConfirmOverlay.tsx · src/lib/humangate/ · every action's overlay matched to Figma · Kimi-backed guardrail

The Human Gate stopped being a placeholder in this pass: every one of the seven action overlays plus a free-form Add-a-Note overlay now run against the real Figma design and real backend logic. A genuinely new mechanism came out of this pass: a uniform 4.2-second grace window on every action's confirmation screen — the action is held in memory only until the window closes, so a same-second correction leaves no trace, while Approve and Recoupment get their one and only chance to catch a misclick.

Per the original plan, the Deny justification-quality guardrail was built and tested against Kimi first, not Claude — confirmed live only after catching that the first implementation had quietly skipped that ordering and defaulted to Claude. Three real bugs surfaced and fixed live: a client-bundle build failure (Turbopack refused to ship a client component that transitively imported Node's fs through the Deny guardrail's own retrieval call); a Strict Mode double-write; and a stale-overlay flash where a just-committed confirmation screen briefly reverted to its own prior form during its fade-out.

Bulk actions, StatTiles wiring, and Anchor's selection awareness

src/lib/ui/bulk-actions.ts · src/components/BulkActionBar.tsx

The checkbox-driven bulk-action bar had been visual-only since Phase 9, deliberately deferred until real persistence existed to wire it to. Deny and Request Recoupment are deliberately excluded from bulk entirely — both call for individual review. Anchor gained real awareness of the checkbox selection itself, not just a single open card — a genuine gap found live when a multi-select "tell me about these claims" question came back empty-handed.

Five real bugs surfaced and were fixed during live testing of this pass, the most of any single pass this build produced: an empty-selection crash (the bar's own state calculation assumed at least one claim was always selected, which is false on every page's default load — this broke the entire worklist until fixed); a server-side localStorage reference; a stale-state bug where the sequential queue's overlay was keyed only by action type, reusing one component instance instead of remounting; a selection that cleared the instant a queue started rather than when it finished; and the Anchor filter-guessing gap described above. None of the five were caught by typecheck, lint, or the unit suite — all five needed a real running server and a real click-through.

Pass F — 103 new claims, 20 → 123

src/lib/claims/claims-seed-data.json · scripts/author-pass-f-claims.ts

I negotiated the category split before any data was written: 72 clean / 18 fraud / 13 complex-math / 12 ambiguous / 8 missing-data across all 123 claims. I set demographics the same way — ages 25–80 skewed above 50, four non-binary patients (a genuine schema widen), last names spanning several naming traditions.

Built as a one-time authoring script rather than hand-typed JSON, so every claim's total charge is derived from its own line items instead of manually re-summed. Three real bugs, caught by the existing verification loop before any of this reached the live app: a counting error double-counted UB-04-only clean claims; the CPT/revenue-code crosswalk was scoped to only the 20 original codes and threw on every new procedure code; and one authored missing-data claim nulled the exact billing-provider NPI field the coverage calculation depends on for its own network lookup.

Procedure/revenue-code plain-English lookup

src/lib/claims/code-descriptions.ts · ClaimsCard.tsx

"Care provided" showed only a raw CPT/HCPCS/revenue code, correct but not self-explanatory. A hand-curated code-to-short-label table now renders alongside every code rather than replacing it, verified by script against every claim in the full claim set, zero gaps.

Live Pipeline scope: capped back to 20, pending Pass G

src/lib/pipeline/cache.ts · src/lib/claims/generate-claims.ts

Two distinct bugs surfaced back to back the moment the 123-claim set actually got read by the running app, both crashing the worklist entirely. First: the per-ISO-week Pipeline cache had no way to notice the underlying seed data had changed mid-week — fixed by adding a cheap fingerprint of the actual claim set as an explicit second cache-key argument. Fixing that surfaced the deeper issue: a fresh live run tried to push all 123 claims through the existing two-call batched Pipeline in one shot and timed out outright — the exact batching-scale risk Pass G was always going to have to answer, just arriving earlier than planned. Rather than force that decision under pressure, a temporary cap (ACTIVE_CLAIM_LIMIT = 20) held the live app steady while Pass G's real fix got built properly.

Pass G — chunked batching, live at full 123-claim scale

src/lib/pipeline/chunk.ts · orchestrator.ts · batch-retry.ts

Measured before guessing: a real, uncached Kimi run at the proven 20-claim scale showed Call 1 using 10,316 of its 16,000-token budget and Call 2 only 3,111 — real numbers, not the flat, unmeasured ceiling both calls had shared since Phase 11. Extrapolated linearly, a single 123-claim Call 1 would need roughly 63K completion tokens and close to 25 minutes — past any of Kimi's own practical per-request limits and obviously past this project's own timeout.

Chunked at ~20 claims per call, chunk count derived from claim count. Getting the chunk distribution right took three real, live-tested attempts: a naive contiguous slice clustered clean claims together, producing real per-chunk token variance. A first fix balanced only a binary clean/non-clean split — helped Call 2 but made Call 1 worse, since fraud/complex-math/ambiguous/missing-data all stayed clustered within the "non-clean" bucket. A second fix distributed every category independently — still wrong, since every category's own round-robin started at chunk index 0, compounding across all five instead of canceling out. The actual fix, caught by a unit test before it ever reached a live call this time: concatenate every category's items in file order and round-robin a single continuous cursor across the whole concatenation, never resetting between categories.

One more real failure: a live run threw a connection-timeout error on one of six Call 1 chunks — a genuine transient network failure. Exposed a real gap: the existing completeness retry only retries a response that comes back incomplete, not one that throws outright, so a single flaky chunk was aborting the entire batch and discarding five other chunks' already-succeeded, already-paid-for results. Fixed with a new per-chunk retry wrapper, independent of the existing completeness retry.

Evidence-length cap: prose failed twice, a real schema constraint worked

analysis.ts's evidence field · maxItems, Kimi-only

Found live: a documentation-mismatch fraud claim rendered with 10 evidence bullets, despite the field's own schema description already saying "aim for 3-6." Tightening that same prose to an explicit "HARD CAP of 6, never more" made it worse — 13 bullets. Kimi does not reliably honor a numeric ceiling stated only in natural language. Switched to an actual JSON Schema maxItems: 6 constraint instead — confirmed live to work cleanly. Checked against the other live provider before trusting it project-wide: Anthropic's structured outputs reject maxItems outright (400 error) — would have been a real regression on the Claude path had it shipped unconditionally. Spliced in as a Kimi-only schema addition instead.

Two "stale after acting" bugs: Anchor's answers, the Claim Card's own recommendation

router/context.ts · ClaimsCard.tsx

Found live, in direct succession: a just-approved claim that Anchor still described as "Needs Approval," then a just-escalated claim it still called "Submitted, flagged." Root cause was narrower than the cache/pending duplication bug already fixed for a related symptom: Anchor's claim index built from each row's result field — the original, frozen Pipeline output — never from the row's own current status/severity. Fixed by patching the row's current status/severity into the result object stored in the index.

Second, separate bug, same session: the Claims Card's "Recommendation:" text had exactly one display-only override already, but that flag only covers a claim the Pipeline resolved with zero human input — a manually-approved complex-math claim kept showing its original recommendation as if still an open ask. Generalized into a real lookup table mapping each recommendation to the statuses that mean it's already been carried out, by any means.

recommendation_fulfilled reaches Anchor too, plus real-world terminal-state labels

recommend-action.ts · lookup.ts · status.ts

Anchor had the identical gap the Card did: asked "tell me what I should do with this claim" about an already-approved claim, it correctly reported the claim's real current status but still framed "Approve as calculated" as an open ask. Having the right facts in hand doesn't mean a model reliably draws the "already done" inference on its own. Confirmed live on a harder case: the same claim, asked about again after moving into a recoupment cycle, correctly recognized fulfillment still held and correctly distinguished the recoupment step as a separate, layered action.

Separately: three of the "fulfilled" statuses read wrong under a generic "No action needed" label — Escalated, Additional Info Requested, and Recoupment Requested are none of them actually closed. New real-world phrasing built for each. Found live within the hour: a second, untouched copy of the exact same display logic existed in Anchor's own citation mini-card, never updated alongside the Claims Card's fix.

analyze_claim's own gap: no status/severity/recommendation fields at all

analyze-claim.ts

Found live: "tell me about this claim" on an already-DENIED fraud claim came back with a full, correct evidence walkthrough but never once mentioned the claim had already been denied. Different root cause from the recommendation_fulfilled fixes: this result type never carried status, severity, recommended_action, or recommendation_fulfilled at all, on either of its two branches. Anchor had no way to know, not a reasoning gap. Fixed by adding all four fields to both branches.

recommendation_fulfilled's own false-case regression: narrating the check instead of just answering

anchor.ts's SYSTEM_PROMPT

The true-case instruction worked, but leaked into the false case too: asked about a genuinely still-open claim, Anchor narrated its own internal check on the ordinary, default case instead of just answering plainly. Tightened the instruction: the false case gets zero mention of the field at all — it should read identically to how Anchor answered before this field existed.

The real cause underneath all three recommendation_fulfilled bugs: prior-turn memory used as a fact source

anchor.ts's SYSTEM_PROMPT + priorTurnNote

Live-tested rigorously before concluding anything: asked the exact same question twice about the same claim with a real approval action taken in between, got back word-for-word identical text both times. Verbatim-identical output from two separate live model calls is not something stochastic generation produces by chance. Confirmed no caching exists anywhere in the client, the route, or the shared dispatch helper — every question genuinely fires a fresh request.

Root cause: the one-prior-turn memory feature's own instruction told the model to consult the prior Q&A "if the new question actually depends on it," but never said the prior answer's own factual content must not be treated as still valid. This explains all three of this session's bugs at once, not as three independent prompt gaps: if Anchor wasn't reliably re-querying fresh data on a repeat question in the first place, no amount of tuning what it says once it has fresh data would have mattered. Fixed with a prominent standing rule stating explicitly that status/category/recommendation/evidence can change between turns, and the prior turn exists only for reference resolution, never as a fact source.

Confirmed live afterward on a genuinely stronger test: asked about a Resolved claim, correct answer; after the claim was reversed back to Needs Approval, asked again — Anchor correctly picked up the new current state too, rather than repeating either its own prior answer or defaulting to a cached memory.

The claim-set rebalancing pass: 123 → 132, and two "hard" test cases turn out to have been data bugs

claims-seed-data.json · member-accumulators.json · PIPELINE_TARGET_CHUNK_SIZE

Raised directly during live testing: too many flagged claims dead-ended at Escalate, the least satisfying outcome for someone exploring the interface hands-on — a real UX/psychological consideration for a portfolio piece, weighed deliberately rather than dismissed as unnecessary polish. Rebalanced from 123 to 132 claims, with the chunk size bumped so the set divides evenly into 6 chunks instead of 7.

A genuinely surprising find while reconfiguring: two claims tested extensively earlier the same day as "genuinely hard" adversarial fraud cases turned out to already be tagged "clean" in the seed data — the same specialty/procedure-mismatch data bug described below, just discovered from the live-testing side before the systematic audit caught up to them.

A specialty/procedure-mismatch bug, then a systematic audit finds it's a real pattern, not a couple of one-offs

claims-seed-data.json · 3 rounds: one claim, then a linked pair, then a full audit

First found live through Anchor/Dcl testing, not authoring review: a claim tagged "clean" billed an ophthalmological exam under a cardiologist's own NPI — correctly flagged as fraud by the Pipeline despite its intended label. Checking whether that provider showed the same pattern elsewhere surfaced a second instance immediately. Given two instances from one provider, a systematic pass followed rather than assuming that was the whole problem: a generated table of every "clean" claim's provider/specialty, diagnosis, and procedure, read for plausibility.

It wasn't a couple of one-offs — every linked "combo" clean claim across 7 pairs (14 claims) paired a provider with a same-day surgical procedure clearly outside their specialty: an endocrinologist billed for a colonoscopy, an OB/GYN for a hernia repair, a psychiatrist for a knee arthroscopy, a physical therapy clinic for carpal-tunnel-release surgery. Read as a likely authoring artifact: providers and surgical scenarios cycled independently without cross-checking specialty against procedure. Fixed all 7 pairs plus two standalone claims in one pass — swapping the provider where a correctly-matched specialist already existed, swapping the procedure only where none did. Explicitly deferred: roughly 14 more standalone claims flagged at a softer, office-visit-level severity, tracked for a future pass.

Smaller fixes and polish: search, plain-English diagnoses, the SLA-deadline tile, page size

ClaimsTable.tsx · diagnosis-descriptions.ts · stat-range.ts

The claims-table search bar's case-sensitivity bug: the query was lowercased before comparing, but the display number wasn't — meaning a search for "CLM" or "clm" could never match. Procedure-code descriptions alone turned out too generic — a raw E/M code just reads "Office visit" regardless of what it was actually for, since that context lives in the diagnosis. Built a companion diagnosis-description table and combined the two.

The "claims nearing SLA deadline" stat tile's Today/7d/30d toggle produced a deeply counterintuitive spread live (1/50/131 of 132 claims) — traced to using absolute hours-remaining as the threshold when this system's SLA windows are wildly different sizes, meaning "within 30 days" was mathematically almost identical to "hasn't breached yet" for any standard-tier claim. Switched to percent-of-window-remaining, judging every claim on the same relative scale regardless of its actual window size.

The first real Vercel deployment

next.config.ts · embed.ts · vercel.json

Most of Phases 5 through 13 had never actually been committed to git — the repo's last real commit predated the Pipeline, Router, Human Gate, and the real UI entirely. Caught before deployment rather than after: reviewed, staged, and pushed as two commits (removing reference material and local settings that had no business being in a public repo, then one large catch-up for everything else), rather than fabricating a fictional phase-by-phase history for work that was never actually committed incrementally.

The deploy itself surfaced three real bugs, in order, none of which had ever shown up in local dev — exactly the kind of gap a genuinely different runtime environment exists to catch:

• A typo'd environment variable: IMI_API_KEY instead of KIMI_API_KEY surfaced as an OpenAI-SDK credential error, since Kimi's client silently falls back to checking OPENAI_API_KEY when its own key resolves empty.
• A missing native binary: @huggingface/transformers' onnxruntime shared library wasn't reaching the deployed function, since Next's automatic file tracing doesn't reliably catch a library that's dynamically loaded at runtime. Marking it as a server-external package alone didn't fix it — confirmed live, the same error persisted — an explicit file-tracing include for the two routes that actually call retrieve() did.
• A read-only cache path: once retrieval got further along, the library's default cache directory turned out to sit inside node_modules, which is read-only on Vercel at runtime — redirected to /tmp, the one writable path in a serverless function.

A fourth failure that looked related — an Anthropic "credit balance too low" error — turned out to be unrelated to any of this: a test script that happened to omit the provider field and fall back to Anthropic by an SDK default, not a problem with the app itself, which defaults to Kimi. Confirmed by testing both providers explicitly once the account was topped up: both work cleanly, with real grounded citations from the actual corpus.

The pipeline loading screen, and a live debugging arc inside it

loading.tsx · fact-cycler.ts

Replaced the placeholder spinner with real branding — the ClaimsDock and Anchor marks, sized and ordered as a deliberate hierarchy — the five-bar shimmer already built for Anchor's own "thinking" indicator, and a cycling list of real facts about the system, shuffled with no immediate repeats. Colors and type follow whatever style and theme the visitor last chose.

The cycling logic went through its own live debugging arc. A first implementation — a chain of nested setTimeout calls — tested correctly in isolation, including under React StrictMode and through the real hydration path, and yet stayed frozen on the very first fact through a genuine ~90-second cold Pipeline run in production. The actual cause: a Suspense fallback held open that long during a streamed SSR response is exactly the scenario most likely to have its DOM node torn down and recreated mid-wait by the browser or an intermediary proxy, silently resetting an in-memory timer chain each time. Rebuilt to derive the current fact from real elapsed wall-clock time, anchored in sessionStorage rather than kept in memory — self-correcting no matter how many times that happens, verified with a test that specifically simulates the remount rather than just the happy path.

Separately, the ClaimsDock logo itself was invisible on first deploy — a flat white fill meant for the masthead's own dark header, rendering near-invisible against the loading screen's light page background. Recolored via a CSS mask against the same ink-primary token the rest of the screen's text already uses, rather than maintaining a second light/dark asset.

Pre-warming the weekly Pipeline cache

vercel.json

The per-ISO-week Pipeline cache means only the first visitor after a weekly rollover pays for a live run — everyone else that week gets an instant cached result. Closed the remaining gap with a daily Vercel Cron hitting the Pipeline route at 00:05 UTC: cheap, since every call after the first real trigger that week is just a cache hit. A known, accepted limit rather than a full guarantee: a visitor arriving earlier the same day the ISO week rolls over, before that day's cron fires, could still hit one cold run a week — judged an acceptable tradeoff for a portfolio site rather than paying for sub-daily cron frequency.