Why We Built Three Systems to Get One Right

A regional payer with roughly two million members receives about 1.5 million prior authorization requests a year — imaging, procedures, specialty drugs, DME. Each request arrives as a fax or PDF bundle: clinical notes, labs, a form. A nurse reviewer reads the bundle, decides whether it meets the plan’s medical policy or InterQual/MCG criteria, and either approves it or routes it to a physician reviewer for a possible denial.

Turnaround SLAs are tightening under CMS interoperability and prior-auth rules. Nurse reviewers are expensive and scarce. Every denial has to cite the exact criteria that were not met, or it gets overturned on appeal. The target: an assistant that determines criteria met or not-met with a documented rationale, auto-approves the clear cases, and routes the rest with a pre-built summary.

This is the story of three rounds of building that assistant — and why each round was forced by a gap the previous one could not close.

Round 1 — Conventional ML, and Why It Stalled

The first build was an approval-likelihood model: gradient-boosted trees over structured claims history, member demographics, requesting provider, service code, and TF-IDF features from OCR’d notes, trained on five years of historical approve/deny outcomes.

It worked as a triage sort order and nothing more. Here is where it broke:

1. 1.It predicted what the plan historically did, not whether criteria are met. Historical decisions carried reviewer inconsistency and old policy versions, so the model learned drift.
2. 2.No explainability at the criteria level. A denial letter needs “Policy X, criterion 2.b: no documented trial of conservative therapy ≥ 6 weeks.” A feature-importance chart is not that.
3. 3.Each policy update — they change quarterly — shifted the ground truth under the model.
4. 4.Regulators and the medical director were uncomfortable with a score-based approach influencing denials. It looked like the pattern that has drawn lawsuits against payers. Legally, only a clinician can deny, and the tool had to support that decision, not shortcut it.
5. 5.The OCR’d note features were far too shallow to detect things like “failed physical therapy” or “documented neurological deficit.”

The ML model could tell you what the plan would probably do. It could not tell you why, against which criteria, with what evidence. That is a different problem entirely.

Round 2 — LLM-Based Agentic Design

We reframed the problem: read the clinical bundle, find the governing policy, evaluate each criterion, document the evidence. This became a multi-agent pipeline on Azure AI Foundry with a frontier model, broken into four agents:

1. 1.Intake agent — classifies the request by service, urgency, and line of business; OCRs and de-noises the fax bundle; builds a structured clinical summary with page citations.
2. 2.Policy retrieval agent — identifies the applicable medical policy version by service code and effective date, then pulls the criteria tree from a versioned policy store via RAG over the policy corpus.
3. 3.Criteria evaluation agent — walks the criteria tree and for each criterion returns met, not met, or insufficient documentation, with the exact evidence span from the notes.
4. 4.Disposition agent — applies plan business rules: all criteria met means auto-approve; anything not met or insufficient routes to a nurse with a pre-drafted summary. It never emits a denial itself.

Approval-side accuracy climbed into the low eighties on criteria-level agreement with nurses. The real win was upstream of the number: reviewers now started from a filled-in criteria checklist with evidence links instead of a forty-page PDF. Handle time dropped materially.

But it hit a ceiling.

PHI, residency, and audit

Full clinical bundles flowing to a third-party model, even under a BAA, made compliance nervous. They also wanted a frozen, reproducible model version for every decision so they could defend it years later in an appeal or audit. Frontier APIs deprecate models on their own schedule.

Cost and latency

Multi-page bundles times four agent hops ran one to two dollars per request at 1.5 million requests a year. Urgent requests needed an answer in minutes, not the sixty to ninety seconds the chain was taking.

Accuracy plateau on the hard part

The errors lived in criteria evaluation: the general model mistook “recommended PT” for “completed PT,” missed that a lab value was from fourteen months ago and outside the policy window, and read specialist shorthand inconsistently. Larger prompts with more rules made it less consistent, not more. This was a behavior problem, not a knowledge problem.

Round 3 — Fine-Tuning, and Why Open Weights

The organization needed a model that evaluates clinical evidence against criteria the way its nurses do, on infrastructure it controls, at a version it can pin forever. That is a fine-tuning problem with an open-weights answer.

1. 1.Inference runs inside the plan’s own Azure tenant on a private GPU pool. PHI never leaves, and the model version is frozen and archived alongside each decision record.
2. 2.A fine-tuned 8B model in the evaluation and intake slots replaces the frontier model where ninety percent of the tokens are spent. Cost drops roughly ten times; latency falls to a few seconds.
3. 3.No dependency on a vendor’s deprecation roadmap for a regulated, defensible decision pipeline.

The dataset the agentic pipeline generated

After six months in production, Round 2 had accumulated roughly sixty thousand nurse-reviewed requests. For every criterion, the system recorded the model’s call, the nurse’s final call, and the evidence span the nurse accepted or substituted. Nurse corrections became gold labels.

This is the key move: historical decisions were unusable as training data, but the agentic system generated criteria-level, policy-aligned labels as a byproduct of doing the work.

The dataset was de-identified via a PHI scrubber, stratified across policies so low-volume services were not drowned out, and policy-version-stamped on every example.

Model selection and training

Model candidates were Llama 3.1 8B and Qwen 2.5 7B, with a 70B as a ceiling reference. Base-model evaluation ran first on a three-thousand-request held-out set. Training used QLoRA via Axolotl on Azure ML GPU compute — two A100s, hours per run. Two adapters: one for intake summarization with page citations, one for criteria evaluation with evidence spans. Prompts were identical to the agent prompts so the tuned model dropped into the same pipeline slots. Policy text stays in retrieval, not in the weights — that is what lets a quarterly policy update ship without retraining.

Evaluation and deployment

Evaluation covered per-criterion agreement with nurse decisions, evidence-span overlap, auto-approve precision — the number compliance cares about most, because you must never auto-approve something a nurse would have flagged — and a blinded nurse preference review. Fairness checks ran across lines of business and demographics, documented for the model governance committee.

Merged adapters were served with vLLM on a private AKS GPU pool behind APIM. The disposition rules engine stayed unchanged. The pipeline ran in shadow mode against the frontier version for a quarter before medical director sign-off and cutover. Every decision record stores model version, policy version, prompt hash, and the evidence cited.

The feedback loop

Nurse corrections and appeal overturns flow back into the dataset. Monthly re-tunes keep the model current. Policy-store updates are independent of the model — new criteria appear in retrieval without retraining.

What the Three Rounds Prove

Criteria-level agreement reached the low nineties. Roughly sixty percent of routine requests auto-approved with near-zero inappropriate approvals. Urgent turnaround measured in minutes. Per-request cost measured in cents. And a decision record that survives appeal and regulatory audit.

But the numbers are less interesting than the arc.

The ML model predicted history and could not explain itself. The frontier agent could reason against policy but could not be trusted with the data, the bill, or the audit trail. Fine-tuning an open model on the agent’s own nurse-corrected output delivered the accuracy, the cost, and a pinned, defensible model the plan actually owns.

Each round was not a failure replaced by something better. Each round generated the asset that made the next round possible: the ML model proved the volume justified automation; the agentic pipeline proved criteria-level reasoning was feasible and, more importantly, produced the labeled dataset no one could have built from scratch; the fine-tuned model turned that dataset into something the organization could own, audit, and defend.

That is the pattern worth remembering. In regulated domains, the path to a production model you can stand behind is rarely a straight line from data to deployment. It is a sequence of systems, each one generating what the next one needs.