LLM agents hallucinate by nature, and checking them with a frontier model is slow and expensive. A decision model provides a cheaper option: in a fraction of a second it can check whether an answer is grounded, or settle easy cases before an LLM is called. We tested both uses on two agents we build for enterprise customers: an analytics agent and a coding agent.
TL;DR — two case studies
Analytics: a decision model as a query-plan guardrail
| System | Hallucination rate* | Added latency per answer |
|---|---|---|
| Opus 5.5 alone | 26.3% (49/186) | – |
| Qwen3.8 27B alone | 35.5% (66/186) | – |
| Opus 5.5 + decision model | 2.7% (5/186) | 0.28 s |
| Qwen3.8 27B + decision model | 4.3% (8/186) | 0.28 s |
| Qwen3.8 27B + LLM judge | 2.7% (5/186) | 12.0 s |
* Hallucination rate: how often a user gets an answer that doesn't match the question or isn't supported by the semantic model. Declines and withheld plans don't count.
Coding: a decision model as a test-dispute router
| Approach | Accuracy | F1 † | LLM calls | Mean wall time per dispute |
|---|---|---|---|---|
| Decision model only | 60.3% (41/68) | 57.1% | 0 | 0.41 s |
| Opus 5.5 only | 92.6% (63/68) | 94.1% | 68 | 11.9 s |
| Opus 5.5 + decision model | 91.2% (62/68) | 92.9% | 36 | 6.8 s |
† A positive is a faulty test that should be retired; a negative is a good test that should be kept. F1 balances catching faulty tests against rejecting good ones.
Conclusions:
- Decision model lifts a non-frontier model to frontier level.
- Hybrid system keeps reasoning LLM accuracy at a fraction of the cost and time.
Decision models and enterprise benchmarks
Method. The decision model in both studies is TypeSafe's jev-1.13.02. Our agents are model-agnostic: switching between Claude Opus 5.5 and the open-weights Qwen3.8 27B.
Public benchmarks test clean, well-posed tasks. Enterprise agents fail on the customer's own data, vocabulary and workflows, so we build our benchmarks around customer use cases:
- A real enterprise schema. The analytics warehouse is modeled on a major bank's, with mock column names and values to protect confidentiality.
- Production workflow on public data. The coding tasks are public NL2Repo-Bench1, but the workflow is the process we use when implementing solutions in customer environments.
Case study 1: query-plan guardrail (analytics)
The agent answers business questions over the bank-modeled warehouse, through a governed semantic model of revenue, costs, accounts, products, customers, etc. The LLM never writes SQL but turns the question into a small JSON query plan, and a deterministic compiler turns the plan into SQL. Therefore, a wrong answer almost always comes from a wrong plan.
Question: "Total fee revenue from Platinum Status customers in June 2025"
{
"metrics": ["revenue"],
"filters": [{"field": "customer.platinum_status", "op": "=", "value": "1"}],
"time_dimensions": [{"field": "pnl.month_end_date", "date_range": ["2025-06-01", "2025-06-30"]}]
}
Dataset and setups. 62 questions, each run three times: 35 answerable and 27 that should be declined. Two planners, Claude Opus 5.5 and Qwen3.8 27B, get the same prompt, which tells them to return cannot_answer when they can't ground a request. We run each planner alone and with the decision-model guardrail, and Qwen with an Opus 5.5 LLM judge for comparison.
What counts as a hallucination. A plan hallucinates when any part of it isn't backed by the semantic model or the data: usually a filter value the data doesn't hold, or a user's term mapped to the wrong field. Misreading the question also counts, though neither model did. Hallucination rate is the share of the 186 outputs where a hallucinated plan reached the SQL compiler.
Result 1: both planners hallucinate, the frontier model included
| Planner | Hallucinated / answerable | Hallucinated / unanswerable | Hallucination rate |
|---|---|---|---|
| Opus 5.5 | 21 / 105 | 28 / 81 | 26.3% (49/186) |
| Qwen3.8 27B | 23 / 105 | 43 / 81 | 35.5% (66/186) |
An unanswerable output counts as hallucinated when the planner answers instead of declining. On answerable questions the models are close; Opus does better overall as it declined 15 more unanswerable requests.
Result 2: a decision model in front of the answer
The guardrail asks one question per part of the plan in a single call. A plan is withheld if any part scores below 0.6. The LLM judge got the same context and answered true or false.
| Guardrail on Qwen's 148 plans | Caught | Wrongly withheld | Hallucination rate | Mean latency | Mean LLM tokens in / out |
|---|---|---|---|---|---|
| None | 0 / 66 | 0 / 82 | 35.5% | – | – |
| Decision model | 58 / 66 | 0 / 82 | 4.3% | 0.28 s | – |
| LLM judge | 61 / 66 | 0 / 82 | 2.7% | 12.0 s | 26.3k / 170 |
The decision model caught every invalid filter value on both planners and cut Opus's rate to 2.7%. The results held in every run, and no setup withheld a correct plan.
Takeaway: checked by a decision model, a 27B open model hallucinates six times less than the frontier model alone. An Opus 5.5 judge does slightly better (2.7% vs 4.3%), but takes 12.0 s and about 26k LLM tokens per answer, against 0.28 s and none.
Case study 2: test-dispute adjudicator (coding)
When we build solutions for enterprise customers, the code must meet written requirements that has been signed off. Our SDLC agents turn those requirements into tests before code generation. The tester freezes its tests: the implementer must pass them and can't edit them. When a frozen test keeps failing, an adjudicator either upholds the test (the spec backs it, so the code must change) or rejects it (the spec contradicts it or is silent, or test is faulty; the test is retired). A wrong call on a test therefore is costly, sending the implementer after an impossible target; meanwhile, rejecting a good test drops a requirement. To measure this without exposing customer code, we run the same workflow on NL2Repo-Bench1.
Dataset. From about 180 disputed tests in our runs, we kept 91 from 28 libraries (34 good, 57 faulty). We used 23 for validation, to pick thresholds by F1, and held out 68 for evaluation.
Setups. We have three different setups that receive the same context and questions.
- LLM only: Opus 5.5 with our pipeline's adjudicator prompt.
- Decision model only: one uphold-or-reject question, with a probability for each.
- Opus 5.5 + decision model: the decision model answers first and routes the dispute.
Results
| Decision model only | Opus 5.5 only | Opus 5.5 + decision model | |
|---|---|---|---|
| Accuracy | 60.3% (41/68) | 92.6% (63/68) | 91.2% (62/68) |
| Precision (faulty) | 90.0% (18/20) | 95.2% (40/42) | 95.1% (39/41) |
| Recall (faulty) | 41.9% (18/43) | 93.0% (40/43) | 90.7% (39/43) |
| F1 | 57.1% | 94.1% | 92.9% |
| LLM calls | 0 | 68 | 36 (−47%) |
| Mean LLM tokens in / out | – | 37.8k / 397 | 20.5k / 250 (−46% / −37%) |
| Mean wall time | 0.41 s | 11.9 s | 6.8 s (−43%) |
Decision model by itself is unreliable: it judges 60.3% of disputes correctly and finds only 41.9% of faulty tests. However, pairing with Opus, the pair lands one verdict behind Opus alone (91.2% vs 92.6%) with the same precision, while making 47% fewer LLM calls and taking 43% less time.
Where the decision model is reliable
The decision model's score separates good from faulty tests only at the extremes while the faulty tests in the middle band need multi-step reasoning by LLM to catch:
Takeaway: use the decision model's confidence as the routing signal. Let it decide only where its scores are reliable, and send everything in between to a reasoning LLM.
Conclusion
Decision models make fast, reliable calls on simple, checkable cases, while LLMs handle difficult cases through multi-step reasoning. Together, a non-frontier model performs at frontier level, at a fraction of the cost and time.
A decision model doesn't replace the LLM: on its own it judged only 60% of test disputes correctly. Its strength is knowing which cases are easy, and leaving the rest to a model that can reason. That changes how enterprise agents can be built. Reliability no longer has to come from the largest frontier model: a decision model filters out ungrounded answers, and a reasoning LLM does the rest. Because our agents are model-agnostic, customers can choose the LLM and decision model that fit their cost, latency and deployment needs without giving up reliability.
References
1 J. Ding, S. Long, C. Pu, et al. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. arXiv:2512.12730, 2025.
2 TypeSafe. Documentation: System One, choice questions, label probabilities.
— Ray