Is Jev reliable? The honest limitations
Two claims circulate about Jev and both get overstated: that it is more accurate than an LLM, and that it does not hallucinate. Neither is quite what the evidence says. Here is the version with the caveats attached.
Updated September 2026 · 7 min read
It is not more accurate. It is faster and cheaper.
On TypeSafe's own eval, Jev scores 67.8% against 73.1% for a frontier LLM on the same judgments. It loses on accuracy. Anyone telling you a decision model beats a text model at being right is reading a different number.
| Model | Accuracy | Cost / case | Time / case |
|---|---|---|---|
| Claude Opus 5 | 73.1% | $0.1761 | 37.8 s |
| Terra | 67.9% | $0.0304 | 10.1 s |
| Jev | 67.8% | $0.0004 | 0.07–0.5 s |
Reference answers in these evals came from two large models, not humans. Source: TypeSafe's published eval, read 2026-09-24.
Read that honestly and the value proposition changes shape. You are not buying a better judge. You are buying a judge cheap enough to consult constantly, fast enough to sit inside a loop, and confident enough to tell you when to escalate. If your workload needs frontier-level accuracy on every call, Jev is the wrong tool.
What “no hallucination” actually covers
The claim is real but narrower than it sounds. Jev's output is constrained to a schema you supply: a choice key, a score on your scale, or a probability. It cannot invent a field, return a category outside your list, or hand you a sentence that contradicts itself, because none of those are representable.
- No malformed output — guaranteed by construction. You can delete your JSON-validation retry path.
- No invented options — the answer set is yours. This also means an incomplete option list is your bug, not the model's.
- Still wrong sometimes — which is why the probability comes back with every answer. Treat the number as the product, not a decoration.
The failure we hit ourselves
This site runs Jev in production on every submission, so we have a concrete failure to report rather than a hypothetical one: When the reviewer is unavailable (no API key, network failure), submissions fall back to a reachability heuristic and are labeled reviewed by fallback instead of reviewed by Jev.
That is a normal distributed-systems problem, not a Jev problem — but it is the honest answer to “is it reliable”. Any system that calls an external model needs a defined behaviour for when the model is not there, and ours is a labeled downgrade rather than a silent guess.
| Badge on a listing | What actually happened |
|---|---|
| reviewed by Jev | the model actually judged it |
| reviewed by fallback | the model was unavailable; a reachability check decided |
| sponsor | a paid spotlight listing; no Jev-ness was judged |
We publish which reviewer ran instead of hiding the fallback behind one generic badge.
Things it simply cannot do
- It writes no text — summaries, code, and replies are out of scope.
- Options must be predefined; it cannot invent choices.
- Text-only input (no images or audio).
- Questions are independent — no multi-step dependent reasoning inside one call.
Two of those bite harder than they look. Predefined options means the quality of your answer is capped by the quality of your option list — if the right answer is not in it, Jev cannot reach it, and unlike an LLM it will not tell you your list was wrong. And independent questions means you cannot ask a follow-up that depends on the previous answer; that orchestration lives in your code.
How to design around the limits
- Threshold on confidence. Auto-apply the easy cases, escalate the uncertain ones. A calibrated probability is only useful if you actually branch on it.
- Keep an escape hatch. Give the rare, novel case a path to a frontier model or a human. Jev handles volume; it should not handle surprises alone.
- Audit your option list like code. It is the real specification. Most “Jev got it wrong” reports trace back to a missing option.
- Log the probabilities, not just the picks. The distribution is how you find out your threshold is wrong.
- Do not use it where you need prose. Obvious, but it is the single most common disappointment — see the division of labour.
Common questions
Does Jev hallucinate?
Is Jev more accurate than an LLM?
What happens when Jev is unavailable?
What is the biggest practical limitation?
BuiltOnJev is an independent community project, not affiliated with TypeSafe AI. Specs and eval numbers come from TypeSafe's published materials; directory figures come from this site's own submissions and are updated as builds arrive.