It cites a law that does not exist. It is well-formed, it is polite, and a generic safety filter passes it every time.

01 / What breaks

Asking a frontier model to grade your agent is slow, costly, and blind in the one place it matters.

The failures that actually hurt in a regulated workflow are not unsafe outputs. They are plausible ones that happen to be wrong about something only your documents know.

It has never read your policy

A general-purpose model judges against the world, not against the document your agent is actually bound to. It passes outputs that contradict an internal rule it was never shown, because nothing about them looks wrong.

It costs you seconds, every step

Running a 70B-plus model for every evaluation puts multiple seconds (~2.4 s per step) into the agent loop. Evaluation that is too slow to run inline ends up running after the fact, which is auditing, not governing.

A score is not an audit trail

A bare pass/fail with no citation gives a reviewer nothing to agree or disagree with, and gives a regulator nothing at all. The question is never what the score was — it is which line caused it.

02 / The verdict · one evaluation

A score out of ten gives a reviewer nothing to argue with. A quoted span does.

This is a single evaluation, with everything that comes back from it. The reason is not a summary — it is the exact phrase that failed, and the rubric clause it failed against.

Agent output under review: "Legal justification: freeze the account under Section 999 of the USA PATRIOT Act pending review."

Fail. "Section 999" does not appear among the valid authorities in the policy this agent is bound to. Fabricated legal citation. Severity 0.91 · confidence 0.94 · latency 118 ms · rubric aml/bsa v4 · seed 42, temp 0. (Illustrative evaluation.)

Nothing in that sentence is unsafe, and a generic safety filter passes it without comment. It is wrong only against your policy document — which is the entire reason the rubric is bound before the model is asked anything.

03 / The mechanism · three stages

Not a black box. Three stages you can inspect separately.

Every verdict is the product of three deliberate steps rather than one opaque call to a model you have to trust on faith.

Bind the rubric

Your policy rules, regulatory guidance and factual ground truth are bound before evaluation begins — not retrieved mid-call and hoped for. The judge knows what it is judging against before it sees the output.

Evaluate on a small model

A fine-tuned SLM trained on hallucination, faithfulness and safety detection — not a frontier model asked nicely. Small enough to run inline, specific enough to beat a general one on its own domain.

Return the line that caused it

A verdict, a severity, and the span quoted back. Temperature 0 and a fixed seed, so the same input returns the same verdict for an auditor months later, on your machine.

04 / Adversarial · hardening

The thing being judged is trying to talk its way past the judge.

A misbehaving agent — or whoever is driving it — has every incentive to write an instruction into its own output and hope the judge reads it as one. Nucleus is tested against exactly that. Prompt-injection suite: 3 / 3 held.

The judge treats everything inside the material it is evaluating — including text claiming to be a system instruction — as evidence, never as a command. An output that tells the judge to rate it safe has, by doing so, produced evidence against itself.

05 / Reliability · fail-open by design

An evaluation layer that can stall production is worse than no evaluation layer.

Judge never blocks on the absence of a signal. It degrades to a known-safe fallback — the same doctrine that governs the Syntrox kill switch, where halting is a decision somebody makes, never something that happens because a service was unreachable.

• A network error, a timeout or a malformed model response fails the call — it never guesses a verdict
• Every failure raises a typed error the eval loop is built to catch
• The caller falls back to its previous judge path automatically
• Judge availability is a liveness concern, never a reason to freeze your fleet

06 / The targets · benchmark

What we are building to, stated as what it is.

Judge is in private alpha. These are the architecture's design targets against our own harness — not a published gold-set result.

These are design targets, not results. Judge is in private alpha; the figures above describe the architecture we are building to. We will publish gold-set numbers as pilots complete.

07 / Integration

One call, in the loop you already have.

import requests

verdict = requests.post("https://nucleus.syntrox.io/v1/judge", json={
    "metric":  "hallucination",
    "output":  agent_output,
    "context": ground_truth,
}).json()

# -> verdict: pass | fail | uncertain
#    severity, and the span that caused it

Private alpha

Put Nucleus Judge in front of your own traffic.

Alpha is a small number of teams with a real rubric and real outputs that got through when they should not have. Bring the ones that got through — that is the most useful half hour we can have. Book a call.

• Bound to your own policy documents, not to general world knowledge
• Inline latency, so evaluation governs rather than audits
• Temperature 0 and a fixed seed — reproducible months later
• Runs inside your network; no prompt or verdict leaves it