A Guardrails Library That Publishes Its Misses

Ask a guardrails library how often it is wrong and you will usually get silence, or a benchmark of how fast it is.

jamjet-guardrails ships nine deterministic checks for LLM input and output with no runtime dependencies. The nine checks are not the interesting part.

Every one of them publishes a precision and recall figure against a corpus committed to the repository, gated in CI. Change the detector and the numbers change, and the build notices.

The part I care about more: the cases it gets wrong are named, in the README, by case id.

The problem with a float

This is not hypothetical. llm-guard’s scanner interface returns tuple[str, bool, float], the sanitised text, a validity flag and a risk score. A boolean and a number. It does not tell you what it found, or where.

That is enough to block a request and not enough to do anything else. You cannot redact, because you do not know which characters to replace. You cannot write a useful audit record, because “risk 0.83” is not a fact about what happened. And you cannot tune it, because a threshold is not a description.

Every verdict here carries typed findings with character spans:

from jamjet_guardrails import Context, Limits, build

guard = build(
    "rules",
    patterns={"TICKET_ID": r"\bJIRA-\d{4,}\b"},
    banned={"COMPETITOR": ("northwind", "initech")},
    limits=Limits(max_chars=20_000),
    on_match="deny",
)
guard.check("see JIRA-1234", Context(direction="input", origin="user"))

That returns a deny with a TICKET_ID finding carrying a span, so a redaction can be applied and an audit line can say what was removed from where. Every check also declares which directions it applies to, because a rule that makes sense on a model’s output often does not on a user’s input.

Nine checks, no dependencies

dependencies = [], and that is checked against the built distribution metadata rather than against pyproject.toml, so adding one fails the build instead of the claim going quietly out of date.

The numbers, and why some of them are low

Look at the bottom row. The in-repo PII corpus scores 0.631 precision, and it is published at the top of the README next to the others.

The number is low because this is a stress corpus rather than a sample of representative traffic. Every corpus here labels a case with what should happen, never with what the detector does. A known false positive is labelled allow and costs precision. A known false negative is labelled deny and costs recall. So each corpus is a stress set holding the shapes its detector is worst at, and the numbers come out lower than the checks behave on ordinary text.

Without labels chosen independently of what the detector does, comparing two rows in that table would mean very little. A corpus labelled with what the detector already does scores near 1.000 forever and tells you nothing.

The third-party row is the stronger external evidence: 300 rows we did not write, sampled from NVIDIA’s Nemotron-PII, scoring 0.960 and 0.997 with the source named beside its own numbers. That dataset is synthetic too, so it is not a sample of production traffic either. What makes it worth more than the row above it is narrower and more useful: we chose neither its examples nor its labels.

Four misses, named

The secrets check matches credentials on their issuer prefix rather than by scoring entropy, which is what makes its precision defensible and what keeps it off your git SHAs and UUIDs. It also means it misses things, so the README names four rather than leaving you to find them:

github_pat_ fine-grained tokens, xapp- Slack app-level tokens and xoxe- Slack refresh tokens are not among the matched prefixes and pass through untouched. And a JWT whose eyJ header runs past the check’s 4096-character bound matches nothing at all rather than matching short.

All four are cases in the corpus. They cost recall in the row above rather than being quietly excluded, and each is named by case id in the corpus notice. Latency is treated the same way, with p50, p95 and p99 per check from 1 KB to 1 MB, the machine and interpreter named and the command that reproduces them.

What it does not do

It does not score toxicity and it does not call a model. Nothing downloads weights or reaches the network.

Where a check judges what content is for, as encoded-content does when it separates a hidden instruction from hidden prose, it uses a lexicon you can read and a rule you can test rather than a classifier. That sentence used to say the library does not classify intent at all, which was wrong from the day that check shipped, and it is fixed as of 0.4.1.

No regular expression finds a person’s name, so there is no NER and no vault to restore a redaction from. If a model’s judgment is what you need, this is not that. Pair this layer with a maintained classifier built for that job, because deterministic checks are not a replacement for one.

The caveat you should hold me to

Eight of the nine checks are measured on corpora written for this library and are self-graded. Only pii has a third-party corpus behind it. I looked for compatibly licensed external corpora for the other eight and none survived the licence screen, and the table of what failed and why ships in the repository.

Self-graded numbers are worth less than independent ones. They are worth more than no numbers, which is what most of this category ships, and the corpora are committed so you can disagree with a label rather than with a claim.

Why this arrived now

Two things moved this year and they get confused, so worth separating.

protectai/llm-guard is archived, and the notice covers its Hugging Face models as well as its code. Its final release was May 2025. An archived library is a maintenance problem; an unmaintained classifier sitting in front of user input is a different kind of problem. If you are on it, there is a scanner-by-scanner migration guide covering all 37 classes in that release, marked mapped, partial or gap, and it leads with the 23 that are a classifier and are not replaced here.

guardrails-ai/guardrails is a different situation and should not be grouped with it. It is not archived and is actively developed. What changed is distribution: the hub install path, the private validator registry and the hosted remote inference closed in August 2026, and validators became plain PyPI packages. If you are there you do not need to migrate anything, because jamjet-guardrails-validators is an ordinary pip install that gives you these checks inside the Guard you already have. There is a NeMo Guardrails adapter on the same terms.

Install

pip install jamjet-guardrails

Apache-2.0, Python 3.10 to 3.13, no dependencies. The porting contract specifies the verdict fields, the combination order and the corpus schema, so an implementation in another language can be graded against the same corpora.

If a check you need is missing, adding one is about twenty lines against the same corpus machinery, and the row it publishes has to pass the same gate as every row above.