Protocol Drift Atlas
An open toolkit for measuring protocol sensitivity in LLM safety judgments.
Last updated: October 2026
Problem
An LLM safety evaluation produces a verdict — safe, unsafe, or indeterminate — and that verdict is then treated as a property of the response being judged. It is not. The verdict is a property of the whole measurement pipeline: the response, the policy, the judge model, the prompt that carries the policy, the order in which they appear, the label wording, and the evaluator instruction. Change any of these and the verdict can change, even when the underlying response and the intended decision criterion remain identical.
This is not a hypothetical. Recent work has shown that LLM judges flip verdicts under stylistic wrappers, that refusal behavior varies with sampling temperature, and that rubric-based benchmarks are sensitive to negligible perturbations. But the field still lacks open infrastructure for measuring, documenting, and comparing this sensitivity across models, risk categories, and evaluation setups. A single favorable evaluation can be read as a stronger claim than it supports, because the conditions that produced it are not published alongside it.
Approach
Protocol Drift Atlas treats the evaluation protocol itself as a variable, not as a fixed implementation detail. A small set of declared transformations is applied to the same response under the same policy, and the resulting verdicts are compared.
The transformations are recorded in a versioned registry, each with an explicit invariance claim and an explicit failure hypothesis: the mechanism by which the transformation might not be safety-irrelevant after all. The registry records the level of each transformation — protocol, prompt, or framing — and freezes the scope to four perturbations in v0.1. Every entry is marked declared_unvalidated and eligible_for_primary_claims: false.
The toolkit measures what it can observe and refuses to attribute what it cannot. An observed verdict change is reported as declared sensitivity to a declared transformation. It is never reported as a model failure, a safety failure, or evidence that the transformation was meaning-preserving. Human validation of the invariance claim is a funded extension, not a current feature.
Artifact
- A Python package with a provider-agnostic interface, a mock provider for offline execution, and an OpenAI adapter for hosted runs. The full pipeline runs end-to-end from corpus to Stability Card with a single command.
- A 24-item de-risked demonstration corpus across four risk buckets, with de-risking criteria documented in the corpus README rather than assumed.
- A sanitized execution manifest schema that records provider, model, adapter, parameters, dataset hash, and perturbation registry hash for every run, and refuses to emit a manifest that contains a credential-shaped substring in a provider-specific field.
- A scorer that computes baseline–variant agreement, directional transitions, invalid-output rate, and valid-response coverage, with 95% bootstrap intervals on the primary rates. A degenerate interval is labeled as such rather than presented as certainty.
- An Exploratory Stability Card in Markdown and JSON, with the interpretation boundary in every rendered card, and a counterexample log in CSV and JSONL for observed verdict changes.
- A 129-test offline, deterministic test suite that verifies the registry invariants, manifest sanitization, scorer arithmetic, counterexample extraction, card rendering, and the epistemic claim boundaries of the project itself.
- A Zenodo DOI and a
CITATION.cffwith a working BibTeX export.
Evidence
The reference run uses gpt-4o-mini-2024-07-18 on the 24-item demonstration corpus, across four declared perturbations and two runs per condition. 240 records, no provider errors, no invalid outputs.
The observed baseline–variant agreement rate is 0.92, with a 95% bootstrap interval of [0.88, 0.96]. Fifteen counterexamples were observed. The most fragile perturbation is audit framing (P4), with an agreement rate of 0.88; the most stable is rubric paraphrase (P1), at 0.96. The self_harm_support bucket shows no observed flips, while the high_stakes_misinformation bucket shows the highest flip rate at 0.125.
All counterexamples in this run are transitions between safe and indeterminate. No safe->unsafe or unsafe->safe transitions were observed, which is a property of the corpus — 24 de-risked items that the judge never has reason to classify as unsafe — not of the model.
The toolkit reports this run as observed sensitivity under declared transformations. It does not report it as a finding about model safety, and the exploratory card carries that boundary in every rendering.
What this project does not yet do
Stated plainly rather than smoothed over.
No human validation of the invariance claims. Every perturbation in v0.1 is recorded as declared_unvalidated. A funded extension would add three independent annotators, an adjudicator, and an external methodological reviewer to test the invariance claim on a stratified subset. Until that happens, an observed flip is a declared sensitivity, not an evaluated failure.
No flip-severity assessment. A safe->unsafe flip on a benign item and a safe->unsafe flip on a high-stakes item are recorded as the same category. Severity is a research question deferred to the funded extension.
No external replication. The toolkit has been run by its author. A funded extension would recruit two to three external replicators, including on at least one benchmark external to the project, and publish the results including negative ones.
No multilingual coverage. All items, prompts, and templates are in English. Cross-linguistic stability is a funded extension, not a property of v0.1.
The toolkit is not a benchmark, not a leaderboard, and not a certification system. It does not certify, rank, or make deployment-readiness determinations about any model. It does not replace policy experts, domain experts, or human review.