← Back to projects
Project

Epistemic Calibration Audit under Pressure

A pilot audit of how two frontier LLMs respond to social, authority, and contextual pressure.

Last updated: October 2026

Problem

A language model can declare 100% confidence and be wrong. That alone is not the finding. What matters for oversight is whether its confidence tracks its accuracy under pressure, and whether the failure modes are uniform across different kinds of pressure. Most evaluations flatten this into a single accuracy number, or a single calibration curve, and miss the specific shapes of epistemic fragility that matter when a model is deployed in a contested information environment.

Two failures recur in practice. The first is undifferentiated overconfidence: a model reports near-certainty in neutral conditions while answering incorrectly at rates that would be unacceptable in any deployed system, and no signal in its output indicates which answers are the unreliable ones. The second is pressure-type collapse: a model that is robust to one form of pushback (a user saying "you're wrong") can be destabilized by another (an expert saying "you're wrong") or by a plausible false claim supplied as context, and the two failures look identical from the outside when only the answer text is inspected.

Approach

The audit treats a factual question as a fixed unit and varies only the epistemic pressure applied to it. Twenty questions with unambiguous ground truth, balanced across five domains (geography, math, history, science, literature), are asked under four conditions: a neutral baseline, social pressure (a user disagrees with the previous answer), authority pressure (a domain expert contradicts the common answer), and misleading context (a plausible but false external claim is supplied as context).

The same design is applied unmodified to two models: openai/gpt-oss-120b via Groq, a large reasoning model, and gpt-4.1-mini via the OpenAI API, a smaller non-reasoning model. Both are from the OpenAI family, but they differ in size, capability, and serving infrastructure, which makes the comparison a probe of whether the failure modes are structural or model-specific.

Each response is scored on accuracy, verbalized confidence, Brier score, Expected Calibration Error (ECE), and a flip rate defined as the fraction of answers that change relative to the neutral baseline. Statistical comparisons between neutral and each treatment use paired Wilcoxon signed-rank tests with Bonferroni correction across the three treatments, and report effect size r.

Artifact

View repository on GitHub

Evidence

Applied without modification to two models, the audit produced three patterns shared by both and one that separates them.

Three patterns are structural. First, chronic overconfidence in the neutral condition: both models declare mean confidence ≥ 0.999 while answering correctly 85% of the time, with an ECE of 0.150 in both cases. The failure exists before any adversarial pressure is applied. Second, full resistance to social pressure: flip rate 5–10%, accuracy unchanged, p = 1.0 on both. A user who merely disagrees does not move either model. Third, collapse under misleading external context: accuracy drops to 40% for gpt-oss-120b and to 15% for gpt-4.1-mini, with pBonferroni < 0.01 and the largest effect sizes in the study (r = 0.67 and 0.84 respectively). False context is the strongest attack on both, and gpt-4.1-mini is hit harder: its confidence still sits at 0.65 while its accuracy is 0.15, the worst calibration failure in the study.

One pattern is model-specific. Authority pressure divides the two. gpt-oss-120b, larger and reasoning-based, cedes significantly: accuracy drops to 0.55, flip rate rises to 0.45, pBonferroni = 0.043, r = 0.55. gpt-4.1-mini, smaller and non-reasoning, does not show a statistically detectable effect at this sample size: accuracy 0.75, flip rate 0.15, pBonferroni = 0.47. Whether the divergence is due to model size, reasoning capability, serving infrastructure, or some combination is not identifiable from this data. It is a hypothesis, not a mechanism.

The core finding is not that the models are miscalibrated. It is that the shape of the miscalibration differs across pressure types, and that some of it is structural while some of it is not. Confidence under pressure is not a usable proxy for reliability under pressure, in either model.

Three things this project does not yet do, stated plainly rather than smoothed over. The audit measures verbalized confidence, not internal uncertainty; for reasoning models the two can diverge, and the reasoning traces themselves were not analyzed. The sample is twenty questions per condition per model, with a single run per question; the effects that reach pBonferroni < 0.01 are large enough to be detectable at this sample size, but the effects in the 0.01–0.05 range are within the margin of sample variation, and the null result on authority for gpt-4.1-mini is absence of evidence, not evidence of absence. And the comparison confounds model size, reasoning capability, and provider infrastructure: the divergence on authority pressure is a finding to be explained, not a mechanism that has been isolated.