← Back to projects
Project

Aletheia — Plurality Audit for Synthetic Governance Threads

Epistemic review aids for mitigating apparent plurality in synthetic deliberation.

Last updated: October 2026

Problem

A governance thread can contain many accounts and comments while expressing far fewer conceptually distinct positions. Apparent plurality can therefore be inflated: a reviewer sees many voices and may infer broad independent agreement when several comments are paraphrases of the same underlying position.

Aletheia treats this as an epistemic measurement problem. The project defines an inflation factor, I = Papparent / Preal, and asks whether a pipeline combining LLM-based claim normalization and semantic clustering can estimate that gap on synthetic threads with controlled ground truth.

Approach

The project uses a two-stage design intended to separate exploratory calibration from confirmatory testing. Stage 1 uses approximately 30 threads per configuration to measure effect sizes and calibrate thresholds. Stage 2 uses fresh seeds and approximately 50 threads per configuration to test preregistered hypotheses against those frozen thresholds. The protocol and its amendments are versioned publicly before confirmatory data generation.

Synthetic threads vary the number of apparent voices (50, 200, 500), the number of conceptually distinct positions (2, 5, 10), paraphrase level, and noise ratio. Ground truth is known for every thread: the true number of positions, the true inflation factor, and the cluster assignment of each comment.

Aletheia is compared against four baselines: unique accounts, unique comments, raw-embedding clustering without LLM normalization, and keyword Jaccard clustering. The primary metric is median relative error on the inflation factor, with ARI, NMI, cluster purity, and paraphrase-grouping F1 as secondary measures.

Artifact

View repository on GitHub

Evidence

The Stage 1 pilot, using gpt-4o-mini as the claim normalizer, reduced median relative error on the inflation factor by 35.71% relative to the best baseline in the initial configuration and improved median ARI from 0.51 to 0.87.

A subsequent sweep showed that the result is conditional on the inflation regime. At I = 6, Aletheia reduced median relative error by 35.71% versus the best baseline; at I = 3, the improvement was 76.89%. At I = 15, however, Aletheia performed worse than the best baseline by 20.00%. The observed failure mode is over-fragmentation: when many paraphrases express only a few underlying positions, the normalizer splits them into too many clusters.

A merge-oriented prompt reduced the predicted clusters in the difficult I = 15 condition from a mean of 5.13 to 4.50 and improved median ARI from 0.70 to 0.78, but did not solve the problem. The project therefore treats applicability as bounded by the inflation regime rather than presenting the intervention as a general fix.

The current protocol narrows the confirmatory claim accordingly: H1 is tested separately for I ∈ {3, 6} and I = 15, with both positive and negative outcomes reported.

What this project does not yet do

Stated plainly rather than smoothed over. The evaluation is synthetic-only, with controlled ground truth. It does not establish that the method generalizes to real governance threads or that human reviewers actually benefit from the tool.

The project uses a single inflation metric, I, rather than a family of competing operationalizations. It has not been evaluated against adversaries who know how the clustering pipeline works, and all experiments are currently in English.

The reported results depend on the chosen LLM normalizer. The current default is gpt-4o-mini, and the sweep has not been replicated across multiple models. The high-inflation failure also remains a genuine scope boundary: prompt changes improved fragmentation but did not eliminate it.

Aletheia is not a bot detector, not a production tool, not a benchmark or leaderboard, and not evidence that synthetic governance threads behave like real-world deliberation. Its purpose is narrower: to demonstrate a falsifiable research process for measuring apparent plurality, with explicit baselines, ablation, kill criteria, reproducibility controls, and declared limits.