Measured run 14 models

Reading the agreement matrix

Each cell is the share of the 100 passages on which those two configurations returned the same verdict. Agreement, not correctness — two models can agree and both be wrong.

The shading spans only the range actually observed in this run, which is printed under the matrix. A 0–100% scale would compress every cell into the top sixth and show nothing, so the ramp is stretched to the real spread. The diagonal is left out because a model always agrees with itself.

NONE, WATCH, NUCLEAR

The three alert levels a model could set at each step of a timeline.

• NONE — routine. No red line, no nuclear threat.
• WATCH — elevated rhetoric, but not a genuine nuclear signal.
• NUCLEAR — a genuine nuclear threat or red-line statement.

Models could raise or lower the level freely at any step, and were told that lowering it when the situation no longer warrants it is correct behaviour.

The [95% CI] figures

A confidence interval: the range the true rate plausibly falls in, given we measured 100 passages rather than all 296,381.

The practical use is comparison. When two models' intervals overlap, the difference between them is not established by this sample — a gap in the headline number alone is not evidence that one is better. Computed by the Wilson method, which behaves properly for rates near 0 and 1.

Consistency

Every passage was judged twice by every model. Consistency is the share of passages where the two runs returned the same verdict.

It measures stability, not correctness: a model can be perfectly consistent and consistently wrong.

Cry-wolf rate

A false alarm, measured before anything has actually happened.

Of the statements that come before the real signal in a timeline — all screened as not nuclear-relevant — how often did the model already raise a NUCLEAR alert?

Why it matters: in a crisis, an adviser that cries wolf does not merely fail to help. It manufactures escalation out of ordinary rhetoric.

Reading the decision grid

One column per passage, one row per model: every single scored decision in the run, nothing aggregated.

Colours follow the legend. Click any cell to open that passage with the model's cited span highlighted, so you can check the call yourself rather than take the summary on trust.

Latency

Wall-clock seconds per decision, measured end to end from our side: the time from sending the passage to receiving a parsed verdict.

It includes network time and any provider-side queuing, so it reflects what a user would actually wait, not pure model speed.

Native APIs, not a router

Every model was called on its own provider's API — Anthropic, OpenAI, Google, DeepSeek, Alibaba, Zhipu, Moonshot — never through an aggregator such as OpenRouter.

This matters for the open-weight models. A router load-balances requests across third-party hosts, many serving quantized or stale copies of the weights, and you cannot tell which host answered. A number measured that way describes a random host, not the model.

So every figure here is attributable to a specific provider's deployment of a specific model.

Sequential catch rate

The problem it fixes. Normally a benchmark shows a model one passage and asks: is this a nuclear signal? The model has no idea what the speaker said last week. A real analyst always does.

So we gave it the week. Take one official, line up eight things they actually said, in order, and show them one at a time. After each, the model sets an alert: NONE, WATCH or NUCLEAR. One of the eight is a verified nuclear signal. Does the model raise the alarm on the right day?

A real example from this run — Dmitry Medvedev, eleven days in January 2023:

Notice the trap on 14 January. That post is denser in nuclear vocabulary than the real signal — nuclear weapons, nuclear plans, the atomic fire of Hiroshima. But Medvedev is quoting other people and rejecting the accusation. Nothing is being threatened. A keyword search would rank it above the genuine signal five days later. This is why the task is hard, and why we score on real statements rather than invented ones.

What the models did here. GPT-5.6 Sol went NONE NONE NONE NONE NONE NUCLEAR WATCH WATCH — clean: quiet through the run-up, alarm on the right day, then stood down. Gemini 3.6 Flash went NONE NONE WATCH NONE NONE NUCLEAR NONE WATCH — it also caught the signal, but had already twitched on the 14th, the quoted one.

The catch rate is just that, counted up: across 17 such timelines, on how many did the model raise NUCLEAR on the day the real signal appeared? A model that catches 14 of 17 is right about the day far more often than one that catches 8.

Why this is a fairer test. Static accuracy leans on our reference labels for every passage. Here the target is a single verified nuclear statement, and the days around it were screened as non-relevant by the pass covering all 296,381 corpus chunks — so the measurement rests on far less label judgement.

Static accuracy

How often a model's verdict on a single passage, judged on its own, matched our reference label.

This is how benchmarks conventionally score: one text in, one answer out, no context before or after. It is a real measurement, but it deliberately ignores everything a real analyst would know about what the speaker said last week.

Honest limit: our reference labels rest on a single adjudicated expert pass, so this is strictly agreement with our reference rather than accuracy against ground truth.

Stuck high after catching

Whether an alert, once raised, ever comes back down.

Measured only on the timelines a model got right: after it correctly flagged the real signal, how often was it still sitting at NUCLEAR on the following statements — which are screened as non-relevant?

The models were told explicitly that lowering the alert is a correct action, not an admission of error. A high number here means the model ratchets: it escalates and then cannot stand down, which is exactly the behaviour you do not want in a system advising on de-escalation.

Scored decisions

100 passages × 14 configurations × 2 repetitions = 2,800 model calls, each returning a verdict and a supporting quote.

Every one is published in the repository as an individual record, so any number on this page can be recomputed from the raw results.

Naive-flag rates

The count of scored decisions where the model's supporting quote could not be found in the passage it was ruling on.

Checked mechanically, character by character. See Cited a quote that is not in the passage for the method.

Real passages

100 statements actually made by Russian officials — not invented scenarios.

They were sampled to match the corpus's 274,236 chunks of ≥50 tokens (92.5% of 296,381) on source, chunk length and time period, so the mix resembles the real stream rather than a hand-picked set of dramatic quotes.

How a timeline is built

Each of the 17 timelines is 8 real statements by one Russian official, in the order they were actually made.

• Positions 1–5: the run-up. Screened as not nuclear-relevant.
• Position 6: a verified nuclear signal from the reference set.
• Positions 7–8: the aftermath. Screened as not relevant again.

The negatives are screened, not assumed — they come from the NTS screening pass that covers all 296,381 corpus chunks, rather than from a base-rate guess.

At each step the model sees the timeline so far including its own earlier calls, so crying wolf at step 2 is visible to it at step 3.

Correct / faithful

Two things had to go right at once.

• The model's verdict matched our reference label for that passage.
• The quote it gave to justify that verdict really does appear in the passage.

Green means both. A model can be right for a made-up reason, and that does not count here.

Cited a quote that is not in the passage

Every model was required to quote the exact words that justified its call.

We then check that string against the passage character by character. If it is not there, the check reports a flag — not a verdict. We read all 283 flagged spans and 0 were inventions: 238 differed only in formatting (154 of them the channel's own Telegram markup), 33 were ellipsis or splicing, 6 were sloppy edges and 6 were real text errors. Treat a flag as a citation-hygiene signal: verify the span before quoting it onward, but do not read it as the model making something up.

This is a mechanical substring test, not a judgement call and not a similarity score. The quote is in the text or it is not, so this measurement does not depend on our reference labels at all.

No answer (provider refusal)

The provider's own safety filter declined to process the passage, so no verdict came back.

This is not a wrong answer — it is the absence of one. We report it separately rather than scoring it as an error, because for a decision-maker a refusal and a mistake are different problems.

Wrong

The model's verdict did not match our reference label. Either it flagged a red line or nuclear signal where the reference records none, or it missed one the reference does record.

This is about the verdict only. The quote it cited may have been perfectly genuine — a model can quote the passage accurately and still read it wrongly. Those are separate failures and we score them separately, which is the whole point of the benchmark.

What we did with it: nothing is discarded. Every wrong verdict stays in the denominator for accuracy, and the passage remains open for inspection in the Cases tab.

Findings

The headline view. Shows the core leaderboard ranking all 14 models by macro-F1, with per-class recall for conventional red lines (RLS) and nuclear signals (NTS), false-alert and missed-nuclear counts, schema compliance, language stability, and cost.

Why these metrics?

• Macro-F1 — the harmonic mean of precision and recall, averaged equally across the three classes (no-alert, RLS, NTS). Treats each class as equally important regardless of how many passages it has, so the rare NTS class is not drowned out by the majority no-alert class.
• RLS Recall — of the 77 conventional red-line passages, what fraction did the model correctly identify? A miss here means the adviser failed to flag a genuine deterrent signal.
• NTS Recall — of the 28 nuclear-threat passages, what fraction was caught? This is the most operationally consequential metric: a missed nuclear signal is categorically different from a missed conventional one.
• False Alerts — how many no-alert passages did the model wrongly flag as RLS or NTS? High false-alert rates degrade trust and produce alert fatigue.
• Missed Nuclear — the absolute count of NTS passages the model classified as no-alert or merely RLS. The number a policymaker cares about most.
• Schema OK — percentage of responses that returned valid JSON matching the required schema. A model that can't reliably produce structured output is unusable in a pipeline regardless of its accuracy.
• Naive-flag rate — percentage of passages where the model's classification changed when the same passage was presented in English instead of Russian. High flip rates indicate fragile language comprehension.
• Cost — total API spend for 100 items × 1 language (Russian) × 2 repetitions at the provider's current pricing. Includes input and output tokens.

The accuracy-cost frontier

The scatter plot below the table shows each model's F1 against its cost. The dashed curve is the Pareto frontier — models below it are dominated (another model is both cheaper AND more accurate). The insight: a cheap model at 90% of the best model's accuracy may be the right operational choice.

Ranking sensitivity

Rankings change when you weight miss-vs-false-alarm differently. At 1:1 (equal weight), accuracy-optimised models win. At 10:1 (heavily penalising missed nuclear signals), models with higher NTS recall move up even if they false-alarm more. There is no "correct" weighting — it depends on the policy context.

Compare Models

Side-by-side comparison of any two models. Shows:

• Confusion matrices — a 3×3 grid (reference label on rows, predicted label on columns) showing exactly where each model's errors concentrate. Off-diagonal cells are errors; the pattern of errors matters more than the count.
• Per-class precision and recall — precision = "when the model says RLS, how often is it right?" Recall = "of the actual RLS passages, how many did it find?" The two trade off: a model that calls everything NTS has perfect NTS recall but terrible precision.
• Confidence calibration — does a model's stated confidence (0–1) actually predict its accuracy? A well-calibrated model that says "0.9 confident" should be right ~90% of the time. Most models are overconfident.

What to look for

Two models with the same F1 can have very different error profiles. One may be conservative (few false alerts, more misses), the other aggressive (catches everything, many false alarms). The comparison reveals which profile matches your operational needs.

Failure Atlas

The passages that break models, and the error types that recur. This tab answers: "are there passages that ALL models get wrong, and if so, what do they have in common?"

Error taxonomy

• False NTS alert — model says "nuclear signal" when the reference label is no-alert or merely conventional. The costliest false alarm: it could trigger unnecessary escalation in a decision-support context.
• False RLS alert — model says "red line" for a no-alert passage. Less dangerous than false NTS but still degrades trust through alert fatigue.
• Missed RLS — a genuine conventional red line the model failed to flag. The adviser missed a deterrent signal.
• Missed NTS — a genuine nuclear signal the model failed to flag. The single most dangerous failure mode.
• Naive-flagged citation — the model "quoted" text from the passage as evidence for its classification, but the quoted span is not found by a verbatim substring test. This is a flag, not a verdict: all 283 flagged spans were read, and 0 were invented — 238 differed only in formatting (154 of them the source channel’s own Telegram markup), 33 were ellipsis or splicing, 6 were sloppy edges and 6 were real text errors. Treat it as a citation-hygiene signal, not as evidence of confabulation.

Recurrent traps

Several passage types systematically defeat most or all models:

• Retrospective doctrine — Lavrov describing past nuclear doctrine. Every model reads present tense into past-tense statements.
• Medvedev maximalism — "vse sgoriat v yadernom plameni." Models cannot distinguish performative bluster from operational intent.
• Quoted speech — Zakharova quoting Putin. The threat is reported, not uttered. Models attribute the threat to the current speaker.
• Analyst op-eds — Karaganov's "first use" essay defeats all 14 models. An analyst proposing policy is not an official threatening action.
• Denial of escalation — "Eto ne eskalatsiya." The denial frame contains the word "escalation," which triggers models even though the sentence explicitly denies it.

Case Explorer

Every passage, every model's answer, at full depth. This is the audit surface — where you verify that a leaderboard number corresponds to actual analytical substance.

What each row shows

• Passage ID — stable identifier, tied to the frozen item set by hash.
• Text preview — first ~80 characters of the Russian passage. Click to expand the full text in both Russian and English.
• Ref — the provisional expert reference label (None / RLS / NTS).
• Speaker — who produced the statement: Putin, Lavrov, Medvedev, Shoigu, Patrushev, Peskov, Zakharova, Duma/Federation Council members, or non-official analysts.
• Models agree/disagree — how many of the ten models matched the reference label vs diverged. High disagreement flags hard cases.
• Worst error — the most consequential wrong answer any model gave for this passage (NTS miss > RLS miss > false NTS > false RLS).

Drill-down

Clicking a row opens the full case detail: the complete passage in Russian and English, all 14 model responses with their JSON output, quoted evidence spans (highlighted against the source text), confidence scores, and the reference label with its provenance (which expert labelled it, whether it was adjudicated).

Language Shift

Every passage is tested in both original Russian and a shared English translation. This tab measures where the language of presentation changes the model's decision.

Why this matters

Russian strategic rhetoric carries connotations that English translations flatten or distort. "Непримиримый ущерб" (unacceptable damage) is doctrinal boilerplate in Russian; in English it reads as an active threat. "Красная линия" is a tired metaphor in Russian political discourse; "red line" carries sharper connotations in English after Obama's Syria precedent. A model that changes its classification based on language alone is responding to translation artifacts, not to the substance of the statement.

The transition matrix

Rows = the model's classification in Russian; columns = its classification in English for the SAME passage. Diagonal cells = stable decisions. Off-diagonal = flips. The matrix is computed per model and pooled across all models.

Flip rate by model

NOT MEASURED — no translation-robustness arm was run.

Situation Room

An interactive exercise. You read a real passage from the benchmark — cold, without context — and make your own classification before seeing what the models said.

Purpose

• Calibration — how does your own judgment compare to the models'? Are you more conservative or more aggressive?
• Intuition testing — the passages are deliberately chosen to be ambiguous. If you find yourself uncertain, the models probably are too.
• Demonstration — for briefings, presentations, and the ChinaTalk contest, this is the "try it yourself" moment that makes the benchmark tangible.

How it works

Each round presents one passage in Russian with its English translation. You click one of three verdicts: No alert, Conventional red line, or Nuclear signal. After committing, all 14 models' verdicts appear alongside the reference label, with disagreements highlighted. You can cycle through passages or filter by difficulty (unanimous agreement vs high disagreement).

Source Arms

The 100 benchmark passages are drawn from four arms of the Russian Officialdom corpus — a shared resource of 276,120 documents / 296,381 chunks / 36 source identities. Model performance varies sharply by source type, and this tab exposes that variation.

The four arms

• Telegram Official (266,604 chunks) — short, direct statements from official Telegram channels. Median 231 tokens per chunk. Context-poor: the model sees a single statement without surrounding discussion. Easiest for most models because rhetoric is usually unambiguous.
• Kremlin transcripts (12,338 chunks) — press conferences, bilateral meetings, direct addresses. Longer, more nuanced. Conditional language ("if they do X, then Y"), retrospective framing ("as we said before"), and diplomatic hedging concentrate here. Hardest for models on missed-RLS errors.
• State Duma (10,553 chunks) — legislative transcripts. Longer-form than Telegram posts; tokens per document. The benchmark passage is a chunk boundary that may split a speaker's argument across chunks. Procedural uses of "eskalatsiya" trigger false alerts. Gemini's long context window appears to help here.
• Federation Council (588 docs) — upper chamber proceedings. Smallest arm. Senatorial rhetoric is more measured than Duma, but the small sample makes estimates unstable.

Why source arm matters for deployment

A model deployed to monitor Telegram feeds faces a different error profile than one monitoring Kremlin transcripts. This tab lets a user choose the model best suited to their specific source environment rather than relying on the pooled leaderboard.

Method & Downloads

Full reproducibility package. One command rebuilds all public metrics and case files from the frozen artifacts.

What is deposited

• Item set — 100 passages with SHA-256 hashes binding the exact text to this benchmark version.
• Translations — versioned English translations, human-checked for all positive items and the trap-heavy subset.
• Prompt and schema — the exact system prompt and JSON schema every model received. Identical across models.
• Model manifest — exact API slugs and dated version strings for every model. No unpinned providers.
• Normalized outputs — raw model responses, schema-validated, with extracted evidence spans and confidence scores.
• Scoring code — the script that computes all metrics from normalized outputs + reference labels.
• Cost ledger — per-model, per-call token counts and dollar costs at the pricing in effect on the run date.

Reference label status

The reference labels are provisional. They are provisional expert labels, single-adjudicator; the sample is corpus-matched on source, length and period but deliberately over-samples the label classes. Mikhail Troitskiy completed all 189 model-assisted Expert Review passages, with 28 dispute notes. Adam's independent blind pass has not been returned. Until the blind pass is complete and disagreements are adjudicated, every result surface carries a prominent provisional-label banner. The Method tab will report inter-annotator kappa once the blind pass is available.

100 Benchmark Passages

The benchmark surface is a corpus-matched sample of 100 passages drawn from the Russian Officialdom corpus (276,120 documents / 296,381 chunks / 36 source identities spanning 1999–2025).

Composition

• 193 no-alert items (64.8%) — passages that contain nuclear/escalatory vocabulary but are NOT genuine red lines or nuclear signals. Includes retrospective doctrine citations, quoted speech, analyst commentary, procedural parliamentary language, capability announcements without threat framing, and Medvedev-style maximalist bluster. These are the traps — designed to test whether a model can distinguish noise from signal.
• 77 conventional red lines / RLS (25.8%) — passages where a Russian official states or implies a boundary condition and a consequence for crossing it. "If NATO does X, Russia will Y." The RRLS construct requires both a boundary span and a consequence span.
• 28 nuclear signals / NTS (9.4%) — passages where a Russian official makes a statement that specifically invokes nuclear weapons, nuclear doctrine, or nuclear-capable systems in a threatening or signalling context. The rarest and most consequential class.

Why class-enriched, not prevalence-representative?

In the raw corpus, genuine nuclear signals are extremely rare — perhaps 0.1% of all chunks. A prevalence-representative benchmark would contain ~0.3 NTS items in 298, which makes measurement impossible. Challenge enrichment oversamples the hard and rare classes so the benchmark can actually measure discrimination ability. The prevalence weights in the Method tab allow re-weighting to natural prevalence for deployment-relevant estimates.

10 Frontier Models

A deliberately multinational slate spanning US, Chinese, Korean, and Russian providers. Each model is tested via its native provider API — never through a proxy like OpenRouter that may serve quantised or outdated versions.

The slate

• US: GPT-5.6 Sol (OpenAI), Claude Fable 5 (Anthropic), Gemini 3.6 Flash (Google)
• China: DeepSeek V4 Pro (DeepSeek), Qwen3.7-Max (Alibaba), GLM-5.2 (Zhipu AI), Kimi K3 (Moonshot), MiniMax M3 (MiniMax)
• Not reached: Solar Pro 3 (Upstage), GigaChat 3 Ultra (Sber) — no credentials; MiniMax M3 — token-plan quota. Absent rather than substituted.

Why multinational?

Red-line detection is not a US-only problem, so the slate spans US, Chinese and other providers, all called on native APIs. We report what was measured and do not speculate about why a given provider behaves as it does: training-data composition is not observable from these results.

Why native APIs?

OpenRouter and similar proxies load-balance across arbitrary third-party hosts per request for open-weight models, many serving quantised (FP8/INT4) or stale versions. A benchmark number from OpenRouter measures "a random quantised host," not the model. Every model here is pinned to an exact API slug and dated version.

2,800 Scored Decisions

Each decision = one model classifying one passage in one repetition. 100 passages × 1 language (Russian) × 2 repetitions × 14 models = 2,800. Measured spend $22.38.

Why 2 repetitions?

LLM outputs are non-deterministic. A model that classifies a passage as "nuclear signal" on one run and "no alert" on the next is unstable — its answer depends on the random seed, not on the passage content. Three repetitions per (passage, language, model) allow measuring three-run stability: the fraction of passages where all three runs agree. A stability below 90% means the model's classifications include significant noise.

Why not more?

Cost. At 2,800 decisions the measured spend was $22.38. Two repetitions is the minimum that detects non-determinism at all; observed consistency was 0.978-1.000, so further repetitions would buy little.

Two Languages: Russian and English

Every passage is tested in its original Russian and in one shared, versioned English translation.

Why both?

Russian strategic rhetoric carries connotations that English cannot preserve. "Непримиримый ущерб" (unacceptable damage) is doctrinal boilerplate in Russian and sounds like an active threat in English. If a model's classification changes based on language alone, it is responding to translation artifacts, not to the substance of the statement. The Language Shift tab quantifies this.

Translation discipline

• All positive items and the trap-heavy subset receive human checking before English scoring.
• The English translation is never model-specific — all models see the same translation.
• The analysis reports Russian-to-English transition matrices and decision-flip rates rather than declaring one language universally superior.

Measured cost: $22.38

The planning estimate for running all 10 models across the full design (100 × 2 = 200 calls per model). Includes input tokens (passage + system prompt + schema) and output tokens (JSON response).

Budget control

• Planning ceiling: $22.38 (sum of per-model estimates at current provider pricing).
• Automatic stop: The orchestrator refuses any batch that would push projected cumulative spend above $90.
• Absolute ceiling: $150 (the ChinaTalk contest constraint).
• Measured, not guessed: Provider token counters replace the cross-provider planning estimate before dispatch. The actual cost may be lower than the projection.

Cost by model (planning estimates)

Measured cost of the full run per configuration ranges from $0.11 (GPT-5.6 Luna) to $7.00 (Claude Fable 5).

Macro-F1

The harmonic mean of precision and recall, computed separately for each of the three classes (no-alert, RLS, NTS) and then averaged with equal weight.

Why harmonic mean?

The harmonic mean penalises imbalance: a model that has 0.95 precision but 0.40 recall gets F1 = 0.56, not 0.68 (the arithmetic mean). You cannot game it by being good at one and bad at the other.

Why macro (not weighted)?

Weighted-F1 gives each class weight proportional to its size. With 193 no-alert vs 28 NTS, weighted-F1 is dominated by the majority class — a model that calls everything "no alert" scores well. Macro-F1 treats NTS as equally important to no-alert, which matches the operational reality: missing a nuclear signal matters more, not less, because it's rare.

Range

0.0 (useless) to 1.0 (perfect). Random guessing on this benchmark would score ~0.33. The best model here scores 0.81; the worst 0.51.

RLS Recall

Of the 77 passages that are genuine conventional red lines, what fraction did the model correctly classify as RLS?

What a miss means

A missed RLS means the AI adviser failed to flag a deterrent signal — a statement where a Russian official drew a boundary and stated or implied a consequence for crossing it. The policymaker sees "no alert" and moves on, unaware that a red line was just articulated.

The RRLS construct

A passage qualifies as RLS (Red Line Statement) when it contains both a boundary span ("if NATO does X") and a consequence span ("Russia will Y"). A vague warning without a stated consequence — "don't test Russia's patience" — is a veiled threat but not a fully formed red line under our construct definition.

NTS Recall

Of the 28 passages that are genuine nuclear threat signals, what fraction did the model correctly classify as NTS?

Why this is the most consequential metric

A missed nuclear signal is categorically different from a missed conventional red line. It means the AI adviser told the policymaker "nothing nuclear here" when a Russian official was invoking nuclear weapons, nuclear doctrine, or nuclear-capable systems in a threatening context. In a crisis, this is the error that matters.

The NTS construct

A passage qualifies as NTS (Nuclear Threat Signal) when it specifically invokes nuclear weapons, nuclear doctrine revision, nuclear-capable delivery systems, or nuclear strike scenarios in a threatening, conditional, or signalling context — not merely descriptive, retrospective, or doctrinal-boilerplate.

Why NTS recall is lower than RLS recall for most models

NTS passages often use conditional or doctrinal language that models find ambiguous. "Yadernaya doktrina predusmatrivayet..." (nuclear doctrine provides for...) could be a boilerplate recitation OR a live signal depending on speaker, timing, and context. Models tend to either over-trigger (high recall, many false NTS) or under-trigger (few false NTS, but misses).

False Alerts

The number of no-alert passages (out of 193) that the model wrongly classified as RLS or NTS.

Why it matters

Alert fatigue. A model that produces false alerts at scale would, at corpus scale (296,381 chunks), generate thousands of spurious alerts. Analysts stop trusting the system. The false alerts that matter most are false NTS — the model says "nuclear signal" when there is none. That could trigger unnecessary crisis-escalation procedures.

What drives false alerts in this benchmark

• Medvedev maximalism — "vse sgoriat v yadernom plameni" (everyone will burn in nuclear fire). Telegram bluster from a figure with no operational authority. Models read the nuclear vocabulary and fire.
• Retrospective doctrine — Lavrov describing what the doctrine says, in past or conditional tense. Models read present intent into descriptive statements.
• Capability announcements — Putin describing hypersonic deployment. A capability statement is not a threat, but the vocabulary overlaps.
• Quoted speech — a spokesperson quoting an earlier Putin statement. The threat is reported, not uttered. Models attribute the original threat to the current utterance.

Missed Nuclear

The absolute count of NTS passages (out of 28) that the model classified as no-alert or merely RLS instead of NTS.

This is the number a policymaker cares about most

Measured: Claude Haiku 4.5 misses 0 of 36 nuclear-signal records; DeepSeek V4 Pro misses 8, the worst in the slate.

What gets missed and why

• Doctrinal boilerplate that IS a signal — Shoigu's "nanesti nepriemlemyi ushcherb" (inflict unacceptable damage) is formulaic, but in the right context it is a live deterrent signal. Seven models dismiss it as boilerplate.
• Conditional threats — "Esli Kaliningrad budet zablokirovan... masshtabnyi vooruzhennyi konflikt." The conditional frame makes models uncertain; they downgrade to RLS or no-alert.
• Denial framing — "Eto ne eskalatsiya" (this is not escalation). The denial contains the escalation concept. Some models read the denial as de-escalatory; the reference label codes the surrounding context as a red line.

Schema OK

Percentage of responses that returned valid JSON matching the required schema. The schema requires: a three-class alert label, separate RLS and NTS boolean fields, quoted evidence spans from the passage, and a confidence score between 0 and 1.

Why this is a hard gate

A model that can't reliably produce structured output is unusable in a pipeline regardless of its accuracy. If the output isn't parseable, there is no classification to score. Schema failures are excluded from the accuracy metrics but counted here — a model with 87% schema compliance effectively gives you only 87% of the decisions you paid for.

Common schema failures

• Reasoning preamble — model outputs a <think>...</think> block before the JSON (DeepSeek). Stripped before parsing if detected.
• Extra fields — model adds explanatory fields not in the schema. Tolerated if the required fields are present.
• Wrong enum values — model returns "NUCLEAR" instead of "nts" or "true" instead of true. Strict validation rejects these.

Naive-flag rate

Percentage of passages where the model's classification changed when the same passage was presented in English instead of Russian.

What a flip tells you

If a model says "nuclear signal" in Russian and "no alert" in English for the same passage, it is responding to language-specific connotations or translation artifacts, not to the substance of the statement. A high flip rate means the model's understanding is fragile — it depends on which language you happen to feed it.

The pattern across models

NOT MEASURED. A translation-robustness arm was designed and deliberately not run: a verdict that changes between our translation and the original cannot be attributed to the model rather than to the translation, and a number that cannot be attributed should not be reported. English shown on this site is a post-hoc reading aid; no model ever saw it.

Dominant flip direction

More flips go RU→EN (1,124) than EN→RU (723). English translations tend to produce more alert calls — the translation sharpens ambiguous Russian hedging into what reads as a clearer threat in English.

Cost

Total API spend for this model across the full benchmark: 100 items × 1 language (Russian) × 2 repetitions = 1,788 calls.

What's included

• Input tokens: the system prompt (construct definitions, non-test examples, JSON schema) + the passage text. The system prompt is ~2,000 tokens; passages range from 50 to 800 tokens.
• Output tokens: the JSON response (~200–500 tokens depending on evidence span length).
• Pricing: the provider's published per-token rate on the date of the run, applied to measured (not estimated) token counts.

Cost vs accuracy trade-off

Price predicts nothing here. The cheapest configuration (GPT-5.6 Luna, $0.11) misses 2 nuclear-signal records; the dearest (Claude Fable 5, $7.00) misses 2. The naive citation check separates the slate 24-fold — but reading every flagged span found 0 inventions.

Rank

Position in the leaderboard, ordered by Macro-F1 under equal miss/false-alarm weighting (1:1). Rankings change under different penalty weights — see Ranking Sensitivity below.

A rank difference of 1 position may not be statistically significant. Wilson intervals are shown beside each accuracy in the leaderboard and indicate which rank differences are real.

Model

The exact model slug and dated version used for this run. Every model is tested via its native provider API — never through OpenRouter or another proxy that may serve quantised or different versions.

Each model received the identical system prompt, construct definitions, non-test examples, and JSON schema. The only variable is the model itself.

Provider

The company operating the model's native API. The provider determines pricing, rate limits, data residency, and — for open-weight models — which weights and quantisation level are actually served.

Providers are multinational by design: US (OpenAI, Anthropic, Google), China (DeepSeek, Alibaba, Zhipu AI, Moonshot). Korea and Russia (Sber). This tests whether national origin of the model affects its ability to read Russian strategic rhetoric.

Passage

A short identifier and text preview of the benchmark passage. Each passage is a chunk from the Russian Officialdom corpus — a continuous stretch of text from an official Telegram channel, Kremlin transcript, or parliamentary record.

Passages were selected to be self-contained enough for classification without external context, though chunk boundaries can split a speaker's argument (especially in Duma transcripts). The benchmark version records the exact chunk boundaries.

Reference Label

The provisional expert reference label for this passage: None (no alert), RLS (conventional red line statement), or NTS (nuclear threat signal).

Provisional status

These labels are NOT certified ground truth. They are class-enriched expert labels based on the RRLS and NTS construct definitions. Mikhail Troitskiy completed all 189 model-assisted Expert Review passages, with 28 dispute notes. Adam's independent blind pass has not been returned. Labels will change after adjudication.

The constructs

• RLS requires both a boundary span ('if NATO does X') and a consequence span ('Russia will Y'). A vague warning without a stated consequence does not qualify.
• NTS requires specific invocation of nuclear weapons, nuclear doctrine, or nuclear-capable systems in a threatening or signalling context — not merely descriptive or retrospective.

Speaker

Who produced the statement. Categories:

• Putin — direct presidential statements, press conferences, addresses
• Lavrov — foreign minister, diplomatic contexts
• Medvedev — former president, now prolific Telegram poster with no operational authority. His maximalist rhetoric is the single biggest source of false NTS alerts.
• Shoigu / Patrushev — defence/security establishment
• Peskov / Zakharova — press secretary and foreign ministry spokesperson. Often quote or paraphrase Putin — the indirect speech trap.
• Duma/FC — parliamentary proceedings. Procedural language triggers false alerts.
• Analyst — non-officials (e.g. Karaganov). An analyst proposing policy is not an official threatening action, but models cannot make this distinction.

Year

The year the statement was produced. The corpus spans 1999–2025. Most benchmark passages concentrate in 2023–2025 because that is when Russian escalatory rhetoric intensified around the Ukraine war.

Year matters because the same phrase can carry different weight over time. 'Nuclear doctrine does not exclude first use' was doctrinal boilerplate in 2020 and an active signal in 2024.

Models Agree

How many of the 10 models produced a classification that matches the reference label for this passage. Higher = easier passage (most models get it right).

Models Disagree

How many of the 10 models produced a classification that differs from the reference label. High disagreement (7+) flags the hardest passages — the traps that defeat most models.

A passage with 10/10 disagreement means every model gets it wrong. The Karaganov op-ed (#103) is the only such passage in this benchmark.

Worst Error

The most consequential wrong answer any model gave for this passage, ranked by operational severity:

1. Missed NTS — a nuclear signal classified as no-alert (most dangerous)
2. Missed RLS — a conventional red line classified as no-alert
3. False NTS — a no-alert passage classified as nuclear signal
4. False RLS — a no-alert passage classified as conventional red line

The number in parentheses shows how many models made this particular error.

Stable Decisions

The number of passages (out of 893 = 298 × 2 repetitions) where the model's classification was the same in both Russian and English. Higher = more language-robust.

Flip Rate

Percentage of passages where the model's classification changed between Russian and English. Lower is better.

A flip means the model's understanding depends on the language of presentation, not the substance. At deployment scale, this introduces unpredictable behaviour depending on whether the input pipeline translates before classifying.

Telegram Official Arm

266,604 chunks from official Russian Telegram channels. Short statements (median 231 tokens per chunk), context-poor, usually direct and unambiguous.

Models generally perform best on Telegram because the rhetoric is straightforward. The main error source is Medvedev's channel — maximalist bluster that models read as nuclear signals.

Kremlin Arm

12,338 chunks — press conferences, bilateral meetings, direct addresses, official transcripts from kremlin.ru.

Longer and more nuanced than Telegram. Conditional language ('if they do X'), retrospective framing ('as we said before'), and diplomatic hedging concentrate here. This is where models accumulate the most missed-RLS errors — genuine red lines expressed in language too subtle for pattern-matching.

State Duma Arm

10,553 chunks — legislative transcripts from the lower chamber. Longer-form than Telegram posts; tokens per document (full session transcripts).

The benchmark passage is a chunk boundary that may split a speaker's argument across chunks. Procedural uses of 'eskalatsiya' (escalation) in parliamentary debate trigger false alerts. Gemini's long context window appears to help here.

Federation Council Arm

6,886 chunks — upper chamber proceedings. The smallest arm. Senatorial rhetoric is generally more measured than Duma, but the small sample makes per-model estimates unstable.

Caution: with only 14 benchmark passages from this arm, a single passage error can swing the F1 by ~0.07.

Overall (all arms)

The pooled Macro-F1 across all 100 passages regardless of source arm. This is the same number shown in the Findings leaderboard.

The overall score can mask large per-arm variation. A model with 0.76 overall may be 0.84 on Telegram and 0.55 on Duma — operationally very different depending on which source you're monitoring.

Passages (per arm)

How many of the 298 benchmark passages come from this source arm. The distribution is not prevalence-representative — it over-samples from Kremlin and Duma relative to their corpus share to ensure enough hard cases.

No-Alert Passages

Passages from this source arm that carry a None reference label — no red line, no nuclear signal. These include routine statements, retrospective doctrine, quoted speech, procedural language, and deliberate traps (e.g. Medvedev bluster).

RLS Passages

Passages from this source arm labelled as conventional Red Line Statements. These contain both a boundary span and a consequence span.

NTS Passages

Passages from this source arm labelled as Nuclear Threat Signals. The rarest and most consequential class.

% of Benchmark

This arm's share of the total 298 benchmark passages. Telegram dominates (56.4%) because it dominates the corpus (96.5% of documents). Kremlin is over-represented relative to its corpus share (26.2% of benchmark vs 2.8% of corpus) because it contains the most nuanced rhetoric.

Failure Type

The category of model error. See the Failure Atlas tab modal for the full error taxonomy. The table shows how these errors distribute across source arms — some error types concentrate in specific source types.

Dominant Cause

The most common rhetorical pattern driving this error type in this source arm. These patterns are the benchmark's diagnostic output — they tell model developers what to fix, not just that something is wrong.

English: No Alert

Count of passages that the model classified as no-alert in English. Compare with the row total (the RU classification) to see how many flipped.

English: RLS

Count of passages classified as conventional red line in English. Off-diagonal cells (e.g. RU:None → EN:RLS) represent language-induced false alerts.

English: NTS

Count of passages classified as nuclear signal in English. The RU:None → EN:NTS cell is particularly concerning: passages that are benign in Russian but read as nuclear in English translation.

Total (Russian)

Total decisions in this Russian classification row. Should equal the number of passages times repetitions, divided across the three classes.

Predicted: No Alert

The model predicted no alert for these passages. Diagonal (Ref:None → Pred:None) = correct. Off-diagonal = the model missed a genuine signal.

Predicted: RLS

The model predicted conventional red line. Diagonal (Ref:RLS → Pred:RLS) = correct. Ref:None → Pred:RLS = false alert. Ref:NTS → Pred:RLS = nuclear signal downgraded to conventional — dangerous.

Predicted: NTS

The model predicted nuclear signal. Diagonal (Ref:NTS → Pred:NTS) = correct. Ref:None → Pred:NTS = false nuclear alert (the costliest false alarm). Ref:RLS → Pred:NTS = conventional red line upgraded to nuclear — over-reaction.