Technical report · run fedjev-2026-09-20

fedjev-bench

Pairwise textual hawkishness scores for FOMC chair openings, evaluated against same-day funds-rate changes

FedLock-faithful TrueSkill replica

run_id: fedjev-fedlock-replica-2026-09-20 Companion claim (do not conflate): Gate 7 in the main bench is a different experiment. Gate 7 asks whether this repository’s Bradley–Terry and Score series agree in rank with published FedLock scores. This note asks whether a separate tournament, run on the same 95 chair openings under FedLock V3’s documented protocol, recovers a similar ordering. It is not a 4,000-speech / ~60,000-comparison scale copy of the published FedLock run.

This document is written so a reader fluent in finance and monetary policy, but not in natural-language processing, can re-implement the comparison from the prose and the frozen artifacts. Numbers below are taken from agreement.json and cost_performance.json. None were invented for this write-up.

1. What each name means in this note

Federal Open Market Committee (FOMC). The Federal Reserve committee that sets the federal funds target. The documents scored here are the Chair’s opening remarks at post-meeting press conferences (95 openings in data/clean/statements.jsonl).

Behavioral label versus textual hawkishness. The Committee’s voted action is a behavioral label. In the main bench that label is d_same, the same-day change in the federal funds target: hike, hold, or cut. A hold day has d_same = 0 by construction, so the behavioral label cannot say whether the wording of a hold was hawkish or dovish. Textual hawkishness is a separate construct: how hawkish the opening sounds about inflation and the policy path. This replica ranks openings on textual hawkishness relative to the macro conditions attached to each text. It does not use d_same as the scoring target.

FedLock. An independent, published scoring project (methodology; snapshot in data/raw/fedlock/data.json). FedLock V3 runs a large pairwise tournament: a large language model (Llama 3.3 70B) is shown two anonymized speeches and asked which takes the more hawkish monetary-policy stance given contemporaneous macro conditions. Those pairwise wins are aggregated with TrueSkill (defined below) over roughly 60,000 comparisons and 4,000 speeches. This repository does not re-invoke Llama. It only reads the published press_conference fields:

Published fieldMeaning in FedLock
mRaw TrueSkill mean (μ) — the meeting’s absolute text-hawkishness score
maEra-adjusted mean — m after subtracting a quarterly (era) average, so the score is relative to that period’s typical language
sTrueSkill uncertainty (σ). Lower means the tournament has seen enough matches that the mean has settled
nNumber of comparisons that speech entered in FedLock’s own run
stSpeech type. This note uses press_conference only
dCalendar date on the FedLock record

TrueSkill versus Bradley–Terry. Both turn pairwise “A beats B” judgments into a number per document. They are not the same estimator.

Anonymized pairwise tournament. Each comparison shows two stripped texts (Text A, Text B) and asks which is more hawkish. Anonymization here means scripts/strip_meta.py removes speaker titles, calendar dates, and chair surnames before the judge sees the text. Chair identity is not placed in the judge’s input. Presentation order of Text A / Text B is randomized each match so a left/right habit cannot pile onto one meeting.

Macro-conditioned judgment. The judge is not asked “which text uses more hawkish words in isolation.” Each text is paired with four Federal Reserve Economic Data (FRED) series as of that speech date: core personal consumption expenditures inflation (PCEPILFE, year-over-year when a twelve-month change is computable, otherwise the level); the civilian unemployment rate (UNRATE); real gross domestic product growth (GDPC1, quarter-over-quarter at a seasonally adjusted annual rate when available, otherwise year-over-year); and the CBOE Volatility Index close (VIXCLS). The instruction is to judge relative hawkishness given those conditions. A 2 percent inflation remark in 2012 is not treated as the same stance as a 2 percent remark in 2022.

Gate 7 versus this replica (two claims).

Gate 7 (main bench)This replica
QuestionDo the main bench’s Bradley–Terry / Score series agree in rank with published FedLock m / ma?If we re-run a FedLock-style tournament on the 95 openings, do Jev, Haiku, and published FedLock order meetings the same way?
Judge protocolFixed criterion more hawkish about inflation; no macro in the prompt; Bradley–Terry (and a direct Score pass)Macro-conditioned relative hawkishness; TrueSkill; Swiss / uncertainty pairing
What is not claimedA TrueSkill or macro-conditioned replication of FedLockA 4,000-speech / ~60,000-comparison scale copy of FedLock

The three arms.

ArmWhat it isWhat it is not
JevTypeSafe SystemOne Choice, model jev-latest. Live calls in this run; answers cached under runs/fedlock_replica/.Not the main-bench Bradley–Terry graph.
HaikuAnthropic Claude claude-haiku-4-5-20251001, asked for a structured JSON winner and a soft probability.Not Llama 3.3 70B (FedLock’s published judge).
Published FedLockFrozen m, ma, s, n from data/raw/fedlock/data.json.Llama is not re-invoked. This arm is a reference ranking, not a third live tournament.

2. Procedure a reader can re-run (from cached outputs)

Do not mutate data/pairs/gold_pairs.jsonl or the cached replica logs. To recompute the tables in this note from what is already on disk:

  1. Confirm the corpus: 95 openings in data/clean/statements.jsonl.
  2. Confirm macro vintages in data/raw/fred/ (PCEPILFE, UNRATE, GDPC1, VIXCLS) and the as-of join in results/fedlock_replica/macro_asof.json.
  3. Confirm published FedLock scores in data/raw/fedlock/data.json, speech type press_conference.
  4. Read the two tournament logs: comparisons_jev.jsonl (1,034 comparisons) and comparisons_haiku.jsonl (1,033 comparisons).
  5. Read the fitted ratings: trueskill_jev.csv and trueskill_haiku.csv (columns date, doc_id, mu, sigma, n_comps).
  6. Match openings to FedLock on the FedLock d field, trying calendar offsets 0, +1, −1, +2 days. This run matched 92 of 95 openings (fedlock_matches.json).
  7. Rank-correlate the three μ / m / ma series. Agreement statistics are stored in agreement.json (Spearman and Kendall, each with a bootstrap standard error from 1,000 resamples and a percentile 95 percent confidence interval).
  8. Read listed-price accounting from cost_performance.json.

To rebuild the tournament from the same protocol (optional; not required to read this note), the entry point is scripts/run_fedlock_replica.py. Stopping rule, priors, and pairing are listed in §3.

3. Methods

Judge task. Pairwise selection of the more hawkish monetary-policy stance conditional on the four macro series attached to each text, using the instruction string in scripts/run_fedlock_replica.py (JUDGE_INSTRUCTIONS). Texts are anonymized as defined above.

Aggregation. Microsoft TrueSkill with priors μ₀ = 50, σ₀ = 8.33. Pairing is Swiss-style with uncertainty targeting: the next match prefers documents that still have high σ (the rating is still loose) and opponents with a similar μ (so the match is informative). Soft probabilities from the judge update ratings by interpolating a decisive win and a decisive loss, weighted by the judge’s probability that A wins (confidence-weighted). The tournament stops when every document has σ < 2.0, or at about 30 comparisons per document, with a global cap of 2,850 comparisons.

Date match to published FedLock. Offsets 0, +1, −1, +2 on the FedLock d field: 92 of 95 openings matched.

Protocol deviations / notes (observed)

4. How to read the agreement table

Spearman’s rank correlation asks: if you sort meetings by one arm’s score and again by the other, how similar are the two orderings? +1 is identical ranks; 0 is no rank association; −1 is reversed ranks.

Kendall’s rank correlation asks a pairwise version of the same question: for two meetings, do the two arms agree on which meeting is more hawkish? It is typically smaller than Spearman on the same data; that is a property of the statistic, not a second sample.

Standard error (s.e.) on these correlations is the standard deviation of 1,000 bootstrap resamples of meetings (see agreement.json, n_boot). The 95 percent confidence interval is the percentile interval from the same resamples. This note writes “standard error” or “s.e.” It does not use the label STE.

n is the number of meetings in that contrast: 95 when both live arms have a rating; 92 when a published FedLock score must be present.

5. Results

Rank agreement

Cost and performance (listed prices; not an accuracy claim)

Prices used for the accounting: Jev $0.042 per million input tokens, output free; Haiku $1.0 per million input tokens and $5.0 per million output tokens.

ArmComparisonsInput tokensOutput tokensUSDEffective $/million tokens$/comparisonLatency mean / median / 95th percentile (ms)Comparisons/s
Jev10343,376,15028,7680.14180.04160.000137283 / 271 / 41922.978
Haiku10333,347,68848,4123.58971.05700.003475732 / 687 / 9888.156

Jev stop: all_sigma_lt_2; max σ = 1.986; fraction with σ < 2 = 1.000. Haiku stop: all_sigma_lt_2; max σ = 1.965; fraction with σ < 2 = 1.000.

6. Interpretation

Spearman / Kendall concordance between Jev and Haiku, under a shared FedLock-style protocol, measures whether two different live judges produce a stable relative-hawkishness ordering on this openings sample. Concordance with published FedLock m / ma asks whether that same protocol family, applied to chair openings rather than FedLock’s broader speech corpus and Llama 3.3 70B judge, recovers a similar ordering of meeting-day stance.

Agreement with raw m is tighter than with era-adjusted ma on both live arms (Jev ↔ m Spearman +0.965 versus Jev ↔ ma +0.790; Haiku ↔ m +0.945 versus Haiku ↔ ma +0.768). Era-adjusted ma removes quarterly means. Disagreement between the m and ma contrasts therefore partly reflects the era composition of the 2011–2026 openings window, not a second independent sample.

Cost and latency columns are accounting facts at the listed prices above. They are not accuracy claims and are not a gate.

This replica does not replace Gate 7. Gate 7 remains the main bench’s rank agreement with published FedLock scores under Bradley–Terry / Score. See results/fedlock_fidelity.md and ANALYSIS §6.

7. Limitations

8. Artifacts