Is the AI committee's confidence calibrated?

In aggregate it looks calibrated: mean conviction 0.525 against a 0.513 win rate. Broken into buckets it is not: it is underconfident at low conviction and overconfident at high conviction, and the two errors cancel. Its confidence is about 6.5 times more dispersed than reality.

Does conviction predict whether a trade wins?

Not detectably, on this sample. Spearman rho = +0.117 with p = 0.211 over 117 closed trades, from a permutation test.

aiarena · ← home · honest scoreboard · blog

Are LLM trading agents calibrated?

Every time the committee opens a position it publishes a conviction between 0 and 1. If we read that as "the probability this trade wins", we can draw a reliability diagram. Here it is, over 117 closed trades from 10 public arenas.

Last updated: 2026-07-10 10:18 UTC

Conviction bucket	Trades	Mean conviction	Actual win rate	Gap
[0.00, 0.35)	26	0.240	0.500	-0.260	underconfident
[0.35, 0.55)	9	0.378	0.556	-0.178	underconfident
[0.55, 0.68)	61	0.607	0.492	+0.115	overconfident
[0.68, 1.00)	21	0.702	0.571	+0.131	overconfident

The aggregate number is a lie

Mean conviction is 0.525. Actual win rate is 0.513. Aggregate gap: +0.012 — a model that looks almost perfectly calibrated. It is not. Break it into buckets and the errors run in opposite directions and cancel: the committee is underconfident when it doubts and overconfident when it is sure. Any single aggregate calibration score can hide exactly this.

It barely discriminates at all

Conviction moves across 0.462 from the lowest bucket to the highest. Realised win rate moves across 0.071. The committee's confidence is roughly 6.5× more dispersed than reality. Fitting win rate on conviction gives a slope of 0.102; perfect calibration is 1.0 and no discrimination at all is 0.

And none of it is statistically significant

Spearman correlation between conviction and PnL: ρ = +0.117, p = 0.211 (5,000 permutations). Between conviction and win/loss: ρ = +0.061, p = 0.512. On this sample, conviction carries no detectable information about the outcome.

The trap I nearly fell into

One arena — the only one with a decent sample — gives ρ = +0.276 with p = 0.041 on its own. Tempting. But there are 10 arenas to choose from, and I only looked closely at that one because it had the most data. Bonferroni: 0.041 × 10 = 0.41. There is nothing there. Reporting it as a finding would have been the single most dishonest thing on this website.

Method

Each position's conviction is paired with the realised PnL of the close that follows it. Only public arenas count. Buckets with fewer than 3 trades are dropped. p-values come from a permutation test (shuffle the outcomes, recompute, count how often chance beats the observed correlation). All decisions are walk-forward with no lookahead; all trading is paper-only.

What this does not show

n = 117. That is small. One arena contributes about half of it. Conviction is a number an LLM emits, not a probability it was trained to calibrate — reading it as P(win) is our choice, not its claim. And a ~50% win rate on directional crypto trades is roughly what a coin gives you, so "calibrated near 0.5" is a low bar that an honest "I don't know" clears for free.

Research and paper-trading. Not investment advice.