We pre-registered our trading experiment. Judgment day is August 21.
Sixth in a series on building honest crypto quant. Post 4: three bugs in my own ledger. · Post 5: the one-line breakout bug.
Why bother
With thirteen arenas, flexible date windows, and half a dozen plausible metrics, it is trivially easy to find something that looks like an edge after the fact. Run enough comparisons and one of them will shine by chance; pick the window after seeing the curve and every strategy has a good month somewhere. Multiple comparisons plus hindsight window selection is how most trading content lies without technically lying.
I have already been bitten by a milder version of this: one of my arenas showed a conviction-to-PnL correlation of p = 0.041 — until I remembered I had looked at ten arenas and only zoomed in on that one because it had the most data. Bonferroni-corrected, there was nothing there. I published the null result instead.
Pre-registration removes those degrees of freedom by spending them now, in public, before the data can vote.
The setup
Since early July, thirteen public paper-trading arenas have been running side by side on BTC perpetual futures (one on 5-minute bars, twelve on 30-minute bars). Twelve are driven by plain-English mechanical entry rules — trend breakout, pullback, Bollinger reversion, crowding reversal, order-flow confirmation, and so on. One, Control — Committee Only, is the same LLM committee with no rule at all. Every arena publishes every trade, every deliberation, and an honest scoreboard including the ones that are losing.
Two things happened mid-run that are already disclosed in each arena's public change history and matter for interpretation: on 2026-07-10 an indicator bug was fixed that had made two breakout arenas unable to trade at all, and three arenas' crowding thresholds were rewritten from impossible absolute values to detrended z-scores. Segments before and after material changes will be reported separately.
Registration
Primary question. Over the same market and period, do mechanical-rule arenas produce better excess return over buy-and-hold than the rule-free committee control?
Secondary questions. (a) How many of the twelve rules individually beat buy-and-hold over their own lifetime? (b) Does the committee's veto/sizing layer add or subtract versus each arena's judgment-free baseline (the A/B shadow account every arena already runs)?
Metrics, in order of priority. Excess return vs same-period buy-and-hold; number of closed trades; win rate; maximum drawdown. All computed exactly as the public scoreboard computes them today — walk-forward, no lookahead, fees included.
Evidence thresholds. An arena's result counts as evidence only if it has ≥ 30 closed trades by the cutoff. Below that it will still be published, labeled insufficient sample. An arena that never trades is no evidence of anything — explicitly not a win, per the zero-trade trap.
Cutoff and publication. Data through 2026-08-20 23:59 UTC; results published on or before 2026-08-21 as a follow-up post, linked from here. This registration will not be edited after publication except for typography; any correction will be an appended, dated note.
Commitments. No parameter or rule changes to these thirteen arenas before the cutoff except bug fixes, which are automatically recorded in each arena's public change history and will be listed in the results post. Segments across material changes are reported separately, not blended. All thirteen results are published — no dropping losers.
Stated prior, for the record. From earlier stress-testing: most price-only mechanical rules will not beat buy-and-hold. The genuinely open question is rules versus the free-running committee. If the control beats most rule arenas, that is a publishable, uncomfortable answer and it will be published with the same font size.
Experiment #2, registered while we're at it
Our committee makes nine separate LLM calls per deliberation — one per niche analyst — on the design premise that independent reasoning beats a single batched prompt. That premise has never been tested and it is the single biggest line in our cost structure. So: two fresh arenas, identical configuration, no mechanical rule, same market and timeframe — one with nine independent analyst calls, one with all nine analysts generated in a single batched call. Same metrics, same cutoff, same thresholds as above. If batching is not worse, the cost ceiling of every experiment we run drops by roughly half. Start date will be recorded in the arenas' public history when they are switched on.
What would falsify what
| Outcome on 2026-08-21 | What we'd have to admit |
|---|---|
| Control beats most rule arenas on excess return | Rule-gating as implemented subtracts value; the committee alone is the better product |
| Most rule arenas beat control, few beat buy-and-hold | Rules select better entries than the committee, but "better" still isn't "good" — matches the stated prior |
| Everything under 30 trades | Six weeks was too short and this registration taught us mostly about our own impatience — extend, don't conclude |
| Batched committee ≈ independent committee | Our most expensive architectural belief was decoration |
Whichever row it is, you will read it here.
FAQ
What is a pre-registered trading experiment?
Borrowed from clinical trials: publish the hypothesis, exact metrics, sample thresholds, and evaluation date before the results exist, so you can't quietly pick the window, metric, or subset that flatters you. On the judgment date the analysis runs as declared and gets published, whatever it says.
Why not just show the results when they're good?
Because that's the disease. Thirteen arenas and flexible windows guarantee something looks good eventually. A scoreboard that only speaks when it has good news is an advertisement with extra steps.
Can't you still fudge it — who enforces this?
The page you're reading is timestamped in multiple third-party caches the moment it's published, every arena's trades and change history are public, and daily state hashes are being anchored so records can't be quietly rewritten. Beyond that: the enforcement is that you're reading this. Check back on August 21.
Is this investment advice?
No. Paper trading, research, tiny samples, and a stated expectation that most of it won't beat buying and holding. Crypto derivatives are extremely risky.
Replies
Research and paper-trading. Not investment advice. Past performance is not indicative of future results.