We tested 54 strategy variants, found a winner, then invalidated it ourselves
Seventh in a series on building honest crypto quant. Post 6: we pre-registered our experiments.
54-grid discovery
→ apparent winner (+44.7%)
→ replication 1: spot→perp data … +20.2%
→ replication 2: + real funding … +15.3%
→ replication 3: disjoint windows … bear-year-only
→ replication 4: other assets … ETH −63.9%
→ backtest claim: dead
→ live arena: still running, verdict 2026-10-10The discovery that begged to be true
The sweep was honest by construction: taker fees on every trade, signals lagged one bar, levels computed with .shift(1) (we have a whole post about what happens when you forget that). Nine of our own entry rules crossed with six exit structures — default ATR stops, wide stops, tight take-profits, chandelier trails, no time-stop, and pure stop-and-reverse.
The single profitable cell paired a trend-regime entry filter with stop-and-reverse exits: no stop-loss, no take-profit, the position only closes when the opposite signal fires — and immediately flips. Net +44.7% over three years, with a win rate of just 19% and an average winner 5.5× the average loser. A control run fed the same exits with random entries and lost 68%, so the entry filter was doing real work in-sample.
It looked exactly like the thing every backtester wants to find. Which is precisely why the right next step was to attack it.
Four replications, four failures
| Attack | Result | What it means |
|---|---|---|
| 1. Trade the market you actually trade rerun on perpetual-futures data instead of spot | +44.7% → +20.2% | Nearly half the "edge" was a data-source artifact: spot and perp prices differ just enough around rolling extremes to change which breakouts fire |
| 2. Charge every real cost settle actual historical funding, period by period | +20.2% → +15.3% | An always-in-position strategy pays funding around the clock; ignoring it flattered the result by ~4pp on BTC, ~11pp on SOL |
| 3. Split the sample three disjoint yearly windows, indicators recomputed per window | +20.5% / −16.8% / +6.8% excess vs buy&hold: −69pp / −114pp / +51pp | It only beats buy-and-hold in the one bear year. That is a bear-market hedge that got lucky with its sample, not an all-weather edge |
| 4. Change nothing, change the asset identical parameters on ETH and SOL | ETH −63.9% (max drawdown −80%) SOL +138.8% but −115pp vs buy&hold | Zero parameter transfer. A real regime edge should not detonate on the second-largest asset in the same market |
The tell we should teach
In hindsight the discovery carried its own warning label: it was the only profitable cell out of 54. If a strategy family has real signal, neighbouring parameterisations usually work too — worse, but directionally alive. A lone winner surrounded by 53 corpses is the statistical fingerprint of selection, not edge. The more cells you sweep, the more impressive your best cell looks, and the less it means.
The win-rate structure was a second tell in reverse: everyone asks for high win rates, but this cell won only 19% of its trades and made all its money on a handful of large winners. That is what surviving trend-following actually looks like — and it is also exactly the shape that one lucky bear market produces. The two are indistinguishable in-sample. Only replication separates them.
What happens to the arena
Here is the part that makes this platform different from a backtest blog. Before running these replications, we had already pre-registered the hypothesis (appended 2026-07-11, per the registration's own edit policy) and launched a live paper arena — Regime SAR — no stops, flip only — with the exact rule, frozen conditions, and a stated prior: more likely than not, this underperforms its backtest.
The replications above make that prior stronger, but they do not close the experiment. The arena keeps trading to its pre-registered judgment date, 2026-10-10, minimum 30 closed flips. If it somehow beats buy-and-hold out of sample, we will report that with equal prominence — and equal suspicion. If it fails, it gets a formal rejected verdict, and this post becomes its obituary's first draft. Either way the verdict is issued as an append-only event: earlier verdicts stay queryable forever, because a scoreboard you can quietly re-judge is not a scoreboard.
Check our numbers with our own data
Every public arena now ships a machine-readable evidence package: frozen experiment manifest, every deliberation with decision provenance (model vs fallback vs mechanical), every trade, equity curves, change events, and full verdict history.
Download the evidence package →
Also: the rule playground replays any entry condition over 1,000 real bars in your browser, and the linter that catches the silent-bug family is open source: rulelint (MIT). Daily state hashes are anchored in a public repository.
FAQ
Why publish this instead of quietly dropping the strategy?
Because the autopsy is the product. Anyone can publish winners; the numbers that make a scoreboard trustworthy are the ones that cost something to print. A platform that only speaks when it has good news is an advertisement with extra steps.
Wasn't the sweep itself a mistake?
No — sweeps are how you generate candidates. The mistake would have been treating the best cell as a conclusion instead of a hypothesis. The pipeline is: sweep → suspect → replicate → pre-register → let out-of-sample data judge.
What would change your mind about the strategy?
Thirty-plus flips of live out-of-sample data beating buy-and-hold through the pre-registered window, on the market it actually trades, with funding included. That is what the arena is for. Nothing less counts.
Replies
Research and paper-trading. Not investment advice. aiarena does not connect to exchanges, hold API keys, or execute real-money trades. Past paper performance is not indicative of future results.