The one-line bug that makes your breakout backtest impossible — or worse, profitable

Fifth in a series on building honest crypto quant. Post 4: three bugs in my own ledger.

Zero out of one thousand

Here is a breakout rule. It looks fine. It has appeared in a thousand tutorials.

hh = high.rolling(20).max()      # "the 20-bar high"
entry = close > hh               # "price breaks out above it"

Count how often entry is true on real BTC 30-minute bars:

Not "rare". Zero. And it is zero on any price series you will ever load, because it is arithmetic, not markets:

close[t] ≤ high[t] ≤ max(high[t-19 .. t])

The current bar's own high is inside the window. The rolling maximum is therefore always at least the current bar's high, which is always at least its close. close > hh cannot be true. Ever.

The fix is one method call. The level must be built from prior bars only:

hh = high.rolling(20).max().shift(1)   # the high of the PREVIOUS 20 bars

Same 1000 bars, same data, one .shift(1):

The same bug, wearing the opposite mask

In a live system, an always-false condition is merely embarrassing: your strategy does nothing, forever, and looks patient.

In a backtest, the identical family of mistakes flips sign. Suppose that instead of comparing against the window you take the window's extreme as a target or a stop, and you allow fills inside the same bar that produced it. Now the level you are reacting to was computed using the high and low of a bar you are still trading. You are not reading a signal. You are reading the answer.

The equity curve that comes out is smooth, steep, and completely fictional. This is the more expensive version of the bug, and it is the same line of code — the only difference is whether the contaminated value ends up on the losing side of a comparison that can never be satisfied, or the winning side of one that always can.

The rule is simple and worth burning in: any level derived from a rolling window of highs, lows, or closes, that you intend to compare against the current bar, must exclude the current bar. Donchian channels, prior-day high/low, swing highs, pivot levels, "N-bar breakout" — all of them.

The rest of the family: silent false

Once I went looking, the same failure mode was hiding in two more places, and neither of them raised a single exception.

1. An average of one

vol_ratio = volume / volume.rolling(period).mean()   # period = 1

A rolling mean over a window of one is the value itself. So vol_ratio ≡ 1.0 on every bar, and vol_ratio > 1.5 — "on rising volume" — is false forever. The parser that produced the rule had chosen period = 1 because the human wrote "on volume", and nobody, human or machine, ever asked what the resulting series actually looked like.

2. A threshold that lives outside the data

Binance publishes a retail long/short account ratio. A crowding-reversal strategy wants to short when the crowd is very long, and buy when the crowd is very short. Symmetric. Obvious.

short: retail_ls > 1.5      # fires often
long:  retail_ls < 0.7      # fires never

Here is what that series actually does on BTC:

"Retail is net short" is not a rare state on Bitcoin. It is not a state. The long branch of that strategy was never going to fire — not because of a coding error, but because the threshold was invented by symmetry, and the data is not symmetric. Two other strategies had the same disease, on top_ls and on a different retail threshold.

Three of my twelve rule-driven arenas could only ever trade in one direction. And they were doing precisely that, in public, on the scoreboard, presented as long/short experiments.

What made all three invisible

They are all silent falses. Syntactically valid. No exception. No warning. The condition simply evaluates to False on every bar, and the strategy — which is supposed to wait patiently for its conditions — waits patiently forever.

Nothing in the output distinguishes "my rule is broken" from "my setup hasn't appeared yet." The dashboard shows 0 trades for both.

I want to be precise about how badly this hid. I had a monitor running every thirty minutes for ten hours, checking every position against the rule that opened it. It reported zero violations, every round, correctly. Its question was: "does the side we opened match the branch that fired?" It never asked the other question: "is the other branch alive?" I even logged the line [long]✗ retail_ls < 0.7 = False dozens of times and read straight past it.

Consistency checking cannot find a dead condition. Only counting can.

The linter

So: replay every condition of every rule across the last N bars, vectorised, and count the true bars. The whole idea is that boring. The part that matters is that it reports three states, not two:

Collapsing unknown into never is how you build a linter that cries wolf. Missing data must never be reported as an impossible condition. That distinction is the only subtle thing in the entire feature.

A branch also gets flagged if every condition fires individually but they never co-occur. That one is a softer warning: rare confluence is a legitimate strategy, and impossibility is not.

Run it over my twelve rules and it finds all three dead branches, in about a second. I then checked its verdict against 23 real trades those three arenas had actually made — 0 long / 8 short, 7 long / 0 short, 0 long / 8 short. Zero counterexamples. The strategies had been telling me for days.

One more trap, on the way out

The obvious repair for the crowding thresholds is a percentile: don't say "below 0.7", say "in its own bottom 20%". Distribution-aware, level-drift proof, elegant.

I implemented it, then measured it before shipping:

A rolling percentile rank inherits the trend of the series it ranks. Over this window retail_ls drifts down, so the current value is almost always near the bottom of its own recent range — and the upper tail empties out entirely. I had rebuilt the exact bug I was fixing, pointing the other way.

What works is detrending first: compare the value to its own rolling mean in units of its own rolling standard deviation. A plain z-score.

Symmetric, on both series, on real data. That is the version that shipped.

The thing I actually want you to take away

Four bugs, and three of them were silent: no crash, no stack trace, no warning — just a boolean quietly returning False until the end of time. The fourth was the linter's first fix, which would have introduced a new silent false in the opposite direction if I had not measured the distribution before shipping it.

Reading the code will not save you. I wrote that rolling(20).max(), I reviewed it, and it looks exactly like what it is supposed to be. What saves you is computing a statistic on real data and looking at the number: how often is this actually true?

If your strategy has a condition you have never seen fire, you do not have a patient strategy. You have an untested one.

The linter is open source

Everything above ships as rulelint — MIT, no dependencies beyond pandas and numpy, and about 240 lines.

pip install git+https://github.com/momoddo/rulelint

from rulelint import lint
report = lint(rule, bars)
# dead  ·  price(1) cross_above high(20)  ·  hits=0

The tests are the point: each one pins a bug that reached production, and the crowding-threshold test runs against a real Binance retail long/short series shipped in the repo. One test is named test_zscore_is_NOT_immune_to_a_strong_deterministic_trend and asserts that the z-score fails too — deliberately, so nobody mistakes it for a silver bullet.

If you find a fourth member of this bug family, open an issue. I would rather hear it from you than from my own scoreboard.

rulelint on GitHub →   the honest scoreboard →

FAQ

Replies

Research and paper-trading. Not investment advice. Past performance is not indicative of future results.