Claude Code entwickelte eine Trading-Strategie ohne Blick auf die letzten zwei Jahre

I gave Claude Code ten years of USD/JPY daily bars and asked it to find a trading rule. It ran backtests through an API, as many as it wanted up to 30, but the last 20% of the data was hidden from it the whole time. At the end it got exactly one look at the hidden part, and it had to write down what it expected before it looked.

The point wasn't to find a strategy. It was to see what happens to "the best backtest an AI found" when the AI can't quietly try its way into it.

It used 13 of its 30 runs and then stopped on its own: "further searching would mostly have added selection bias."

Run 1 was a sizing bug, not a test of the signal, and the agent said so in its next note. Runs 10 to 13 weren't new ideas; it was checking whether #3 sat on a plateau or a spike. The lookback held up (20, 50 and 100 days were all positive). The trailing-stop distance didn't (2 yen lost money, 4 yen made a third as much), and it noted that "3 yen is partly a lucky value."

Here is the dashboard's research note at the end. The labels are Japanese: hidden period 20% (520 bars), hidden period used once, 13 trials, and the run list with each trial's deflated Sharpe in the last column.

Every search run came back "within chance, given the number of trials." The luck threshold (the per-trade Sharpe you'd expect from the best of N noise strategies) jumped to about 0.5 by the third trial and stayed between 0.47 and 0.53. The pick, #3, had a per-trade Sharpe of 0.127 and a deflated Sharpe of 0.003.

The agent agreed with it. From its notes before the final run:

Ledger says chance, and I agree: I expect roughly break-even, not a confirmation.

Written before the run: about +1% over the hidden two years (range −6% to +8%), max drawdown about 6%, about 10 trades, shorts losing money again.

The pick didn't fall apart, but it didn't prove anything either. It earned a small positive return in the hidden period, per trade about half as good as on the visible data. That fits a weak edge, or no edge plus some luck. I would not call this a working strategy.

The interesting part isn't the strategy. It's that the agent didn't oversell it. Without the ledger, the story writes itself: "AI finds a USD/JPY breakout system, +12% with a 7.5% drawdown." With the ledger in every response, the agent kept reading "within chance," stopped searching early, ran sensitivity checks instead of new ideas, and predicted the hidden period close to what happened.

The ledger, the hidden period and the single holdout are part of Hawk Backtester's Agent Mode. Strategies are plain Python files that run on your machine ( pip install hawk-bt ).

If you let agents run backtests, what do you use to keep them from picking the lucky run?