I Let an LLM Agent Search for Trading Strategies. A Simple Heuristic Beat It.

Posted on Oct 20 • Originally published at pkazantsev.com

I gave DeepSeek V3 forty chances to find a profitable trading strategy.

A simple heuristic found +52% net. The LLM's early proposals lost money. After fixing three silent bugs the agent surfaced, it improved — then optimized its way to +123.7%, a number that didn't survive scrutiny. Here's what happened across four rounds.

In earlier posts in this series I tested six dead-simple strategies and three more sophisticated edges — all killed by fees, adverse selection, or data leakage. What survived was a backtest harness: point-in-time universe, walk-forward, trading costs, out-of-sample scoring only. Same rules for every strategy, no exceptions. I put an agent in front of it.

@dataclass class Proposal: family: str # "reversal" | "factor_residual" | "momentum" bar_size: str # "15m" | "1h" | "1d" lookback: int # days holding_period: int entry_threshold: float # The agent loop def agent_loop ( n_rounds: int = 40 ) -> list [ ScoredProposal ]: history: list [ ScoredProposal ] = [] for _ in range ( n_rounds ): prompt = build_prompt ( history ) # what failed, what survived json_str = llm ( prompt ) # DeepSeek V3 via OpenRouter proposal = parse_proposal ( json_str ) verdict = harness. score ( proposal ) # walk-forward over 451 symbols, 76 windows history. append ( ScoredProposal ( proposal, verdict )) return history Enter fullscreen mode Exit fullscreen mode The key rule: the agent never touches the scoring. It proposes; the same judge that every hand-built strategy went through decides. There is no way for it to talk its way to a good number.

I also kept a heuristic proposer alongside it: samples the space randomly, occasionally perturbs the current best. It is not purely random — the perturbation step is local optimization — but it needs no model and costs nothing. Both the heuristic and the agent ran for the same 40 evaluation rounds. That baseline turned out to matter a lot.

The first LLM run was worse than the heuristic search.

DeepSeek had the right instinct — it leaned toward factor-residual reversion on daily bars (buy coins whose recent return lagged the cross-sectional average, hold for ~30 days), which turned out to be the only corner that actually survives. But it picked short lookbacks and fast rebalances:

factor_residual/1d/lb30/reb15 -23.7% reject factor_residual/1d/lb20/reb10 -44.4% reject reversal/15m/lb5/reb2 -99.1% reject Enter fullscreen mode Exit fullscreen mode Meanwhile the heuristic had stumbled into factor_residual/1d/lb60/reb30: about +52% net, positive in most out-of-sample windows, tiny turnover. The model landed one step away from the edge and picked the wrong side of it. Right family, wrong calibration. The heuristic beat it.

Before I gave up on the model, something more useful turned up.

Several proposals came back with zero turnover and zero return. Not negative — flat. The strategy had never opened a position. I found three bugs, all caught the same way: strategy returned flat, zero turnover, no crash, no error message. Just silence.

Bug 1: Momentum's volatility window was scaled wrong. The signal normalized by a 96-bar trailing volatility. On 15-minute data that is one day — fine. On daily bars it is 96 days, longer than the formation window, so the signal never became valid.

# Bug: hardcoded window ignores bar size vol_window = 96 # Fix: scale to timeframe; floor of 5 bars avoids a 1-bar window on daily BAR_MINUTES = { " 15m ": 15, " 1h ": 60, " 1d ": 1440 } vol_window = max ( 5, round ( 96 * 15 / BAR_MINUTES [ bar_size ])) # 15m → 96 bars (one day) | 1h → 24 bars | 1d → max(5,1) = 5 days Enter fullscreen mode Exit fullscreen mode Bug 2: Lookback could exceed the formation window. The agent proposed a 100-day lookback on a 90-day formation window. Fix: clamp the lookback to what is actually available.

lookback = min ( proposal. lookback, formation_window - 1 ) Enter fullscreen mode Exit fullscreen mode Bug 3: The rebalance grid starved itself. The wrapper refreshed positions on a fixed grid from bar zero: bar 0, bar 40, bar 80. If the signal only came alive around bar 90, every refresh point landed in the warmup period. A holding period of 40 was dead while 30 and 45 traded — non-monotonic results that mean "bug", not "no edge".

# Bug: grid anchored at bar 0 rebalance_bars = range ( 0, n_bars, holding_period ) # Fix: anchor at first live bar first_live = warmup_length rebalance_bars = range ( first_live, n_bars, holding_period ) Enter fullscreen mode Exit fullscreen mode None of these were crashes. Each returned a flat book with no error. A backtest that silently treats "no position" as zero return would have buried all three.

With the space cleaned up, I reran the LLM agent. This time it clustered on factor-residual daily and swept the lookback itself:

factor_residual/1d/lb70/reb35 +67.1% factor_residual/1d/lb50/reb25 +65.9% factor_residual/1d/lb40/reb20 +6.5% factor_residual/1d/lb30/reb15 -26.6% Enter fullscreen mode Exit fullscreen mode

The model drew the line itself. Long lookback positive, short lookback negative, and it found the boundary from feedback alone — no knowledge of what the heuristic had found. Right family, right direction, just needed a clean space to work in.

So is the edge real? The honest answer is modest.

Across lookbacks from 40 to 80 days with monthly rebalance it stays net positive, roughly +46% to +84% over the backtest period. (The edge map above shows lb40 at only +6.5% because that sweep used a faster reb20 rebalance — different configuration, same lookback.) Positive in four of five years, down hard in 2023. Survives a pessimistic 30 bps per side. Annualized Sharpe around 0.9. A label-permutation null — which scrambles which coin gets which cross-sectional residual, testing whether the ranking actually carries signal — puts it at p = 0.048, right at the edge of significant.

A promising but unconfirmed lead. The parameters and family were selected on these same results, so the p-value doesn't account for the full search process. Not a confirmed strategy — the backtest window is used up.

I wrapped the model in a hybrid proposer — the LLM picks the family and timeframe, but on alternating turns it sweeps the lookback and rebalance around the current best.

It found a peak at a 50-day lookback with monthly rebalance: +123.7% net.

(Round 2 showed lb50 at +65.9% with reb25. Same lookback, different rebalance, and now optimizing both jointly — that's where the extra 60 points came from. Also the best of 35 configs, not the average.)

This is a trap, and one column gives it away:

* 20 bootstrap runs is preliminary; treat the positive-run counts as directional, not precise.

Magnitude doubled. Breadth barely moved.

You can inflate magnitude by picking better parameters. Breadth doesn't work that way. It didn't move.

Here's the problem with reporting the best of 35 configs: even if none of them had any real edge, random variation across 35 tries would still produce one winner. A deflated Sharpe corrects for this — it asks whether the best result clears a significance bar that already accounts for the search. Ours came back at 0.70 against a required 0.95. The best config didn't clear the bar — which doesn't prove there's no edge, but means we can't claim significance after a 35-way search.

Overfitting has known cures. I hadn't reached for any of them.

Stop picking a winner. Average the whole plateau instead.

I took all factor-residual configs on daily bars with lookback between 40 and 80 — the range where the sign stayed consistent — and averaged them without picking a winner.

The plateau range itself came from inspecting these same results, so it doesn't fully eliminate selection bias. But it does reduce sensitivity to any single lucky configuration. The +75.8% is a less parameter-sensitive in-sample research result — not independent confirmation of alpha.

# Plateau: family=factor_residual, bar=1d, lookback in [40..80] # Range chosen where sign was consistent — but still data-derived, not pre-specified. plateau = [ r for r in results if r. family == " factor_residual " and r. bar_size == " 1d " and 40 <= r. lookback <= 80 ] ensemble_signal = np. mean ([ r. signal for r in plateau ], axis = 0 ) Enter fullscreen mode Exit fullscreen mode I equal-weighted the factor-residual signal across lookbacks from 40 to 80 days, one combined strategy, monthly rebalance.

Result: +75.8% net, annualized Sharpe 0.61, positive in 62% of windows.

Both configurations target the same hypothesized signal, but the ensemble isn't riding a single lucky (lb, reb) combination. That's what the bootstrap difference is showing.

I built a cheap agent with the right instinct and the wrong calibration. A harness that caught three silent bugs by returning a flat book as flat. An optimizer that used the clean space to inflate a number by 60%. And an ensembling step that separated a promising research signal from an inflated backtest result.

Run a heuristic baseline first. Until the search space was clean, it beat the model. Don't skip it just because the LLM reasons better.

The agent is only as honest as the thing that scores it. The judge comes first. The proposer is easy to swap out.

Once the proposer can optimize, it will. Watch breadth, not magnitude. And when you can average the plateau instead of picking the best config, do it — the gap between +124% and +76% was entirely parameter sensitivity.

Give a language model a truthful referee and a small enough space and it will find the corner worth looking at. It won't do it on the first try. It won't necessarily beat a simple heuristic, especially when the search space contains silent bugs. And once it can optimize, it will hand you a bigger number than the edge is worth — until you correct for the search.