Why ML Trading Systems Fail Before Training Even Starts

A developer building StratCraft, an ML-based trading system, found that a hidden data insufficiency caused Run 51 of a model sweep to fail before training could begin. Three separate components — the data puller, fold planner, and refusal checker — each held conflicting assumptions about how much historical data was required. The experience highlighted a broader pattern: in ML trading systems, the majority of meaningful work happens outside the model training step itself. Rather than predicting exact returns, the system focuses on ranking assets by signal scores and testing whether higher-ranked assets consistently outperform lower-ranked ones. The author argues that model.fit() is often the simplest line in such a project, while data integrity, label design, selection bias, and signal combination represent the real engineering challenges.