Systematic traders do one thing almost no discretionary trader copies. Before they trust a rule, they hide part of their own data from themselves.
They find the rule on the first chunk, then run it on the chunk they never looked at. If it survives, they take it seriously. If it doesn't, they drop it.
That's the whole technique. No code, no platform, no backtesting engine, though almost everything written on this assumes you own one. You need a journal with dates in it, and a rule about which rows you're allowed to open.
The two words, in plain language
In-sample is the data you used to find the rule. The part you sliced and filtered and stared at on a Sunday afternoon.
Out-of-sample is data you deliberately did not look at while searching. You set it aside first, then open it once at the end to see whether the thing you found is still there. Also called a holdout.
Both words exist because of one uncomfortable fact: any rule found by searching looks good on the data it was found on. That's not evidence, that's arithmetic. Go hunting for the best-performing subset of your trades and you will find one, whether or not anything real is in there. It's the core of why a journal overfits as easily as a backtest.
A holdout is the cheapest way anyone has found to tell a pattern apart from a coincidence.
Cut by date, not at random
Here's where I'll plant a flag, because most people get this wrong the first time. The obvious move is to shuffle your trades, put 70% in one pile and 30% in the other. Random, unbiased, feels scientific. Don't. For a trader's journal a random split is close to worthless.
Your trades are not independent. Trades from the same week are often the same idea twice, or the same move, or the same mood. Shuffle, and Tuesday's trade lands in the discovery pile while Wednesday's near-copy lands in the test pile. You end up "testing" on trades you already saw.
Markets change, and a random split hides it. If a setup worked beautifully for ten months and died in the last four, a random split smears the good period across both piles and everything looks fine. A date split shows you the death, which ties straight into why setups stop working with no warning shot.
You changed too. The trader who logged trade 20 is not the one who logged trade 200. Random splitting mixes old-you into the test set.
And the real question is a forward one. Not "does this rule describe my history" but "will it hold up next month." Only a chronological split is shaped like that.
So: sort by date, draw a line, and don't look right of the line until you've finished searching.
Where to put the cut
Start with the reserve ratio. Around 70/30 works for most journals: 70% of trades in front of the line for discovery, 30% behind it for the check. Not sacred. It's a compromise between having enough data to find something and enough left to test it.
Now the part people get wrong. Cut by trade count, not by calendar time. Trading frequency is never steady. Take a break, get busy at work, and six months might hold fifteen trades while another six holds ninety. Reserving "the last six months" can hand you a holdout of eleven trades. Sort by date, count to 70% of your rows, and put the line at whatever date that row falls on.
There's also a floor. A holdout only says something if enough qualifying trades land in it. My working rule: fewer than about 30 trades and you don't have a test, you have an anecdote with an opinion attached. Thirty is generous, too. It's a threshold for "worth a look," not "confirmed," and how many trades it takes to validate a setup walks through where the honest numbers sit. A narrow filter can starve a good holdout as well: if 28% of trades qualify, a 78-trade reserve leaves you 22.
A worked example, numbers and all
Imagine 260 trades over about eighteen months, logged in R. 1R is what you risked on the trade, so double that is +2R.
Sort by date. Trade 182 sits 70% of the way through and lands in early February. That's the line: 182 in-sample trades before it, 78 out-of-sample trades after, which stay closed.
On the 182 in-sample trades the baseline is +0.06R per trade. Basically flat, slightly negative once costs are honest.
Now the search. Session, higher-timeframe state, entry timing, volatility, the usual tags. Something surfaces: New York open, higher timeframe (HTF) trending. +0.44R over 51 trades, 48% win rate.
That looks like the answer. It always does. So I write it down word for word, dated: NY open plus HTF trending, entry on the first valid trigger, stop at structure, target 3R. Then I open the holdout.
Of the 78 held-back trades, 22 qualify. Result: +0.09R over 22 trades.
Gut reaction: it failed. Do the arithmetic instead.
Standard deviation is just how spread out your individual trade results are. Say this setup's sits around 1.6R, meaning a typical trade lands about 1.6R away from your average, which is normal when you mix small losses with occasional runners. Then the uncertainty on an average of 22 trades is 1.6 ÷ √22, about 0.34R. A rough two-standard-error band around +0.09R, which is a rough range the true number is likely to sit in given how few trades you have, runs from −0.59R to +0.77R.
That band contains +0.44R. It also contains zero, and it contains meaningfully negative.
So the holdout did not refute the rule. It also did not confirm it. It said "I can't tell," which is the most common honest answer a small holdout gives and the one nobody writes articles about. The right response isn't to trade it at size. It's to keep logging under that rule and check again in three months.
The uncomfortable version: eighteen months of trading, and the best your data can say about your best-looking rule is maybe. That's not a failure of method. That's what 260 trades buys you.
The rules that keep a holdout honest
A holdout is easy to break, and people mostly break it without noticing.
Write the rule down in full before you open the holdout. Dated, in words. Entry condition, filter, stop, target, management. If your future self can read it and still argue about whether a trade qualified, it isn't written properly yet.
Look once. The instant you check, that data is in-sample. It's spent.
Don't tune after the look. The holdout comes back weak, you think "ah, but if I'd excluded Fridays," and you have just used the holdout for discovery. This is the most common way the exercise dies, and it dies quietly.
Count how many rules you check against it. One pre-registered rule is a real test. Twelve rules with the best one reported recreates the original problem on smaller data. Counting your hypotheses applies to the confirmation step as much as to the search. Two candidates per holdout, maximum.
Don't move the line. Re-cutting until the split flatters you is the same sin in a different coat.
Keeping discovery and confirmation on genuinely separate trades is fussy and easy to fudge by accident, which is why we made it part of the discovery step in EdgeFlow instead of leaving it to willpower: the history splits chronologically, the search runs on the earlier part, the check on the later part. Below roughly forty tagged trades it won't split at all, because at that size both halves are anecdotes — it runs on everything and marks the output unverified instead. Same idea as doing it by hand, minus the temptation to peek.
What a failed check actually tells you
Say the holdout comes back clearly negative. Not ambiguous. Negative. There are three explanations, and they lead to different decisions.
- The rule was noise from the start. You searched thirty combinations and reported the winner. Statistically the most likely explanation, and the one nobody wants.
- The rule was real and the market changed. It worked, then conditions shifted. Happens constantly.
- The holdout was small and unlucky. With 22 trades and a wide distribution, a genuinely +0.3R rule can print −0.2R with nothing wrong.
The honest part: you usually can't tell which one you're in. Not from 22 trades. Anyone who claims they can separate "noise" from "regime change" on a couple of dozen trades is guessing with confidence.
What the failed check does tell you, unambiguously, is that you have no business sizing up on this rule. That's a result, and it's worth more than it feels like.
Think about the alternative. Without the holdout you'd have taken +0.44R at face value, raised risk, traded it six months and found out anyway, except with real money, and by then you'd be invested enough to explain the drawdown away. A month of held-back data is not a wasted month.
I'd rather kill four ideas a year with a holdout than trade one dead idea at full size for six months. Most of us have done the second thing at least once.
What a passed check does not prove
Suppose it works. The holdout comes back at +0.38R over 34 trades and you're delighted. Fair enough, that's more than most traders ever have. But be clear about what it is: one piece of evidence, from one period, on a small sample.
It doesn't prove the rule keeps working, and it doesn't mean the effect is the size you measured. You picked the rule for looking good, so that number is almost always optimistic. It also says nothing about conditions your holdout didn't contain. If those four months were all trending, you've learned nothing about ranges.
The next step is forward validation: lock the rule, trade it small, and let genuinely future trades pile up as a second holdout nobody can contaminate. Slower than a date split, but it's the cleanest evidence a discretionary trader can produce. The whole sequence sits in how to build and validate an edge.
You can also roll the line forward as trades accumulate. Discover on months 1 to 12, check on 13 to 16. Six months later, discover on 1 to 18, check on 19 to 22. A poor man's walk-forward, and it works fine by hand.
If you don't have enough trades to split
Plenty of people reading this have 60 trades. Splitting that gives a 42-trade search and an 18-trade check, and neither half carries a conclusion.
So don't split. Treat everything you have as discovery, form one hypothesis, write it down with today's date, and start collecting the holdout going forward. You're waiting instead of testing, which is less satisfying and a lot more honest.
While you wait, tag properly. A rule you can't reconstruct from your own notes can't be tested against anything. Recording each trade across its setup, execution, environment and management is what makes the question answerable when the data finally arrives.
Frequently asked questions
Is 70/30 the right split?
A reasonable default, not a law. What matters more is whether the holdout contains enough qualifying trades for your specific rule. If 70/30 leaves 8 trades behind the line, the ratio isn't the problem. The rule is too narrow for your sample.
Can I use the holdout again later?
Not for the same rule. Once you've looked, it's in-sample forever. You can add newly accumulated trades to build a fresh holdout and re-check.
Does this apply if I trade discretionary?
It applies more. A backtester has thousands of trades to absorb its accidents. You have a few hundred, and you're also the one who wants the answer to be yes.
If you'd rather not police the split by hand, EdgeFlow runs the chronological discover-then-verify step for you and ranks candidates by the cautious end of their range rather than the flattering one. Have a look at how it works.