Train, validation and test sets: why three?

Almost every machine-learning mistake that reaches production starts the same way: someone reported a score that was too good to be true. A model that scored well on data it had effectively already seen tells you nothing about how it will behave next week. Splitting your data three ways is the simplest defence, and it is worth understanding properly rather than copying from a tutorial.

The three sets and their jobs

The training set is where the model learns. The validation set is your workbench: every time you ask "did that change help?", you look at validation performance. The test set is sealed. You open it once, after all your decisions are made, and report the number without going back.

Why two sets are not enough

Suppose you only have train and test. You try ten model types and pick the one with the best test score. That sounds harmless, but you have just used the test set to make a decision. With enough attempts, some model will look good on it by luck, and the number you report is optimistic. The more choices you make against a dataset, the more that dataset stops being independent evidence.

The validation set absorbs that selection pressure. It is allowed to become slightly "used up", because the test set stands behind it, untouched.

Rule of thumb: any dataset you look at while making a decision belongs to the development side of the ledger. Only data that played no part in any decision can give an unbiased estimate.

A minimal, correct split

from sklearn.model_selection import train_test_split

# First carve off the test set, then split what is left.
X_dev, X_test, y_dev, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y
)
X_train, X_val, y_train, y_val = train_test_split(
    X_dev, y_dev, test_size=0.176, random_state=42, stratify=y_dev
)

The second test_size is 0.176 because 15% of the remaining 85% gives roughly 15% overall, leaving about 70 / 15 / 15. stratify keeps the class balance similar in every part, which matters when one class is rare. Fixing random_state makes your split reproducible, so a colleague gets the same rows.

When a random split is the wrong split

A random shuffle assumes each row is independent. Often it is not.

• Time series. If you are predicting next month, training on the future and testing on the past is cheating. Split by time: train on earlier dates, validate on the next block, test on the last.
• Grouped rows. Several rows from the same customer, patient or device? Put the whole group on one side, or the model will memorise the person and look brilliant on their other rows. Use GroupShuffleSplit or GroupKFold.
• Duplicates and near-duplicates. The same document, image or transaction appearing twice can land in both train and test. Deduplicate before you split.

The question to ask is always: when this model is used for real, what will be different from the data it trained on? Make your test set differ in the same way.

Cross-validation, and where it fits

When data is small, a single validation set is noisy. K-fold cross-validation rotates the validation role across several folds and averages the result, so every row is used for validation exactly once. It replaces the fixed validation set, not the test set. You still hold out a test set first.

from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

model = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))
scores = cross_val_score(model, X_dev, y_dev, cv=5)
print(scores.mean(), scores.std())

Putting the scaler inside the pipeline matters: it is refitted on each training fold, so nothing from the validation fold leaks into preprocessing.

Common mistakes

1. Fitting scalers, imputers or encoders on all the data before splitting. Fit on training data only, then apply to the rest.
2. Tuning until the test score looks good. If you peek and adjust, the test set has become another validation set. Get a fresh one or accept the number is optimistic.
3. Splitting after feature selection that used the labels. Selecting "the most predictive features" on the full dataset leaks the answer.
4. Forgetting the baseline. A model that predicts the majority class is your floor. If your model barely beats it, the split is not the problem; the signal is weak.

A checklist before you trust a number

• Was the test set touched only once, at the end?
• Do related rows (same person, same session, same document) stay together?
• Does the split respect time if the real task looks forward?
• Is all preprocessing fitted on training data only, ideally inside a pipeline?
• Do you know the score of a trivial baseline on the same test set?

Try it yourself

Take any small tabular dataset. Train a decision tree with no depth limit and report accuracy on the training set, then on a properly held-out test set. Then repeat with a random split that deliberately puts duplicated rows on both sides. Compare the gap. Seeing the same model look brilliant and then ordinary is the fastest way to remember why the split exists.