Data leakage: the reason a great model fails in production

Data leakage is what happens when information that would not be available at prediction time sneaks into training. The model looks excellent in your notebook and disappoints the moment it meets real data. It is one of the most common reasons a "great model" fails in production, and it is often invisible because the score keeps going up.

The idea in one sentence

If your evaluation used information the model will not have when it is actually used, the evaluation is not measuring what you think it is.

Four common forms

1. Target leakage. A feature is a consequence of the outcome. Predicting whether a customer will cancel using a column like "cancellation_reason", or predicting loan default using "days overdue", hands the answer to the model. It looks like a strong feature because it is the label in disguise.
2. Train-test contamination. Test information reaches training. The classic case is fitting a scaler or imputer on the whole dataset before splitting, so test statistics shape the training data.
3. Duplicates across the split. The same or near-identical rows appear in both training and test sets, so the model is graded on things it has memorised.
4. Temporal leakage. Using the future to predict the past: random splits on time-ordered data, or features computed with information from after the moment of prediction.

A fifth is subtler: group leakage, where several rows come from the same person, device or document and are spread across both sides of the split.

Spotting it

Leakage leaves fingerprints:

• A score that is suspiciously high for a hard problem.
• One feature with outsized importance that you cannot explain by domain knowledge.
• A big drop between validation and real-world performance after deployment.
• Performance that collapses when you remove a single column.

Trust the surprise. When a result is much better than expected, look for leakage before celebrating.

Preventing it

Split first, then transform. Anything that learns from data (scalers, imputers, encoders, feature selectors, target encoding) must be fitted on the training portion only and then applied to the rest. A pipeline enforces this automatically.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score

pipe = Pipeline([
    ("impute", SimpleImputer(strategy="median")),
    ("scale", StandardScaler()),
    ("model", RandomForestClassifier(random_state=0)),
])
scores = cross_val_score(pipe, X_train, y_train, cv=5)

Inside cross-validation, each fold refits the imputer and scaler on that fold's training part, so nothing leaks from the validation part.

Ask the timing question for every feature. "At the exact moment we need this prediction, will this value already exist, and be final?" If not, exclude it or recompute it as of that moment.

Split the way you will be used. Predicting the future means a time-based split. Predicting for new customers means grouping by customer. Make the test set resemble deployment.

Deduplicate before splitting, including near-duplicates (same text with different whitespace, images with slight edits).

Keep a real hold-out. Data you never touched during development is your best protection against your own optimism.

A worked example

You predict whether a hospital patient will be readmitted. One feature, discharge_summary_length, is very predictive. But summaries for readmitted patients are longer because they were updated after the readmission. At prediction time (at discharge) the final summary does not exist yet. The feature encodes the future. Removing it lowers the score and makes the model honest.

Common mistakes

1. Normalising the whole dataset, then splitting.
2. Random-splitting time-ordered data.
3. Building features from aggregates that include the test period.
4. Using a "cleaned" dataset that already had label information baked in.
5. Repeatedly tuning against the test set (a slower kind of leakage).

A checklist

• Was every learned transformation fitted on training data only?
• Does each feature exist, final, at prediction time?
• Do related rows stay on the same side of the split?
• Are duplicates removed before splitting?
• Is there an untouched test set, opened once?
• Have we investigated any result that seems too good?