Data Leakage Can Make a Customer Churn Model Look Better Than It Is

A machine-learning model can produce impressive test results and still fail when it is used on new data.

One reason is data leakage: information that would not be available at prediction time has found its way into the training or evaluation process. The model learns from clues about the answer, so its metrics look stronger than its real-world performance.

This can happen in ordinary analytics work, often without anyone deliberately doing anything wrong.

What leakage looks like

Imagine a company wants to identify customers who may cancel next month. The dataset contains customer activity, account details and a churn label.

If a feature records the date a customer cancelled, or a status updated after cancellation, it may reveal the outcome directly. A model trained on that feature could perform extremely well, but the information arrives too late to help prevent the churn.

Leakage can also be less obvious:

• A feature is calculated using the entire dataset before the train-test split.
• A random split places records from the same customer in both sets.
• Future observations are used to predict earlier outcomes.
• A label-related field remains in the feature table after data preparation.

The common issue is a mismatch between the information available during modelling and the information that will be available when predictions are made.

Write down the prediction moment

Before building features, define the moment when the prediction is supposed to happen.

For example:

At the end of each month, use information available up to that date to estimate whether a customer will churn in the following month.

This gives the data-preparation work a clear boundary. Features must be calculated using information available before the prediction moment. The target describes what happens afterward.

A simple way to organise the data is to create one row per customer and prediction date:

customer_id | prediction_date | feature values | churn_next_month

The actual column names depend on the dataset. The important part is keeping the feature period and outcome period separate.

Split data before fitting transformations

A common mistake is to calculate preprocessing values using the full dataset, then split the data afterward. For example, an imputer or scaler might learn statistics from both training and test records.

Instead, split first and put transformations inside a pipeline. Scikit-learn then fits those transformations on the training data during model fitting.

from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
    stratify=y,
)

model = Pipeline([
    ("imputer", SimpleImputer(strategy="median")),
    ("scaler", StandardScaler()),
    ("classifier", LogisticRegression(max_iter=1000)),
])

model.fit(X_train, y_train)

This example assumes all features in X are numeric. If the dataset includes categorical columns, those need appropriate encoding, ideally within the same pipeline.

Use a split that matches the real use

A random split is not always the right way to evaluate a model.

If the model will predict future churn, a chronological split is usually a more realistic test: train on earlier observations and evaluate on later ones. If customers have multiple rows, it may also be important to keep all records for a customer in the same split.

The right approach depends on how the data is structured and how the prediction will be used. The aim is to avoid an evaluation setup that gives the model access to information it would not have in practice.

Check suspiciously strong results

A very high score is not automatically wrong, but it’s a good reason to inspect the workflow.

Ask:

• Could any feature be recorded after the outcome?
• Were transformations fitted before the split?
• Do customer records appear in both training and test sets?
• Does the split reflect when predictions will be made?
• Are identifiers acting as shortcuts for the target?
• Can the features be collected at the intended prediction time?

These checks do not prove that a model is leakage-free, but they help uncover common sources of overly optimistic results.

Keep the data timeline visible

A practical modelling workflow should document when each feature is known and what period the target covers. This makes assumptions easier to review and helps another person reproduce the evaluation.

For a churn project, that might mean recording:

• The date each customer snapshot was created
• The period used to calculate behavioural features
• The future window used to define churn
• The rule used to split training and test data

The model score matters, but it only means something if the evaluation resembles the real task. By defining the prediction moment, separating past features from future outcomes and choosing a suitable split, teams can get a more honest picture of how their churn models may perform.