A better score can hide a worse experiment

Decide what the experiment is testing

Imagine a fictional model predicting whether an order will arrive late. The experiment should approximate the information available when the prediction is made. A delivery timestamp recorded after arrival is not a useful input at that moment, even if it creates an impressive development score.

Split before learning transformations

Scikit-learn recommends splitting training and test data before preprocessing. Fitting a scaler or selecting features on the whole dataset lets information from the test set influence training. Fit learned transformations on training data, then apply those fitted transformations to held-out data. Pipelines help keep those steps together during cross-validation.

Write the boundary into the workflow

This conceptual sequence makes the distinction visible. It is a plan for an experiment rather than a runnable training script; the split strategy and metric still depend on the question being tested.

training data → fit preprocessing → fit model
held-out data → transform only   → evaluate

Do not fit preprocessing on the held-out set.

Inspect features as carefully as scores

For the hypothetical delivery model, list when each input becomes available. Ask how a value was produced and whether it could contain information about the target. Keep the reasoning with the experiment record. A small, explainable feature set is a useful starting point for finding mistakes before trying a larger model.

Keep a review record

A useful experiment note includes more than the final score:

• The prediction moment and target definition.
• How training and evaluation records were separated.
• Which transformations learned parameters from training data.
• A simple baseline and the chosen evaluation metric.
• Known limitations and the next question to investigate.