A checklist for training a neural network that works

Training a neural network that works is less about clever architectures and more about disciplined debugging. Most failed runs come from a small set of avoidable problems. This checklist is a practical order of operations: start simple, prove each piece works, and change one thing at a time.

Before any training

1. Look at your data. Print samples. Check labels are correct, images are the right way up, text is decoded properly, classes are balanced. Bugs here poison everything downstream.

2. Set up a fair split. Training, validation and a sealed test set, with related examples kept together and no leakage.

3. Choose a baseline. A trivial model (majority class, mean prediction) and a simple one (logistic regression or a small network). If a huge network barely beats these, the problem is elsewhere.

4. Normalise inputs. Scale features, or normalise images with training-set statistics. Unscaled inputs make optimisation slow and unstable.

Prove the pipeline works: overfit a tiny batch

Take 8 to 32 examples and train on only those. A correct setup should drive the loss close to zero and reach near-perfect accuracy on them. If it cannot, you have a bug, not a tuning problem.

xb, yb = next(iter(train_loader))
model.train()
for step in range(300):
    optimizer.zero_grad()
    loss = loss_fn(model(xb), yb)
    loss.backward()
    optimizer.step()
    if step % 50 == 0:
        print(step, loss.item())

Typical causes when it fails: wrong loss for the task, labels not matching outputs, forgetting zero_grad(), a frozen layer, or an activation in the wrong place.

Check the basics of the loop

• Initial loss makes sense. For a classifier with C balanced classes and a sensible initialisation, the starting cross-entropy should be near log(C). A very different value points to a bug.
• Shapes and dtypes are as you think. Print them.
• The model is in the right mode. Training and evaluation modes differ for dropout and batch normalisation.
• Data are shuffled each epoch for training.
• Gradients flow. Check that gradients are non-zero for all trainable parameters, and not exploding.

Choose and tune sensible settings

Change one thing at a time and keep notes.

• Learning rate is the most important setting. Too high: loss jumps or becomes NaN. Too low: progress crawls. Try values across orders of magnitude, and consider a schedule (warm-up, then decay).
• Optimiser. Adam-style optimisers are forgiving defaults; SGD with momentum can generalise well with tuning.
• Batch size. Affects noise, memory and speed; interacts with learning rate.
• Initialisation and normalisation layers help deeper networks train.
• Loss and output activation must match (for example logits with cross-entropy, not a softmax followed by a loss that applies its own).

Read the curves

Regularise when you overfit

• Weight decay, dropout, and early stopping.
• Data augmentation suited to the data (flips and crops for images, but not where they change the label).
• More or cleaner data. Often the strongest fix.
• A simpler model.

Evaluate honestly

Report results on the sealed test set once. Look at per-class metrics and error examples, not only the overall number, and compare against your baseline. Check calibration if the probabilities are used for decisions.

Keep it reproducible

• Set random seeds and record library versions.
• Log configuration, data version, metrics and code revision for every run.
• Save checkpoints and the best model by validation performance.

A short debugging order

1. Verify the data and labels.
2. Verify the pipeline by overfitting one small batch.
3. Verify the loss and initial value.
4. Get the training loss going down.
5. Then close the gap to validation with regularisation.
6. Only then tune and scale up.

Common mistakes

1. Tuning a model before proving the pipeline can overfit a tiny batch.
2. Changing several things between runs.
3. Evaluating on training data or peeking at the test set.
4. Forgetting evaluation mode.
5. Ignoring the baseline.

Return to this list whenever a run misbehaves. Most problems announce themselves at one of these steps.