Precision, recall and choosing the right metric

"The model is 97% accurate" is one of the least informative sentences in machine learning. Accuracy counts how often the model is right, and says nothing about which mistakes it makes. When one outcome is rare, that gap is exactly where the problems hide.

Why accuracy misleads

Imagine a screening task where only a small fraction of cases are positive. A model that always answers "negative" is correct almost every time, and completely useless. Accuracy rewards it because most cases really are negative.

The fix is to look at the four things that can happen for a positive/negative decision:

Every useful classification metric is a different way of combining these four counts.

Precision and recall in plain words

• Precision = TP / (TP + FP). Of everything the model flagged, how much was really positive? High precision means few false alarms.
• Recall = TP / (TP + FN). Of everything that was really positive, how much did the model find? High recall means few misses.

They pull against each other. A model that flags everything has perfect recall and terrible precision. A model that flags only the single most obvious case has great precision and poor recall.

Choosing by the cost of being wrong

The right metric follows from the decision the model supports, not from habit.

• A missed case is expensive (fraud that slips through, a fault that is not detected): favour recall, and accept more false alarms.
• A false alarm is expensive (blocking a legitimate customer, sending a wrong warning to thousands of people): favour precision.
• Both matter and classes are imbalanced: look at F1, the harmonic mean of precision and recall, or better, the whole precision-recall curve.

Write the sentence down before you pick the metric: "If the model misses a real case, the cost is ___. If it raises a false alarm, the cost is ___." The bigger blank tells you which side to protect.

The threshold is a dial, not a constant

Most models output a score, and a threshold turns it into a yes/no. Moving the threshold trades precision for recall.

from sklearn.metrics import precision_recall_curve

scores = model.predict_proba(X_val)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_val, scores)

# Pick the lowest threshold that keeps precision at or above 0.90
for p, r, t in zip(precision[:-1], recall[:-1], thresholds):
    if p >= 0.90:
        print(f"threshold={t:.2f}  precision={p:.2f}  recall={r:.2f}")
        break

Choosing the threshold on the validation set, and reporting the result on the untouched test set, keeps the estimate honest.

Beyond a single number

• Confusion matrix. Print it. It shows where the errors are, which a single score hides.
• Per-class metrics. In a multi-class problem, macro-averaging treats every class equally; micro-averaging weights by frequency. Say which you used.
• PR curve versus ROC curve. ROC curves can look flattering when negatives vastly outnumber positives, because false-positive rate stays small even when false positives are numerous compared with true positives. Precision-recall curves expose that.
• Calibration. If the model says "80% likely", are about eight in ten such cases actually positive? A model can rank well and still be poorly calibrated.

Common mistakes

1. Reporting accuracy on imbalanced data with no baseline.
2. Tuning the threshold on the test set.
3. Averaging metrics across classes without saying how.
4. Treating F1 as always right: it assumes precision and recall matter equally, which is rarely true in the real decision.

Try it yourself

Build a dataset where 2% of rows are positive. Train any classifier and print accuracy, precision, recall and the confusion matrix. Then move the threshold from 0.5 to 0.2 and 0.8 and watch how the numbers trade off. Finally, write one sentence describing which error your imagined users would hate more, and choose the threshold that reflects it.