Understand the idea
Training data fit a model; validation data help choose among versions; a held-out test set estimates performance after those choices. Repeatedly choosing models based on the test set lets information about that test influence development. [S42] Overfitting means fitting observed examples in ways that fail to generalize well to unseen cases. [S43] Leakage and deployment mismatch are additional concerns: even an honestly held-out test can represent the wrong population or time period.
Worked example
An invented review classifier sees nearly identical reviews in both training and test sets. It scores highly by recognizing duplicates. Split by the entity or time boundary relevant to deployment, then test a genuinely new batch. Explain why a random row split was misleading here.
Where the idea stops
A holdout is necessary in many workflows but not sufficient for deployment reliability. There is no universal accuracy threshold, and repeated testing can contaminate an initially clean estimate. [S42, S43]
Connect it
Science and learning: passing a familiar test versus transferring knowledge.
Go deeper
How can a technically held-out dataset still be a poor deployment test?
Sources
Google Machine Learning Crash Course · Research reference · Date: Date not established
Explains training, validation, test splits and repeated-test contamination.
Link checked 2026-10-17 · Public webpage accessed; no specialist peer review of this brief.Google Machine Learning Crash Course · Research reference · Date: Date not established
Distinguishes fitting training examples from generalizing to new ones.
Link checked 2026-10-17 · Public webpage accessed; no specialist peer review of this brief.