Choosing a Measure of Success
Before choosing a model or deciding how to train it, state what success means and how success will be observed.
Start with the Target
Machine learning is not only the act of constructing a model. It is a workflow of connected decisions. You first define the real-world problem, its inputs, and its target output. Then you prepare the data, decide how success will be measured, develop a model, and evaluate whether the result solves the broader problem rather than merely fitting the training data.
Define What Counts
A measure of success is a metric used to evaluate model performance. It should connect to the higher-level goal of the project, such as the success of a business, rather than being selected only because it is convenient to calculate. Choosing the metric is therefore part of defining the machine learning problem, not an afterthought added after a model has been built.
| Decision | Question it answers | Role in the workflow |
|---|---|---|
| Problem definition | What real-world task are we solving? | Identifies inputs and target output |
| Measure of success | What result counts as good performance? | Defines the performance target |
| Evaluation protocol | How will we estimate current performance? | Defines the assessment procedure |
| Feature engineering | Which prepared inputs should the model use? | Creates useful features from prepared data |
These decisions are connected, but they answer different questions.
The problem type influences the metric choice. A balanced-classification problem, a class-imbalanced classification problem, a ranking problem, and a multilabel problem do not automatically share the same notion of success. The source identifies these as different problem settings, but the appropriate metric must be chosen from the higher-level goal of the particular task. The key habit is to ask what outcome matters before selecting a metric name.
Prepare the Data Path
Raw data does not move directly into a model as one undifferentiated step. Preparation can involve cleaning the available data, transforming it, preparing features, and separating data for training and evaluation. Together, these stages create a usable path from raw information to a model.
Data preparation and evaluation planning are related. Separating data for training and evaluation helps create a basis for judging how the model performs beyond the data used to train it.
Estimate Unseen Performance
An evaluation protocol is the procedure used to assess a metric and estimate current performance. The metric tells you what performance means; the protocol tells you how the available data will be used to estimate it. A meaningful metric with a weak protocol can produce a poorly supported progress estimate, while a careful protocol with a poor metric measures the wrong target carefully.
| Protocol | How it uses data | When to choose it |
|---|---|---|
| Hold-out validation | Sets aside part of the available data for validation | When plenty of data is available |
| K-fold cross-validation | Divides data into folds and evaluates on each fold while the other folds provide the remaining data | When there are too few samples for hold-out validation to be reliable |
| Iterated K-fold validation | Repeats K-fold cross-validation multiple times | When little data is available and highly accurate evaluation is needed |
Protocol choice depends largely on data availability.
Recognize Overfitting
After the problem, success measure, data preparation, and evaluation plan are defined, model development still has a central danger: overfitting. Overfitting occurs when a model becomes fitted too closely to the training data. Strong performance on the training data is not automatically the same as solving the broader machine learning problem.
Trace a New Scenario
Imagine a team wants to create a machine learning solution for a real-world task. The team has collected raw data, but it has not yet stated the target output, selected a measure of success, prepared the data, or considered overfitting. The correct response is not to begin by choosing a model. The team should move through the workflow in order, checking how each decision affects the next.
From Raw Task to Evaluation Plan
A team has raw data for a real-world machine learning task but has not yet defined its target, measure of success, preparation process, or evaluation approach.
- 1Define the problem
State the real-world task, identify the inputs, and specify the target output.
- 2Choose success
Select an observable measure that aligns with the higher-level goal rather than choosing a metric only for convenience.
- 3Prepare the data
Clean and transform the available data, prepare features, and separate data for training and evaluation.
- 4Choose the protocol
Use hold-out validation when data is plentiful, K-fold cross-validation when samples are limited, or iterated K-fold validation when little data is available and highly accurate evaluation is needed.
- 5Develop and inspect
Train a model while watching for the possibility that it fits the training data too closely.
The team now has a connected workflow in which the target, success measure, data path, evaluation procedure, and overfitting risk can be reasoned about together.
A project has a small dataset and needs a highly accurate estimate of model performance. Which evaluation protocol should the team consider, and why?
Hints
- Look for the protocol that repeats a fold-based evaluation.
- The source connects this choice with little data and highly accurate evaluation.
What do you think happens?
A project has plenty of data. Which protocol is the simple choice according to the workflow?
Reveal answer
Hold-out validation
Hold-out validation sets aside part of the available data for validation, and the source describes it as the simple choice when plenty of data is available.
Avoid Workflow Mistakes
Common mistake Choosing a model before defining the target output
What it looks likeThe team begins comparing models while the real-world task and desired output are still unclear.
Why it failsModel development is being started before the problem has been defined.
Do this insteadState the inputs and target output before evaluating models.
Common mistake Choosing a metric only because it is convenient
What it looks likeA team selects a measure without asking whether it reflects the higher-level goal.
Why it failsThe project may measure a convenient result rather than meaningful success.
Do this insteadChoose a measure that aligns with the higher-level goal.
Common mistake Confusing a metric with an evaluation protocol
What it looks likeThe team names a performance measure but does not decide how the available data will be used to estimate it.
Why it failsA metric defines what performance means, while a protocol defines the assessment procedure.
Do this insteadChoose both a meaningful metric and a suitable evaluation protocol.
Common mistake Treating training fit as proof that the broader problem is solved
What it looks likeA model fits the training data closely, so the team assumes the result will generalize to the broader task.
Why it failsThis is the danger of overfitting.
Do this insteadUse an evaluation procedure to make the risk of fitting the training data too closely visible.
Common mistake Using the same validation protocol regardless of data availability
What it looks likeThe team uses hold-out validation even when there are too few samples for the estimate to be reliable.
Why it failsProtocol choice depends largely on how much data is available.
Do this insteadUse hold-out validation with plentiful data, K-fold cross-validation with limited data, and iterated K-fold validation when little data and highly accurate evaluation are required.
Keep two questions separate throughout the project: What performance matters, and how will that performance be estimated? The first selects the measure of success. The second selects the evaluation protocol.
Workflow Checklist
- 1Define the real-world problem, its inputs, and its target output.
- 2Choose an observable measure of success that aligns with the higher-level goal.
- 3Clean and transform the raw data, prepare features, and separate training and evaluation data.
- 4Choose an evaluation protocol based largely on data availability.
- 5Develop the model while checking whether it is fitting the training data too closely.
- 6Interpret the evaluation result as evidence about the broader problem, not merely about training fit.
The central idea is simple: machine learning progress is meaningful only when the project has defined what success means and has a suitable way to estimate it. Metrics and evaluation protocols solve different parts of that problem. Data preparation creates the path to a model, and evaluation helps expose the difference between fitting training data and solving the broader task.
Key Takeaways
- Machine learning is a connected workflow of problem definition, success measurement, data preparation, feature engineering, training, and evaluation.
- A measure of success must be observable and aligned with the higher-level goal of the project.
- A metric defines what performance means, while an evaluation protocol defines how that performance is estimated.
- Hold-out validation suits plentiful data, K-fold cross-validation suits limited data, and iterated K-fold validation suits little data when highly accurate evaluation is needed.
- Overfitting occurs when a model fits the training data too closely, so training performance alone does not establish that the broader problem has been solved.