Main Challenges of Machine Learning

Training a machine learning model is rarely as straightforward as feeding data into an algorithm and calling it done. In practice, practitioners spend the majority of their time wrestling with a predictable set of problems — most of which fall into two broad categories: bad data and bad algorithms. Understanding these challenges before you encounter them is the hallmark of a skilled ML engineer.

This chapter is a systematic tour of the nine most common and consequential challenges in machine learning. For each one, we cover what it is, why it happens, how to detect it, and how to fix it — with real code examples throughout.

Insufficient Training Data

Too few examples to learn from

Non-Representative Data

Sampling bias and skewed distributions

Poor-Quality Data

Noise, errors, and missing values

Irrelevant Features

Too many uninformative inputs

Overfitting

Memorising training data, failing on new data

Underfitting

Model too simple to capture patterns

Bias-Variance Tradeoff

Balancing complexity against generalisation

Data Mismatch

Train distribution differs from production

No Free Lunch Theorem

No single best algorithm for all problems

Challenge 1: Insufficient Quantity of Training Data

Data Challenge Not Enough Examples to Learn From

Machine learning algorithms need data the way an engine needs fuel. Without a sufficient quantity of training examples, even a sophisticated model cannot identify reliable statistical patterns. It will instead latch onto coincidences in the small sample and fail to generalise.

A landmark 2001 study by Microsoft researchers Banko and Brill showed that for natural language disambiguation tasks, simple algorithms trained on massive datasets consistently outperformed complex algorithms trained on small datasets. This led to the famous phrase in the ML community: "It's not who has the best algorithm that wins. It's who has the most data."

The amount of data needed depends heavily on the task complexity:

Problem Type	Typical Minimum Data	Notes
Simple linear classification	50 – 500 examples	Low dimensionality, clear decision boundary
Tabular classification / regression	1,000 – 10,000 examples	Rule of thumb: 10× examples per feature
Image classification (CNN)	10,000 – 100,000+	Transfer learning reduces this significantly
Large language models	Billions of tokens	GPT-3 trained on ~300 billion tokens

Data Augmentation (flip, rotate, crop images) Transfer Learning (reuse pretrained weights) Synthetic Data Generation Semi-Supervised Learning Active Learning (query most uncertain samples)

The following code demonstrates how model accuracy grows dramatically as training data size increases — a pattern known as the learning curve:

Python — Learning Curve: Data Size vs. Accuracy

1import numpy as np
2from sklearn.datasets     import load_breast_cancer
3from sklearn.ensemble     import RandomForestClassifier
4from sklearn.model_selection import learning_curve
5
6X, y = load_breast_cancer(return_X_y=True)
7
8# Evaluate accuracy at various training set sizes
9train_sizes, train_scores, val_scores = learning_curve(
10    RandomForestClassifier(n_estimators=100, random_state=42),
11    X, y,
12    train_sizes = np.linspace(0.05, 1.0, 10),  # 5% → 100% of data
13    cv          = 5,
14    scoring     = 'accuracy'
15)
16
17# Print validation accuracy at each data size
18for size, val in zip(train_sizes, val_scores.mean(axis=1)):
19    print(f"Training samples: {size:4.0f} | Validation accuracy: {val:.3f}")
20# Training samples:   28 | Validation accuracy: 0.907
21# Training samples:   85 | Validation accuracy: 0.934
22# Training samples:  142 | Validation accuracy: 0.951
23# Training samples:  455 | Validation accuracy: 0.964  ← more data, better model

Rule of thumb: For tabular data, aim for at least 10 training examples per feature. If you cannot collect more data, use cross-validation to maximise the use of what you have, and consider transfer learning or regularisation to reduce the data requirement.

Challenge 2: Non-Representative Training Data

Data Challenge Sampling Bias and Skewed Distributions

Even a large dataset can be dangerously misleading if it does not represent the population the model will be deployed on. This is called sampling bias, and it is one of the most insidious problems in ML because the model trains successfully and only fails silently in production.

A textbook historical example: in the 1936 US presidential election, the Literary Digest magazine conducted a poll of 2.4 million people and confidently predicted Alf Landon would beat Franklin Roosevelt. Roosevelt won in a landslide. The Digest had polled their own subscribers — who were disproportionately wealthy and Republican. More data did not help when the data was biased.

Real-world ML examples of sampling bias:

Amazon Hiring Algorithm (2018)

Trained on 10 years of CVs — predominantly from male engineers. The model learned to penalise words like "women's" (e.g., "women's chess club") and downgraded graduates of all-female colleges. Amazon scrapped the tool.

Google Photos (2015)

Object recognition model trained mostly on light-skinned faces failed to correctly label dark-skinned individuals. Caused significant public harm and highlighted the cost of non-representative training data.

Medical Imaging

Skin cancer detection models trained primarily on images from light-skinned patients underperform on darker skin tones. Dangerous when deployed globally without understanding the demographic mismatch.

Stratified Sampling (preserve class proportions) Collect diverse and representative data Fairness Auditing across demographic groups Resampling and class weighting Adversarial Validation (detect train/test distribution mismatch)

Python — Stratified Sampling to Ensure Representation

1from sklearn.model_selection import train_test_split
2from sklearn.datasets       import load_breast_cancer
3import numpy as np
4
5X, y = load_breast_cancer(return_X_y=True)
6
7# Without stratification — class ratios may not be preserved
8X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
9print(f"Unstratified — class 1 ratio in test: {y_te.mean():.3f}")
10
11# With stratify=y — class proportions are guaranteed to match overall dataset
12X_tr, X_te, y_tr, y_te = train_test_split(
13    X, y, test_size=0.2, random_state=42, stratify=y  # the key argument
14)
15print(f"Stratified   — class 1 ratio in test: {y_te.mean():.3f}")
16print(f"Original     — class 1 ratio overall: {y.mean():.3f}")
17# Stratified — class 1 ratio in test: 0.627
18# Original   — class 1 ratio overall: 0.627   ← perfect match

Challenge 3: Poor-Quality Data

Data Challenge Noise, Errors, Outliers, and Missing Values

Garbage in, garbage out. Real-world data is almost always imperfect. If your training data contains mislabelled examples, measurement errors, or corrupted values, the model will faithfully learn those mistakes. Data cleaning typically consumes 60–80% of a data scientist's time on any real project.

Label Noise

Examples that are mislabelled — a spam email marked as "not spam" confuses the classifier. Even 5–10% label noise can significantly degrade accuracy.

Missing Values

Fields with NaN or null entries. Most algorithms cannot handle missing data natively. Requires imputation or removal strategies.

Outliers

Extreme values that are either genuine (a billionaire's salary in income data) or erroneous (age = 999). Can skew the model disproportionately.

Duplicates

Repeated rows cause the model to over-weight certain examples. Particularly harmful when the same record appears in both training and test sets, inflating evaluation metrics.

Remove or cap outliers (IQR, Z-score) Impute missing values (median, KNN, MICE) Deduplicate the dataset before splitting Cross-check labels with multiple annotators Use robust loss functions (Huber Loss)

Python — Data Quality Audit Pipeline

1import pandas as pd
2import numpy as np
3
4def audit_data_quality(df):
5    """Run a systematic data quality report."""
6    print("=" * 55)
7    print(f"Shape          : {df.shape}")
8
9    # 1. Missing values
10    missing = df.isnull().sum()
11    print(f"Missing values : {missing[missing > 0].to_dict()}")
12
13    # 2. Duplicates
14    dups = df.duplicated().sum()
15    print(f"Duplicate rows : {dups} ({dups/len(df)*100:.1f}%)")
16
17    # 3. Outlier detection via IQR (numeric columns)
18    for col in df.select_dtypes(include=np.number).columns:
19        Q1, Q3 = df[col].quantile([0.25, 0.75])
20        IQR      = Q3 - Q1
21        outliers = ((df[col] < Q1 - 1.5*IQR) | (df[col] > Q3 + 1.5*IQR)).sum()
22        if outliers > 0:
23            print(f"  Outliers [{col}]: {outliers} rows")
24    print("=" * 55)
25
26# Usage
27df = pd.read_csv("your_dataset.csv")
28audit_data_quality(df)

Challenge 4: Irrelevant Features

Data Challenge Too Many Uninformative or Redundant Inputs

Adding features to a model does not always improve it. Irrelevant features add noise, increase training time, consume memory, and — crucially — can actually degrade model performance. This phenomenon is particularly pronounced in high-dimensional data and is related to the curse of dimensionality (covered in Chapter 9).

The solution is feature engineering — the art and science of selecting, transforming, and creating the features that are most informative for the task. Good feature engineering is often the difference between a mediocre model and a state-of-the-art one.

Filter Methods (correlation, mutual information) Wrapper Methods (Recursive Feature Elimination) Embedded Methods (Lasso, Tree Feature Importance) Dimensionality Reduction (PCA, UMAP) Domain knowledge to engineer meaningful features

Python — SelectKBest: Finding the Most Informative Features

1from sklearn.feature_selection import SelectKBest, mutual_info_classif
2from sklearn.datasets         import make_classification
3from sklearn.pipeline          import Pipeline
4from sklearn.ensemble          import RandomForestClassifier
5from sklearn.model_selection   import cross_val_score
6
7# 50 features, only 5 are actually informative
8X, y = make_classification(
9    n_samples=1000, n_features=50, n_informative=5,
10    n_redundant=5, random_state=42
11)
12
13# Model A: all 50 features (includes 40 noisy ones)
14all_features_score = cross_val_score(
15    RandomForestClassifier(random_state=42), X, y, cv=5
16).mean()
17
18# Model B: top 8 features selected via mutual information
19pipe = Pipeline([
20    ('select', SelectKBest(mutual_info_classif, k=8)),
21    ('clf',    RandomForestClassifier(random_state=42))
22])
23selected_score = cross_val_score(pipe, X, y, cv=5).mean()
24
25print(f"All 50 features : {all_features_score:.3f}")
26print(f"Top 8 features  : {selected_score:.3f}")
27# All 50 features : 0.841   ← noisy features hurt performance
28# Top 8 features  : 0.876   ← fewer, better features win

Challenge 5: Overfitting the Training Data

Model Challenge The Model Memorises Instead of Learning

Overfitting occurs when a model learns the training data too well — including its noise, random fluctuations, and coincidences — instead of the underlying true pattern. The result is a model that achieves near-perfect accuracy on training data but performs poorly on any new, unseen data.

Think of a student who memorises the exact textbook questions and their answers word for word. If the exam asks the same questions verbatim, they do brilliantly. But slightly rephrase a question, and they are completely lost — they never actually understood the underlying concepts.

When does overfitting occur?

Common Causes of Overfitting

Model Too Complex

High-degree polynomial, deep neural network, or unbounded decision tree with too many parameters relative to the training data.

Too Little Data

When you have a complex model and only a handful of training samples, the model finds spurious patterns that happen to fit those few examples but do not generalise.

Noisy Features

Many irrelevant features give the model more "coincidences" to latch onto. The model finds apparent correlations that are purely coincidental in the training set.

No Regularisation

Without constraints on model complexity (L1, L2 penalty, dropout, max depth), the model is free to grow arbitrarily complex to fit every training point.

Regularisation (Ridge L2, Lasso L1, ElasticNet) Simpler model (reduce degree, depth, parameters) Collect more training data Cross-validation for honest evaluation Early Stopping in neural networks Dropout (for deep learning)

Underfitting vs. Good Fit vs. Overfitting

Underfitting (High Bias)

The model is too simple (straight line) to capture the underlying curved pattern. Both training and validation errors remain high.

Good Fit (Sweet Spot)

The model captures the true underlying pattern. Training and validation errors are both low. The model generalises well to new data.

Overfitting (High Variance)

The model memorises every training point perfectly. Training error is near zero, but the model fails completely on new data it has not seen.

Dashed purple curve shows the true underlying pattern the data was generated from.

The classic diagnostic: If your training accuracy is much higher than your validation accuracy — say 99% vs 72% — your model is almost certainly overfitting. The gap between training and validation error is your primary signal.

The following example demonstrates overfitting with a very high-degree polynomial, and how Ridge regularisation (L2 penalty) brings it under control:

Python — Overfitting Demonstrated and Fixed with Regularisation

1import numpy as np
2from sklearn.preprocessing  import PolynomialFeatures
3from sklearn.linear_model   import LinearRegression, Ridge
4from sklearn.pipeline       import Pipeline
5from sklearn.model_selection import cross_val_score, train_test_split
6
7np.random.seed(42)
8X = np.random.rand(30, 1)
9y = np.sin(2 * np.pi * X).ravel() + np.random.normal(0, 0.3, 30)
10
11X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
12
13def evaluate(name, pipe):
14    pipe.fit(X_train, y_train)
15    tr  = pipe.score(X_train, y_train)
16    te  = pipe.score(X_test,  y_test)
17    print(f"{name:35s} | Train R²: {tr:+.3f} | Test R²: {te:+.3f}")
18
19# Underfitting: degree 1
20evaluate("Degree-1 (Underfitting)",
21          Pipeline([('p',PolynomialFeatures(1)),('r',LinearRegression())]))
22
23# Overfitting: degree 15 — memorises training noise
24evaluate("Degree-15 (Overfitting)",
25          Pipeline([('p',PolynomialFeatures(15)),('r',LinearRegression())]))
26
27# Fixed: Ridge regularisation controls complexity
28evaluate("Degree-15 + Ridge (Regularised)",
29          Pipeline([('p',PolynomialFeatures(15)),('r',Ridge(alpha=5))]))
30#
31# Degree-1  (Underfitting)          | Train R²: +0.241 | Test R²: +0.198
32# Degree-15 (Overfitting)           | Train R²: +0.998 | Test R²: -2.341  ← disaster
33# Degree-15 + Ridge (Regularised)   | Train R²: +0.876 | Test R²: +0.821  ← good

Key observation: The degree-15 polynomial achieves R² = 0.998 on training data — virtually perfect. On the test set it scores -2.341 (worse than a flat horizontal line). Adding Ridge regularisation with alpha=5 brings the test score up to 0.821 without changing the model's capacity — only its effective complexity.

Challenge 6: Underfitting the Training Data

Model Challenge The Model is Too Simple to Capture the Pattern

Underfitting is the opposite of overfitting. It occurs when the model is too simple — it does not have enough capacity or appropriate structure to capture the underlying relationships in the data. The signature: both training error and validation error are high. The model does badly on everything.

A classic example: trying to model a curved relationship (a sinusoidal wave) with a straight line. No matter how much data you provide, a linear model simply cannot represent a non-linear function.

Symptoms of Underfitting:

Training accuracy is low (the model cannot even fit the training data)
Validation accuracy is similarly low — close to training accuracy
Learning curves plateau quickly at a high error level
Model makes systematic, consistent errors (not random noise)

Use a more powerful model (higher degree, deeper network) Engineer better input features Reduce regularisation strength (lower alpha) Train for more epochs or iterations

Diagnosing Underfitting

Classic symptoms in your training logs:

Training accuracy ≈ 58%, Val accuracy ≈ 55%
Both errors are high and close together
No improvement after many epochs
Model predicts the same value for most inputs

Remedies to Apply

Steps to resolve underfitting, in order:

Try a more expressive algorithm (e.g., SVM, Random Forest, MLP)
Add polynomial or interaction features
Remove or relax regularisation constraints
Ensure features are properly scaled and encoded

Challenge 7: The Bias-Variance Tradeoff

Overfitting and underfitting are two sides of the same fundamental tension in machine learning: the bias-variance tradeoff. Formally, the total generalisation error of a model on unseen data can be decomposed into three components:

Decomposition of Generalisation Error

Bias²
Wrong assumptions

Variance
Sensitivity to data

Irreducible
Error
Inherent noise

Total
Error

Only Bias and Variance can be controlled through model choices. Irreducible error is inherent to the data-generating process.

Component	Definition	Cause	Direction
Bias	Error from wrong assumptions in the model. How far off are the model's average predictions from the true values?	Model too simple (underfitting)	Decreases as complexity increases
Variance	Error from excessive sensitivity to fluctuations in training data. How much do predictions change if you use a different training set?	Model too complex (overfitting)	Increases as complexity increases
Irreducible Error	Error from noise inherent in the data — labelling noise, missing variables, measurement error. Cannot be reduced by any model.	Properties of the data itself	Constant — cannot be reduced

The Bias-Variance Tradeoff Curve

Bias² Variance Total Error Irreducible Error

Diagnosing Bias and Variance with Learning Curves

Learning curves plot training and validation error against the number of training samples. They are an essential diagnostic tool to determine whether your model is suffering from high bias, high variance, or both.

High Bias (Underfitting)

What you see: Both curves converge to a high error. Small gap between them. Adding more data helps only marginally.
Fix: Increase model complexity or engineer better features.

High Variance (Overfitting)

What you see: Training error is very low, validation error is high. Large gap between the two curves.
Fix: Add more training data, apply regularisation, or use a simpler model.

Python — Estimating Bias and Variance Empirically

1import numpy as np
2from sklearn.model_selection import cross_val_score
3from sklearn.datasets       import load_breast_cancer
4from sklearn.tree            import DecisionTreeClassifier
5from sklearn.ensemble        import RandomForestClassifier
6
7X, y = load_breast_cancer(return_X_y=True)
8
9def bias_variance_proxy(model, X, y, cv=10):
10    """
    Train error ≈ proxy for Bias (if high, model underfits)
    Train error - Val error ≈ proxy for Variance (if large gap, model overfits)
    """
13    train_sc = cross_val_score(model, X, y, cv=cv, scoring='accuracy')
14    # Refit on each fold's training portion to get train accuracy
15    val_mean   = train_sc.mean()
16    val_std    = train_sc.std()
17    print(f"Val Accuracy: {val_mean:.3f} ± {val_std:.3f}")
18    print(f"Variance proxy (std): {val_std:.4f}  — higher means more variance")
19
20print("--- Unpruned Decision Tree (High Variance) ---")
21bias_variance_proxy(DecisionTreeClassifier(random_state=42), X, y)
22# Val Accuracy: 0.919 ± 0.037  — large variance across folds
23
24print("--- Random Forest (Lower Variance via Bagging) ---")
25bias_variance_proxy(RandomForestClassifier(n_estimators=100, random_state=42), X, y)
26# Val Accuracy: 0.964 ± 0.018  — much lower variance, higher accuracy

Challenge 8: Data Mismatch (Train-Serve Skew)

Generalisation Challenge The Training Distribution Differs from Production

A model can generalise perfectly to its validation set yet still fail dramatically in production — if the production data comes from a different distribution than the training data. This is called data mismatch or train-serve skew, and it is one of the most common silent killers of deployed ML systems.

Real-world examples:

A speech recognition model trained on studio-quality audio fails on noisy street recordings
A fraud detection model trained on 2020 transaction patterns fails to catch 2024 fraud tactics
An image classifier trained on high-resolution photos fails on smartphone camera images
A recommendation model trained on desktop behaviour misfires on mobile users

Andrew Ng's Train/Dev/Test split for data mismatch: Keep your test set and one dev set from the production distribution. Create a second dev set from the training distribution. If your model performs well on the training dev set but poorly on the production dev set, the problem is data mismatch — not overfitting.

Adversarial Validation (detect distribution shift) Domain Adaptation techniques Collect data from the production environment Continuous monitoring and retraining pipelines Use production-representative data in validation

Python — Detecting Distribution Shift with the KS Test

1import numpy as np
2from scipy.stats import ks_2samp
3
4np.random.seed(42)
5
6# Simulate a feature at training time (e.g., customer age, centred at 35)
7train_age = np.random.normal(loc=35, scale=8,  size=2000)
8
9# Simulate the same feature in production 18 months later (centred at 48)
10# The user base has aged / shifted — this is concept drift
11prod_age  = np.random.normal(loc=48, scale=10, size=1000)
12
13# Kolmogorov-Smirnov test: are these two samples from the same distribution?
14stat, p_value = ks_2samp(train_age, prod_age)
15
16print(f"KS Statistic : {stat:.4f}")   # 0 = identical, 1 = completely different
17print(f"P-value      : {p_value:.8f}")
18
19if p_value < 0.05:
20    print("WARNING: Significant distribution shift detected in feature 'age'.")
21    print("The model may underperform on current production data.")
22    print("Consider retraining with more recent data.")
23# KS Statistic : 0.4312
24# P-value      : 0.00000000
25# WARNING: Significant distribution shift detected in feature 'age'.

Challenge 9: The No Free Lunch Theorem

Theoretical Challenge No Single Algorithm is Best for All Problems

"Any two algorithms are equivalent when their performance is averaged across all possible problems."

— Wolpert & Macready, No Free Lunch Theorems for Optimization, 1996

The No Free Lunch (NFL) Theorem proves that, averaged across all possible datasets and tasks, no single machine learning algorithm outperforms any other. An algorithm that works brilliantly on one class of problems will necessarily be worse than random on some other class of problems.

In practice, this means there is no universally "best" algorithm. Random Forests do not always beat Neural Networks, and vice versa. The correct algorithm depends entirely on the structure of your specific problem and data. This is why practitioners must try multiple models and validate carefully.

Practical implications of the No Free Lunch Theorem:

Always baseline with a simple model (Logistic Regression, Decision Tree) before investing in complex ones
There is no substitute for understanding your data — its distribution, dimensionality, and noise level matter
Cross-validation is not optional — it is the only honest way to compare models on your specific problem
Domain expertise informs which model families to try first (e.g., tree-based for tabular, CNNs for images)

Try multiple algorithms and compare with cross-validation Use domain expertise to narrow algorithm choice Always establish a simple baseline first

A practical heuristic: For structured tabular data, tree-based ensemble methods (Random Forest, XGBoost) tend to perform well out of the box. For unstructured data (images, audio, text), deep learning is typically the starting point. But always validate on your specific data — never assume.

Practical Diagnosis and Remedy Reference

Use this table as a quick reference when your model is underperforming. Identify your symptom and apply the corresponding remedies.

Symptom Observed	Most Likely Cause	Primary Remedies
Train accuracy ≈ 60%, Val accuracy ≈ 58% (both low)	Underfitting / High Bias	Use more complex model; engineer better features; reduce regularisation
Train accuracy ≈ 99%, Val accuracy ≈ 72% (large gap)	Overfitting / High Variance	Add regularisation; collect more data; reduce model complexity; cross-validate
Train accuracy ≈ 95%, Val accuracy ≈ 94%, Prod accuracy ≈ 70%	Data Mismatch / Distribution Shift	Collect production-representative data; run KS tests on features; retrain regularly
Model performs well for some groups, poorly for others	Non-Representative Training Data	Stratified sampling; collect diverse data; fairness auditing; resampling
High accuracy on validation, poor on new time period	Concept Drift	Retrain on recent data; implement online learning; monitor drift metrics
Model performs inconsistently across different random seeds	High Variance / Instability	Use ensemble methods; increase training data; fix random seeds for reproducibility
Model trains fine but accuracy drops sharply in later epochs	Overfitting in Neural Networks	Apply early stopping; add dropout; reduce learning rate; use batch normalisation

A Comprehensive Model Health Check

The following function performs a rapid health check on any trained scikit-learn model, diagnosing the most common challenges automatically:

Python — Model Health Check: Diagnose All Common Challenges

1import numpy as np
2from sklearn.model_selection import cross_val_score, train_test_split
3from sklearn.metrics        import accuracy_score
4
5def model_health_check(model, X, y, task='classification'):
6    """Diagnose overfitting, underfitting, and instability."""
7    scoring = 'accuracy' if task == 'classification' else 'r2'
8
9    # 1. Train / test split scores
10    X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=42)
11    model.fit(X_tr, y_tr)
12    train_score = model.score(X_tr, y_tr)
13    test_score  = model.score(X_te,  y_te)
14
15    # 2. Cross-validation scores
16    cv_scores = cross_val_score(model, X, y, cv=10, scoring=scoring)
17    gap         = train_score - test_score
18
19    print(f"Train Score   : {train_score:.4f}")
20    print(f"Test Score    : {test_score:.4f}")
21    print(f"CV Mean±Std   : {cv_scores.mean():.4f} ± {cv_scores.std():.4f}")
22    print(f"Train-Test Gap: {gap:.4f}")
23
24    # 3. Automatic diagnosis
25    if   train_score < 0.80:
26        print("DIAGNOSIS: High Bias (Underfitting)")
27    elif gap > 0.10:
28        print("DIAGNOSIS: High Variance (Overfitting)")
29    elif cv_scores.std() > 0.05:
30        print("DIAGNOSIS: High model instability (variance across folds)")
31    else:
32        print("DIAGNOSIS: Model looks healthy — good fit.")

Key Takeaways

ML challenges fall into two broad categories: bad data (insufficient, biased, noisy, irrelevant) and bad models (overfitting, underfitting).
More data is not always the solution — if the data is biased or non-representative, adding more of the same problem makes things worse.
Overfitting is diagnosed by a large gap between training and validation error; underfitting is diagnosed by both errors being high and close together.
The Bias-Variance Tradeoff is fundamental: reducing one typically increases the other. The optimal model lives at the sweet spot of minimal total error.
Learning curves are your best diagnostic tool — they reveal whether you need more data, a more complex model, or regularisation.
Data Mismatch is the silent failure mode in production. Always ensure your validation set reflects the distribution of data the model will encounter at serving time.
The No Free Lunch Theorem mandates experimentation. Always benchmark multiple algorithms and validate rigorously for your specific problem.
Regularisation (Ridge, Lasso, Dropout) and cross-validation are the two most important tools for combating overfitting in practice.

What's Next?

In Chapter 1.4 — Testing and Validation, we will go deep on the methodologies that let you honestly evaluate how well your model will perform in the real world: holdout sets, K-Fold cross-validation, stratified splits, time-series validation, and the critical distinction between the validation set and the test set. We will also cover the subtle ways in which evaluation can be gamed — and how to prevent it.

1.3 Main Challenges of Machine Learning