Finding the Invisible String

The string nobody can see, but everyone can feel

A cricket coach has years of data on his bowlers: hours of practice per week, and average bowling speed. Plotted as a scatter plot, the dots trend upward — more practice, generally, though not perfectly, goes with higher speed. He wants more than “they seem related.” He wants to hand a new trainee a number of practice hours and get a prediction of what speed to expect.

The dots do not sit on a perfect line. Real relationships rarely do. Instead, they cluster into a rough diagonal band, as though a string runs through the cloud, taut between the bottom-left and the top-right, with every dot tugging gently at it.

The question is precise: among every possible straight line that could be drawn through this cloud, which one is actually the best one? “Looks about right” is not a method. We need something a machine can check without ever looking at the picture.

For a candidate line, each observation has a gap between its observed value and the value predicted by the line. That gap is the residual. Positive residuals sit above the line; negative residuals sit below it.

Least squares: finding the best line

Regression applies the same basic idea used earlier to understand variance: measure deviations, square them so positive and negative errors cannot cancel, and add them together. Here the “centre” is an entire line rather than a single number.

For every candidate line, calculate every residual, square each residual, and add the squared residuals. Somewhere among all possible slopes and intercepts is one line with the smallest possible total.

Every straight line is fully described by two numbers. The slope tells you how steeply it rises; the intercept anchors the line. In the chapter's cricket example, the illustrative relationship is approximately:

The slope of 0.8 means that each additional hour of weekly practice is associated with an average increase of about 0.8 km/h in predicted bowling speed across this dataset.

The warning label everyone skips

Imagine noticing that whenever an ice-cream truck's jingle plays, more people are wearing shorts. Does the jingle cause people to wear shorts? No: summer is a hidden variable producing both observations.

A clean regression relationship can therefore coexist with confounding, reverse causation, or other explanations. Naturally gifted bowlers might practise more and also bowl faster; coaching access might influence both.

Establishing genuine causation requires experimental machinery such as randomised assignment and control groups, or dedicated causal-inference methods. A strong fit is not proof of causation.

How good is “good,” really?

A least-squares line exists even when the underlying variables have little useful relationship. The fitted line therefore needs a second number that describes how much predictive variation it accounts for.

R-squared can be understood as the fraction of the original prediction error — the error made by simply guessing the overall mean for everyone — that the fitted line removes.

R² is therefore not “70% accurate.” It describes explained variation relative to a baseline predictor. The remaining variation can reflect talent, technique, conditioning, measurement noise, omitted predictors, and other factors not represented by the model.

Where the word “regression” actually came from

The name comes from Francis Galton's work on the heights of parents and their adult children. Exceptionally tall parents tended to have children who were taller than average, but somewhat shorter than their exceptionally tall parents. Exceptionally short parents showed the corresponding movement back toward the population average.

Galton called this pattern “regression toward mediocrity,” later softened to “regression toward the mean.”

The phenomenon is not a biological force pulling extreme values toward the middle. It follows naturally from imperfect correlation: an extreme value of one variable tends, on average, to be paired with a less extreme value of the other.

When the line breaks its own promise

A straight-line model is inappropriate when the outcome is constrained. Probability, for example, must remain between 0 and 1. A linear model can predict impossible values such as 1.15.

Logistic regression solves this by modelling a transformed quantity, log-odds, on an unrestricted linear scale and then converting the result back to probability with the logistic function.

Logistic coefficients are commonly interpreted through odds ratios. An odds ratio of 1.15 for age means each additional year multiplies the odds of the outcome by 1.15, while the corresponding change in raw probability depends on the starting probability.

The number the line was hiding: Pearson's r

Before fitting a line, we may want to ask how strongly two variables move together and in which direction. Pearson's correlation coefficient, r, is a unitless measure ranging from −1 to +1.

For a simple one-predictor linear regression, R² is the square of Pearson's correlation. Squaring removes the sign, so R² alone cannot tell you whether the relationship is positive or negative.

When one string isn't the whole story

Real prediction problems rarely have a single useful predictor. A medical researcher predicting blood pressure might need age, weight, sodium intake, family history, and other variables simultaneously.

In multiple regression, each coefficient describes the predictor's contribution holding the other included predictors fixed. This is a different quantity from the coefficient in a simple regression.

Adding predictors always increases or leaves unchanged the training R², even if the new variables are useless. Adjusted R² therefore penalises model complexity and only rises when an additional predictor improves the fit enough to justify its inclusion.

One idea, wearing three different outfits

The chapter's broader lesson is that regression does not need to be reinvented for every kind of outcome. A generalized linear model combines three components:

For counts, which cannot be negative, a log link can be used. The model works on the logarithm of the expected count and exponentiates the result, ensuring the predicted count remains positive.

This is one of the chapter's bridges toward machine learning: the model's output transformation is often what converts an unconstrained internal computation into a valid probability, count, or class.

Four datasets, one perfect disguise

In 1973, Francis Anscombe constructed four datasets with nearly identical summary statistics: the same means, variances, regression line, and R². Yet their actual structures are radically different.

One dataset is genuinely linear. Another is curved. A third is dominated by an influential outlier. The fourth has almost all its x-values clustered together, with one point determining the apparent slope.

The lesson is not that regression is unreliable. It is that a fitted model and its diagnostic checks are two halves of one complete practice.

Try it yourself: Part VI