You cannot measure everyone, so you measure some and reason about the rest. That step is where almost every argument about evidence lives — and where most of the ways to be wrong are hiding. This page builds a population whose true answer is known, then takes samples from it, so you can watch how often the honest procedure gets it right and how easily a dishonest one manufactures something.
The population below has a true average of exactly 100 — the page built it, so there is no doubt. Draw a sample and its average lands somewhere near, but not on it. Draw many and the misses form their own distribution, narrower than the population by a factor of √n. That is the single most useful formula in the subject: to halve your uncertainty you need four times the data.
Each horizontal line is one sample's confidence interval. The promise is not that the true value is 95% likely to be in your interval — it is that if you repeat the whole procedure, 95% of the intervals you generate will contain it. Below, the ones that miss are marked in red. One batch of a hundred will not land on exactly five — anywhere from one to ten is ordinary luck, which is itself the lesson. Press the button a few times and watch the running coverage underneath settle onto 95%.
Set Real difference to zero, so the two groups are genuinely identical, and then run twenty tests at once. Each test individually has a 5% chance of crying wolf — but run twenty and the chance that at least one does is not 5%, it is 64%. Report only that one and you have a publishable result about nothing at all. This is not a hypothetical: it is the single commonest way real findings turn out to be false.
Significance answers "could this be a fluke?" and nothing else. With enough data even a difference far too small to care about becomes statistically significant, because the question being answered was never "is this big?". The table runs the same trivial effect at growing sample sizes and shows the p-value collapsing while the effect stays exactly as unimportant as it was.
| sample size | difference | p-value | significant? | worth acting on? |
|---|
The population is generated, so its true mean is known exactly rather than estimated. Sampling is real random draw without replacement from it. The confidence intervals use the t distribution with the sample's own standard deviation, and when you take a hundred of them the coverage comes out at about 95% — not because it is set to, but because that is what the procedure does. The multiple-comparison figure lands on 64% against a predicted 1 − 0.95²⁰ = 64.2%.
Two things this page simplifies. The population is normal and the samples are genuinely random, which is the easy case — most real-world failures come from samples that were not random, and no amount of statistics fixes a biased sample. And the t-test here assumes equal variances and independent observations; real data routinely violates both, which is why applied statisticians spend more time arguing about whether a test applies than about the arithmetic once it does.