A resampling experiment produces a picture of how a summary statistic varies when you draw samples. It does not prove anything about a process, a cause, or a future. This guide shows how to read the 13 Labs experiment and how to write down findings that stay inside the evidence.
Start with what the tool actually does
The experiment takes your typed numeric dataset (3 to 200 values), reports summary statistics such as mean, median and range, and lets you remove one flagged value to compare. It then draws repeated samples of a chosen size, with replacement, from your data, computes the mean of each sample, and plots those sample means as a dotplot with an accessible results table. The seed and the settings are displayed so a run can be described and repeated.
Every number on that page is illustrative. It is a demonstration of a computation on numbers you typed, not a measurement of any real process, instrument, or population. NIST's handbook treats the bootstrap as a way to display the sampling distribution of a statistic and to reason about uncertainty, and it warns that the bootstrap is not appropriate for all statistics and distributions — for example, statistics heavily dependent on the tails, such as the range, are not well served by it (NIST EDA bootstrap plot). Keep that caveat in view: the tool resamples the mean, not the range.
Describe before you interpret
Write a description first. For example: hypothetical dataset of seven values, one of which is far larger than the rest; the sample mean is pulled toward that value while the median sits in the middle of the cluster. That sentence is a description of arithmetic on your list. It is true whenever you type those numbers.
Interpretation is a separate, weaker sentence. With a single outlier, you can say the mean and the median tell different stories about the same list. NIST's case study on beam deflections makes exactly this kind of observation: the data contained points identified as outliers from the lag plot, and refitting after removing three outliers cut the residual standard deviation by only about 5 percent — and the handbook calls the choice between the two fits a judgment call (NIST beam deflection case study). Outlier removal is a modeling decision with modest effects, not a cleanup step that reveals the truth.
Report the settings, not just the picture
A dotplot without its settings is unreadable as evidence. Always state: the dataset (or a link to the downloaded JSON), sample size, number of repeats, and seed. The download button gives you the dataset, results and settings together; keep that file with any note you write. NIST's own bootstrap description uses 500 subsamples as a practical default while noting the number is somewhat arbitrary (NIST bootstrap plot) — useful context for why the repeat count matters and why it is a setting, not a fact.
If someone reruns your seed with your sample size and repeats, they should see your dotplot. That reproducibility is what makes the page shareable; it is not a guarantee that the result generalizes.
Keep conclusions inside the method
Four sentences you can write:
- The displayed spread of sample means is the spread this resampling procedure produced on this list.
- Increasing sample size generally reduces the variation of sample means for this procedure. Compare the actual spreads in your runs; a finite simulation does not guarantee a narrower observed range every time.
- An extreme value moves the mean more than the median in this dataset.
- The seed and settings shown are sufficient to repeat this run.
Four sentences to avoid:
- This proves the process is (or is not) stable.
- These numbers show a significant difference between groups.
- This predicts the next measurement.
- Removing the outlier gives the true value.
The reason to avoid them is structural, not stylistic. The tool computes on a typed list with no sampling plan, no measurement model, and no inference procedure. NIST is explicit that resolving unexpected results "cannot be made on the basis of statistics alone" and requires knowledge of the process (NIST filter transmittance case study). A dotplot of resampled means does not supply that knowledge.
When sample size looks convincing
The sample-size control (bounded by your dataset length) can make the dotplot look tight or wide. It is tempting to narrate that as "more data equals more certainty." Read it more narrowly: on this list, samples of size k have less spread than samples of size j. The list is still the list. NIST's uncertainty formulas show the same structural point — the standard deviation of an average divides by the square root of the number of measurements and depends on the standard deviation of the underlying data (NIST propagation of error). Both factors matter; the tool holds one of them fixed by construction.
A short checklist before you publish a note
- State that the dataset is illustrative or yours, and that the tool does not measure a physical process.
- Quote the settings: sample size, repeats, seed.
- Separate the description sentence from the interpretation sentence.
- Name what the method does not do: no significance claims, no p-values, no predictions.
- Attach the downloaded JSON so readers can rerun.
Done this way, the experiment is a teaching instrument: it shows how a summary statistic moves under resampling, and it leaves the science to the people who collected the data. Continue with averages and outliers and sample size for the parts of this reasoning the tool makes visible.
