Working AtlasMeasurement & uncertainty

One Answer Isn’t a Trend

Repeat the same observation and see how a small sample can look like a change.

TryKeep both rates at 40%, then run several paired batches.

Watch forA wider band means this small batch leaves more uncertainty about the underlying rate.

Project capability model

Synthetic model of this capability; it does not recreate production.

Interactive model

A small, synthetic model of repeated mention observations. It does not query live AI assistants or represent CiteTrail production data.

Set up the synthetic experiment

First synthetic mention rate40%

Second synthetic mention rate40%

Observations per batch25

SYNTHETIC OBSERVATIONS

Two periods. Repeatable samples.

Seed240617

Each row is a fresh synthetic batch. The dot is the observed rate; the band is a 95% Wilson uncertainty range that accounts for sample size; the diamond is the setting used to generate that batch.

Batch

0%25%50%75%100%

Rate

A · batch 140%

B · batch 160%

Observed rate 95% sample-uncertainty range (Wilson interval) Simulation setting

The range shows which underlying mention rates are still plausible given this sample; small samples leave a wider range.

Period A mentions

10 / 25

Observed 40.0%; batches used 40%.

Period B mentions

15 / 25

Observed 60.0%; batches used 40%.

Observed difference

+20.0points

A sample comparison, not a significance test.

The underlying settings match; the samples may not.

Run more batches and watch the observed rates move around the shared setting. More independent observations tend to narrow the interval; they do not make future answers certain.

Preparing the interactive experiment…

Skill practicedInterpret small samples without overstating change

Connection to my workCiteTrail runs repeated prompts through AI provider APIs. This synthetic sample shows why small results need careful interpretation.

Related project:CiteTrail →

Understand the result

This synthetic experiment draws independent yes/no mention observations from two selected rates. Repeat batches with the same setting to see how finite samples vary, or change the second setting to explore how an underlying difference can be obscured by noise. The model illustrates sampling uncertainty; it does not predict real assistant answers.

Controls

First synthetic mention rate: The configured probability that one synthetic observation in period A records a mention. It is a model input, not a measured product result.
Second synthetic mention rate: The configured probability for period B. Keep it equal to period A to explore sample variation, or change it to model a hypothetical difference.
Observations per batch: How many independent synthetic yes/no observations are drawn for each period in a batch.
Run another batch: Draws another reproducible pair using the current settings and a different derived seed. Up to five batches are shown at once.
Reset experiment: Returns to one batch while keeping the current rates, sample size, and seed.
Reseed: Chooses another deterministic seed and redraws the initial paired batch.

What to watch for

The observed rates can differ even when both settings match. The 95% Wilson interval is an uncertainty range around each sample rate; it is not a significance test, so a visible gap alone does not prove a real change.
Increase observations per batch. The Wilson uncertainty ranges generally become narrower, while individual outcomes remain variable.
Change the second setting and compare its observed rate and interval with the first. The display reports a sample difference; it does not perform a hypothesis test.

Why it happens

A finite sample is only one set of outcomes drawn from the configured probability. The Wilson interval summarizes uncertainty for an independent binomial sample and behaves more reliably near zero or one than a simple normal interval. Real assistant responses may be correlated, and prompt selection, model versions, and orchestration choices also affect results; this intentionally small model does not account for those factors.

How this project capability informs the model

This exercise practices interpreting uncertain evidence and explaining what a small sample cannot establish. It does not predict provider answers or reproduce CiteTrail analytics.

See the broader engineering brief