analysis.replication_precision
analysis.replication_precision(
values,
*,
ci_level=0.95,
deviation_threshold=0.05,
)Running confidence-interval precision as replications accumulate.
For k = 1..n, in the order values is given, computes the confidence interval using only the first k replications, plus its relative half-width (deviation) - the standard diagnostic (Hoad, Robinson & Davies, 2010) for deciding how many replications a study needs: run more until the interval is tight enough relative to the mean, not just “run some fixed number and hope.”
Parameters
| Name | Type | Description | Default |
|---|---|---|---|
| values | array - like | Per-replication values, already in run order - typically replication_means(...)["value"]. This function does not sort them; a cumulative diagnostic is only meaningful walked through in the order replications were actually generated. |
required |
| ci_level | float | Confidence level for each cumulative interval - see mean_confidence_interval. |
0.95 |
| deviation_threshold | float | Relative half-width threshold used for stays_below_threshold (see Returns). 0.05 means the CI half-width must be within 5% of the cumulative mean. |
0.05 |
Returns
| Name | Type | Description |
|---|---|---|
| pandas.DataFrame | One row per k = 1..n, columns: - n_replications : int - k. - cumulative_mean : float - mean of values[:k]. - half_width, lower, upper : float - the confidence interval from the first k values. NaN at k=1 - a spread needs at least 2 points, matching mean_confidence_interval. - deviation : float - half_width / abs(cumulative_mean), the relative precision. Always non-negative, so a metric with a negative mean (e.g. a before/after difference) is not read as trivially “precise” by a negative ratio. NaN wherever half_width is NaN, or if cumulative_mean is 0. - stays_below_threshold : bool - True at row k if deviation is defined and no greater than deviation_threshold at k and every later row, up to n. Deliberately “stays below”, not “first drops below”: a noisy early curve can dip under the threshold once by chance and rise again, which would be a spurious recommendation. The smallest n_replications with stays_below_threshold=True is the recommended minimum replication count; always False at k=1. This is a property of the batch of n replications actually supplied, not a guarantee that deviation stays low forever - a run flagged True from a 20-replication batch could fail to qualify once replications 21+ are added and re-checked. |
Raises
| Name | Type | Description |
|---|---|---|
| ValueError | If values is empty. |
|
| ImportError | If scipy is not installed and n >= 2 - see mean_confidence_interval. |
See Also
mean_confidence_interval : The single-k confidence interval this is built from. replication_means : Produces the per-replication values this function consumes. vidigi.plots.plot_replication_analysis : Plots this table.
Notes
Recomputing a confidence interval at every k and reading off the first crossing of deviation_threshold risks “early convergence” - a spurious dip from a run of similar-by-chance values, not a genuinely settled interval. Hoad, Robinson & Davies (2010) name this problem directly and address it with a “look ahead”: once precision first crosses the threshold, their algorithm runs a further, fixed number of replications (their kLimit, for which they recommend a default of 5) and checks precision stays crossed before accepting the result - shown empirically to remove the coverage failures a naive first-crossing rule produced in their own tests. stays_below_threshold here is a simpler stand-in for that same idea - checking all the way to the end of the supplied batch, not a fixed look-ahead window - rather than a literal implementation of kLimit, and, like the source paper’s own procedure, validated only empirically rather than derived as a formal statistical correction for the underlying repeated-test problem. Treat the reported stays_below_threshold/recommended count as a starting point for judgement, not a fully-automated stopping rule - matching this package’s deliberate choice not to fully automate welch_moving_average’s warm-up selection either.