Stable Online Variance and Welford's Algorithm

Install and import#

npm install fintech-algorithms

import { stableOnlineVarianceAndWelfordsAlgorithm } from "fintech-algorithms/foundations/statistical-computing-and-reproducibility/stable-online-variance-and-welford-s-algorithm";

Signature#

stableOnlineVarianceAndWelfordsAlgorithm(input)

Runs Welford's single-pass update to produce the mean and sample variance without ever accumulating a sum of raw squares.

Parameters#

Returns#

D00Output

count is the number of observations consumed, mean the running mean after the last of them, m2 the accumulated sum of squared deviations, and sampleVariance that sum divided by count minus one.

Errors#

• When values is absent, empty, or holds a non-finite number — throws RangeError
• When values holds fewer than two observations — throws RangeError

Complexity: time O(n), space O(n).

Worked example#

verified This is the worked example published in the article, replayed by the test suite on every run. The output cannot drift.

Input#

{
  "values": [1, 2, 3, 4, 5],
  "floatingValue": 0.1,
  "maxSafeMagnitude": 1.7976931348623157e+308,
  "window": 3,
  "rawValues": [1, null, 2, "bad", 3],
  "invalidPolicy": "drop-and-report",
  "seed": 42,
  "sampleCount": 5,
  "leftVector": [
    {
      "index": "A",
      "value": 1
    },
    {
      "index": "B",
      "value": 2
    }
  ],
  "rightVector": [
    {
      "index": "B",
      "value": 20
    },
    {
      "index": "C",
      "value": 30
    }
  ],
  "train": [10, 12, 14, 16],
  "test": [18, 20],
  "actual": [1, 2.0000001, 3],
  "expected": [1, 2, 3]
}

Showing 14 of 18 fields.

Call#

stableOnlineVarianceAndWelfordsAlgorithm(input)

Returns#

object with 2 fields: count, mean

{
  "count": 5,
  "mean": 3
}

Diagrams#

How it works#

This page states the contract — how to call it correctly. The article explains the concept: why it works, and where it breaks.

Read the article →

References#

• Floating-Point Arithmetic — IEEE 754-2019
• math.fsum — Python Documentation
• Random Sampling — NumPy Documentation
• Common Pitfalls and Recommended Practices — scikit-learn
• Historical-example decision

The rest of the Statistical Computing and Reproducibility family#

• Floating-Point Representation, Overflow, and UnderflowInspects one double-precision number: whether it is finite, how large one representable step at its magnitude is, and whether s…
• Stable Summation and Mean CalculationAdds a list twice, once straight left to right and once carrying a running compensation term, so the two totals can be put side…
• Batch, Rolling, and Streaming Statistic EquivalenceComputes the same fixed-window mean two ways, by re-averaging each window slice and by a running add-and-drop sum, then checks …
• Missing, Infinite, Invalid, and Unsupported-State PoliciesScreens a mixed raw list down to the entries that are genuinely finite numbers and reports how many were turned away.
• Pseudorandom Numbers, Seeds, and ReproducibilityDraws a run of unit-interval numbers from a seeded linear congruential generator, so the same seed always replays the same stream.
• Vectorization, Index Alignment, and Shape SafetyInner-joins two labelled vectors on their index field and reports what each side lost in the process.
• Leakage-Free Fitting, Scaling, and PreprocessingFits a standardizing centre and scale on the training split alone, then applies that one transform to both splits.
• Fixtures, Numerical Tolerances, and Property TestsCompares an actual vector against an expected one under an absolute tolerance, the way a fixture check does.
• Reproducible Analysis, Metadata, and Audit TrailsSerializes a run's inputs, method, version and seed into a deterministic JSON string and fingerprints it with a 32-bit FNV-1a h…