Cleaning and Validation

6 algorithms in Market Data Engineering.

In this family#

1. OHLC Consistency Validator contract

   Checks each bar against the invariants an OHLC bar must satisfy — high is the maximum, low is the minimum, open and close lie between them — with a tick-size tolerance so representable rounding is not reported as corruption.

   validateBars(bars, config)
2. Hampel Bad-Tick Filter contract

   Flags points that sit too far from a rolling median, measured in robust deviations rather than standard deviations — so one fat-finger print cannot inflate the very statistic used to detect it.

   hampelFilter(values, options)
3. Median Absolute Deviation Outlier Filter contract

   Whole-sample outlier detection against the median absolute deviation. Where the Hampel filter is local and rolling, this judges every point against one global robust spread.

   madOutliers(values, threshold, scale, minimumSamples)
4. Stale-Quote Detector contract

   Separates the several distinct ways a quote can be stale: old at the source, delayed in transport, unchanged for too long, or arriving after a missed heartbeat. They have different causes and different remedies, so they are reported separately.

   detectStaleQuotes(events, config)
5. Duplicate-Trade Resolver contract

   Reconciles a stream of new, corrected and cancelled trade messages into one authoritative set. Replays of the same message must be idempotent, and a cancellation must survive a later redelivery of the trade it cancelled.

   resolveTrades(input)
6. Crossed/Locked Market Detector contract

   Classifies a quote as normal, locked (bid equals ask) or crossed (bid above ask). Crossed markets are usually a stale or misordered feed rather than a real arbitrage, which is exactly why they must be caught before anything downstream trusts the spread.

   classifyMarkets(quotes, options)

What they share#

Every topic here is a row-classify</code> or <code>record-transform, so once you have called one the rest follow the same shape. Import paths differ only in the final segment:

import { validateBars } from "fintech-algorithms/market-data-engineering/cleaning-and-validation/ohlc-consistency-validator";
import { hampelFilter } from "fintech-algorithms/market-data-engineering/cleaning-and-validation/hampel-bad-tick-filter";

Read them in the order above — the sequence is pedagogical, not alphabetical.

Where this sits#

Market Data Engineering collects 31 algorithms across 5 families. For the concept behind this family rather than the call signatures, see the concept guides.