Refresh-Time Sampling
Install and import#
npm install fintech-algorithmsimport { refreshTimeSample } from "fintech-algorithms/market-data-engineering/time-synchronization/refresh-time-sampling";Signature#
refreshTimeSample(observations, requiredInstruments, maxStalenessMs)Builds a common clock for several instruments by advancing only when every one of them has refreshed. All-refresh sampling is one established construction for non-synchronous observations, not a neutral or sufficient remedy for correlation bias.
Parameters#
| Name | Type | Notes |
|---|---|---|
observations | Observation[] | Observations across all instruments, each with partition, instrument, event_time and available_at. |
requiredInstruments | string[] | The instruments that must all have refreshed before a refresh time is emitted. At least two distinct non-empty names are required. min_distinct_names: 2 |
maxStalenessMs | number | How old any instrument's selected event may be at a refresh time, as a non-negative whole number of milliseconds. Equality is accepted; a row above the limit is flagged stale rather than repaired or dropped.min: 0 · integer: true |
Returns#
{ rows, partitions }
The synchronised rows, plus per-partition diagnostics showing which instrument was the binding constraint at each step.
Errors#
- When requiredInstruments holds fewer than two distinct names, or any name is empty — throws
- When maxStalenessMs is not a non-negative integer — throws
- When an observation fails validation: missing field, unknown instrument, empty partition or record_id, negative or non-integer revision, non-finite value, event_time after available_at, non-contiguous or non-causal revision, or two record_ids sharing an instrument and event_time — throws
Complexity: time O(n log n) to validate and order n arrivals, plus O(r * n) for the barrier loop in these reference implementations, where r is the number of emitted rows; the loop re-filters the partition at each barrier for clarity. A production implementation using monotone cursors and indexed revision state can do the same work in linear passes.,
space O(n).
Worked example#
executed Captured by running this function on the input its own test provides. Real output of real code — but not asserted against a published figure.
Input#
[
{
"partition": "P1",
"instrument": "A",
"record_id": "a1",
"revision": 0,
"event_time": "2026-01-01T00:00:01.000Z",
"available_at": "2026-01-01T00:00:01.000Z",
"value": 1
},
{
"partition": "P1",
"instrument": "A",
"record_id": "a3",
"revision": 0,
"event_time": "2026-01-01T00:00:03.000Z",
"available_at": "2026-01-01T00:00:03.000Z",
"value": 3
},
{
"partition": "P1",
"instrument": "A",
"record_id": "a4",
"revision": 0,
"event_time": "2026-01-01T00:00:04.000Z",
"available_at": "2026-01-01T00:00:04.000Z",
"value": 4
}
]Showing 3 of 5 elements.
["A", "B"]5000Call#
refreshTimeSample(observations, requiredInstruments, maxStalenessMs)Returns#
object with 2 fields: rows, partitions
{
"rows": [
{
"partition": "P1",
"sequence": 1,
"previous_refresh_available_at": null,
"refresh_available_at": "2026-01-01T00:00:02.000Z",
"controller_instruments": ["B"],
"status": "accepted",
"values": {
"A": 1,
"B": 20
},
"sources": {
"A": {
"record_id": "a1",
"revision": 0,
"event_time": "2026-01-01T00:00:01.000Z",
"available_at": "2026-01-01T00:00:01.000Z"
},
"B": {
"record_id": "b2",
"revision": 0,
"event_time": "2026-01-01T00:00:02.000Z",
"available_at": "2026-01-01T00:00:02.000Z"
}
},
"event_age_ms": {
"A": 1000,
"B": 0
},
"arrivals_by_instrument": {
"A": 1,
"B": 1
},
"discarded_updates": 0
},
{
"partition": "P1",
"sequence": 2,
"previous_refresh_available_at": "2026-01-01T00:00:02.000Z",
"refresh_available_at": "2026-01-01T00:00:04.000Z",
"controller_instruments": ["B"],
"status": "accepted",
"values": {
"A": 4,
"B": 40
},
"sources": {
"A": {
"record_id": "a4",
"revision": 0,
"event_time": "2026-01-01T00:00:04.000Z",
"available_at": "2026-01-01T00:00:04.000Z"
},
"B": {
"record_id": "b4",
"revision": 0,
"event_time": "2026-01-01T00:00:04.000Z",
"available_at": "2026-01-01T00:00:04.000Z"
}
},
"event_age_ms": {
"A": 0,
"B": 0
},
"arrivals_by_instrument": {
"A": 2,
"B": 1
},
"discarded_updates": 1
}
],
"partitions": [
{
"partition": "P1",
"input_updates": 5,
"refresh_candidates": 2,
"accepted_rows": 2,
"stale_rows": 0,
"discarded_updates": 1,
"unmatched_tail_updates": 0,
"loss_fraction": {
"numerator": 1,
"denominator": 5
}
}
]
}Diagrams#
Calculation flow#
Availability-time refresh sequence
sequenceDiagram
participant A as Fast stream A
participant B as Medium stream B
participant C as Sparse stream C
participant S as Refresh sampler
A->>S: first new availability A1
A->>S: later availability A2
B->>S: first new availability B1
Note over S: wait, C is missing
C->>S: first new availability C1
S->>S: barrier = max(A1, B1, C1)
S-->>S: select latest known effective event per stream
How it works#
This page states the contract — how to call it correctly. The article explains the concept: why it works, and where it breaks.
References#
- Multivariate realised kernels — Ole E. Barndorff-Nielsen, Peter R. Hansen, Asger Lunde, and Neil Shephard
- On covariance estimation of non-synchronously observed diffusion processes — Takaki Hayashi and Nakahiro Yoshida
- RFC 3339 — Internet Engineering Task Force; Graham Klyne and Chris Newman
- Evidence classification