Sequence-Gap Detection and Recovery

Install and import

bash

npm install fintech-algorithms

import { recoverSequenceStream } from "fintech-algorithms/market-data-engineering/order-book-feed-engineering/sequence-gap-detection-and-recovery";

Signature

recoverSequenceStream(arrivals)

Detects missing sequence numbers and decides whether the stream can continue or must resynchronise from a snapshot. A gap is not a nuisance — every message after it is applied to a book that is already wrong.

Parameters

Name	Type	Notes
`arrivals`	`Arrival[]`	Messages with their sequence numbers, in arrival order — which is not necessarily sequence order.

Returns

{ status, gaps, recovered, resync_required, … }

Each gap with whether it was recoverable from buffered messages or requires a snapshot resync.

Errors

When sequence numbers are absent — reported as a status rather than thrown

Complexity: time O(n log n), space O(n).

Worked example

executed Captured by running this function on the input its own test provides. Real output of real code — but not asserted against a published figure.

Input

arrivals

[
  {
    "sequence": 100,
    "source": "live"
  },
  {
    "sequence": 102,
    "source": "live"
  },
  {
    "sequence": 103,
    "source": "live"
  }
]

Showing 3 of 18 elements.

argument 2

{
  "start_sequence": 100
}

Call

recoverSequenceStream(arrivals)

Returns

object with 11 fields: start_sequence, applied_sequences, next_expected, missing_sequences, buffered_sequences, duplicate_sequences, recovery_requests, recovery_request_count, …

{
  "start_sequence": 100,
  "applied_sequences": [100, 101, 102, 103, 104, 105],
  "next_expected": 117,
  "missing_sequences": [],
  "buffered_sequences": [],
  "duplicate_sequences": [107],
  "recovery_requests": [
    {
      "from_sequence": 101,
      "to_sequence": 101
    },
    {
      "from_sequence": 104,
      "to_sequence": 104
    },
    {
      "from_sequence": 106,
      "to_sequence": 106
    }
  ],
  "recovery_request_count": 5,
  "replay_arrival_count": 6,
  "trace": [
    {
      "arrival_sequence": 100,
      "source": "live",
      "next_expected": 101,
      "missing": [],
      "buffered": []
    },
    {
      "arrival_sequence": 102,
      "source": "live",
      "next_expected": 101,
      "missing": [101],
      "buffered": [102]
    },
    {
      "arrival_sequence": 103,
      "source": "live",
      "next_expected": 101,
      "missing": [101],
      "buffered": [102, 103]
    }
  ],
  "state": "current"
}

Other exports

This module also exports normalizeEvents, reconstructL2, aggregatePriceLevels, reconstructL3, reconcileSnapshotIncrementals, consolidateVenues, calculate. Every module additionally exports run as an alias of its primary function, and a meta object carrying its catalog id, domain, family, shape and article URL.