fintech-algorithms
/
Using a coding agent? Give it the skill: npx skills add IslamBaraka90/Fintech-Algorithms-Library What it does →

Schema-Drift Detector

Catch Structural and Semantic Contract Changes

Install and import#

bash
npm install fintech-algorithms
ts
import { detectSchemaDrift } from "fintech-algorithms/market-data-engineering/data-quality/schema-drift-detector";

Signature#

detectSchemaDrift(baseline, candidate, policy)

Compares two versions of a feed schema and classifies every change as compatible or breaking. Vendors rename and retype fields without announcement, and a silently renamed field is worse than an outage because nothing fails.

Parameters#

NameTypeNotes
baselineSchemaThe schema currently relied upon.
candidateSchemaThe schema just observed on the feed.
policyDriftPolicyWhich categories of change are tolerated, and which must fail.

Returns#

{ baseline_version, candidate_version, policy, status, coverage_complete, changes }

Every change with its classification, plus whether the comparison covered the whole schema — an incomplete comparison must not read as a clean bill of health.

Errors#

  • When either schema is missing an identifier or version — throws

Complexity: time O(fields), space O(fields).

Worked example#

executed Captured by running this function on the input its own test provides. Real output of real code — but not asserted against a published figure.

Input#

baseline
{
  "schema_id": "synthetic-trades",
  "version": "v1.0",
  "parent_version": null,
  "completeness": "complete",
  "fields": {
    "symbol": {
      "type": "string",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": null,
      "meaning": "Instrument symbol under the declared venue symbology"
    },
    "price": {
      "type": "number",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": "USD per share",
      "meaning": "Executed price"
    },
    "size": {
      "type": "integer",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": "shares",
      "meaning": "Executed quantity"
    },
    "sale_condition": {
      "type": "string",
      "required": false,
      "nullable": true,
      "enum": [" ", "O"],
      "unit": null,
      "meaning": "Synthetic sale-condition code"
    }
  }
}
candidate
{
  "schema_id": "synthetic-trades",
  "version": "v1.0-clone",
  "parent_version": "v1.0",
  "completeness": "complete",
  "fields": {
    "symbol": {
      "type": "string",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": null,
      "meaning": "Instrument symbol under the declared venue symbology"
    },
    "price": {
      "type": "number",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": "USD per share",
      "meaning": "Executed price"
    },
    "size": {
      "type": "integer",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": "shares",
      "meaning": "Executed quantity"
    },
    "sale_condition": {
      "type": "string",
      "required": false,
      "nullable": true,
      "enum": [" ", "O"],
      "unit": null,
      "meaning": "Synthetic sale-condition code"
    }
  }
}

Call#

detectSchemaDrift(baseline, candidate, policy)

Returns#

object with 6 fields: baseline_version, candidate_version, policy, status, coverage_complete, changes

{
  "baseline_version": "v1.0",
  "candidate_version": "v1.0-clone",
  "policy": {
    "name": "strict_existing_consumer_v1",
    "direction": "candidate_producer_to_baseline_consumer",
    "added_field": "non_breaking",
    "removed_optional": "review",
    "removed_required": "breaking",
    "type_changed": "breaking",
    "optional_to_required": "non_breaking",
    "required_to_optional": "breaking",
    "nonnullable_to_nullable": "breaking",
    "nullable_to_nonnullable": "non_breaking",
    "enum_values_added": "breaking",
    "enum_values_removed": "review",
    "unit_changed": "breaking",
    "meaning_changed": "breaking"
  },
  "status": "unchanged",
  "coverage_complete": true,
  "changes": []
}

Diagrams#

Schema-Drift Detector — article hero
Schema-Drift Detector — schema compatibility matrix

How it works#

This page states the contract — how to call it correctly. The article explains the concept: why it works, and where it breaks.

Read the article →

References#

  • Daily TAQ Client Specification v3.3b
  • Daily TAQ Client Specification v3.3c
  • Market Data Documents
  • JSON Schema Core 2020-12
  • JSON Schema Validation 2020-12
  • Apache Avro Specification
  • Evidence decision

The rest of the Data Quality family#