Schema-Drift Detector

Catch Structural and Semantic Contract Changes

Install and import

bash

npm install fintech-algorithms

import { detectSchemaDrift } from "fintech-algorithms/market-data-engineering/data-quality/schema-drift-detector";

Signature

detectSchemaDrift(baseline, candidate, policy)

Compares two versions of a feed schema and classifies every change as compatible or breaking. Vendors rename and retype fields without announcement, and a silently renamed field is worse than an outage because nothing fails.

Parameters

Name	Type	Notes
`baseline`	`Schema`	The schema currently relied upon.
`candidate`	`Schema`	The schema just observed on the feed.
`policy`	`DriftPolicy`	Which categories of change are tolerated, and which must fail.

Returns

{ baseline_version, candidate_version, policy, status, coverage_complete, changes }

Every change with its classification, plus whether the comparison covered the whole schema — an incomplete comparison must not read as a clean bill of health.

Errors

When either schema is missing an identifier or version — throws

Complexity: time O(fields), space O(fields).

Worked example

executed Captured by running this function on the input its own test provides. Real output of real code — but not asserted against a published figure.

Input

baseline

{
  "schema_id": "synthetic-trades",
  "version": "v1.0",
  "parent_version": null,
  "completeness": "complete",
  "fields": {
    "symbol": {
      "type": "string",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": null,
      "meaning": "Instrument symbol under the declared venue symbology"
    },
    "price": {
      "type": "number",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": "USD per share",
      "meaning": "Executed price"
    },
    "size": {
      "type": "integer",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": "shares",
      "meaning": "Executed quantity"
    },
    "sale_condition": {
      "type": "string",
      "required": false,
      "nullable": true,
      "enum": [" ", "O"],
      "unit": null,
      "meaning": "Synthetic sale-condition code"
    }
  }
}

candidate

{
  "schema_id": "synthetic-trades",
  "version": "v1.0-clone",
  "parent_version": "v1.0",
  "completeness": "complete",
  "fields": {
    "symbol": {
      "type": "string",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": null,
      "meaning": "Instrument symbol under the declared venue symbology"
    },
    "price": {
      "type": "number",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": "USD per share",
      "meaning": "Executed price"
    },
    "size": {
      "type": "integer",
      "required": true,
      "nullable": false,
      "enum": null,
      "unit": "shares",
      "meaning": "Executed quantity"
    },
    "sale_condition": {
      "type": "string",
      "required": false,
      "nullable": true,
      "enum": [" ", "O"],
      "unit": null,
      "meaning": "Synthetic sale-condition code"
    }
  }
}

Call

detectSchemaDrift(baseline, candidate, policy)

Returns

object with 6 fields: baseline_version, candidate_version, policy, status, coverage_complete, changes

{
  "baseline_version": "v1.0",
  "candidate_version": "v1.0-clone",
  "policy": {
    "name": "strict_existing_consumer_v1",
    "direction": "candidate_producer_to_baseline_consumer",
    "added_field": "non_breaking",
    "removed_optional": "review",
    "removed_required": "breaking",
    "type_changed": "breaking",
    "optional_to_required": "non_breaking",
    "required_to_optional": "breaking",
    "nonnullable_to_nullable": "breaking",
    "nullable_to_nonnullable": "non_breaking",
    "enum_values_added": "breaking",
    "enum_values_removed": "review",
    "unit_changed": "breaking",
    "meaning_changed": "breaking"
  },
  "status": "unchanged",
  "coverage_complete": true,
  "changes": []
}

Diagrams

Schema-Drift Detector — schema compatibility matrix

How it works

This page states the contract — how to call it correctly. The article explains the concept: why it works, and where it breaks.

Read the article →

References

Daily TAQ Client Specification v3.3b
Daily TAQ Client Specification v3.3c
Market Data Documents
JSON Schema Core 2020-12
JSON Schema Validation 2020-12
Apache Avro Specification
Evidence decision