Reliability Diagram and Expected Calibration Error

Install and import#

bash

npm install fintech-algorithms

import { reliabilityDiagramAndExpectedCalibrationError } from "fintech-algorithms/model-validation-and-backtesting/classification-and-score-validation/reliability-diagram-and-expected-calibration-error";

Signature#

reliabilityDiagramAndExpectedCalibrationError(inputs)

Worked example#

verified This is the worked example published in the article, replayed by the test suite on every run. The output cannot drift.

Input#

inputs

{
  "records": [
    {
      "id": "R01",
      "label": 1,
      "score": 0.95,
      "probability": 0.92,
      "weight": 1,
      "sector": "Banking",
      "country": "Egypt",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    },
    {
      "id": "R02",
      "label": 0,
      "score": 0.9,
      "probability": 0.88,
      "weight": 1,
      "sector": "Insurance",
      "country": "Egypt",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    },
    {
      "id": "R03",
      "label": 1,
      "score": 0.9,
      "probability": 0.84,
      "weight": 1,
      "sector": "Markets",
      "country": "Saudi Arabia",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    }
  ],
  "evaluation_cutoff": "2026-06-30T00:00:00Z",
  "bins": 5,
  "bin_strategy": "uniform"
}

Call#

reliabilityDiagramAndExpectedCalibrationError(inputs)

Returns#

object with 7 fields: bins, expected_calibration_error, maximum_calibration_error, signed_calibration_error, bin_count, weight_sum, state

{
  "bins": [
    {
      "index": 1,
      "lower": 0,
      "upper": 0.2,
      "right_inclusive": false,
      "record_count": 5,
      "weight_sum": 5,
      "mean_probability": 0.084,
      "event_rate": 0.2,
      "signed_gap": 0.116,
      "absolute_gap": 0.116
    },
    {
      "index": 2,
      "lower": 0.2,
      "upper": 0.4,
      "right_inclusive": false,
      "record_count": 5,
      "weight_sum": 5,
      "mean_probability": 0.27999999999999997,
      "event_rate": 0.4,
      "signed_gap": 0.12000000000000005,
      "absolute_gap": 0.12000000000000005
    },
    {
      "index": 3,
      "lower": 0.4,
      "upper": 0.6,
      "right_inclusive": false,
      "record_count": 5,
      "weight_sum": 5,
      "mean_probability": 0.48,
      "event_rate": 0.4,
      "signed_gap": -0.07999999999999996,
      "absolute_gap": 0.07999999999999996
    }
  ],
  "expected_calibration_error": 0.14250000000000002,
  "maximum_calibration_error": 0.28,
  "signed_calibration_error": -0.044166666666666674,
  "bin_count": 5,
  "weight_sum": 24,
  "state": "calibration-evaluated"
}

Other exports#

This module also exports rocCurveAndRocAuc, precisionRecallCurveAndPrAuc, brierScore, logLoss, gainsLiftAndDecileCapture, costSensitiveThresholdOptimization, scoreStabilityAndMigrationMatrix, sliceBasedValidationBySectorCountryAndRegime, rareEventBacktestAndConfidenceBounds, calculate. Every module additionally exports run as an alias of its primary function, and a meta object carrying its catalog id, domain, family, shape and article URL.