Slice-Based Validation by Sector, Country, and Regime

Install and import#

bash

npm install fintech-algorithms

import { sliceBasedValidationBySectorCountryAndRegime } from "fintech-algorithms/model-validation-and-backtesting/classification-and-score-validation/slice-based-validation-by-sector-country-and-regime";

Signature#

sliceBasedValidationBySectorCountryAndRegime(inputs)

Worked example#

verified This is the worked example published in the article, replayed by the test suite on every run. The output cannot drift.

Input#

inputs

{
  "records": [
    {
      "id": "R01",
      "label": 1,
      "score": 0.95,
      "probability": 0.92,
      "weight": 1,
      "sector": "Banking",
      "country": "Egypt",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    },
    {
      "id": "R02",
      "label": 0,
      "score": 0.9,
      "probability": 0.88,
      "weight": 1,
      "sector": "Insurance",
      "country": "Egypt",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    },
    {
      "id": "R03",
      "label": 1,
      "score": 0.9,
      "probability": 0.84,
      "weight": 1,
      "sector": "Markets",
      "country": "Saudi Arabia",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    }
  ],
  "evaluation_cutoff": "2026-06-30T00:00:00Z",
  "slice_fields": ["sector", "country", "regime"],
  "minimum_support": 4,
  "epsilon": 1e-15
}

Call#

sliceBasedValidationBySectorCountryAndRegime(inputs)

Returns#

object with 8 fields: overall, slices, slice_fields, minimum_support, worst_slice_brier, eligible_slice_count, flagged_slice_count, state

{
  "overall": {
    "record_count": 24,
    "weight_sum": 24,
    "positive_weight": 10,
    "negative_weight": 14,
    "event_rate": 0.4166666666666667,
    "brier_score": 0.24828333333333338,
    "log_loss": 0.7086022750977335,
    "roc_auc": 0.6607142857142857
  },
  "slices": [
    {
      "field": "sector",
      "value": "Banking",
      "status": "ok",
      "record_count": 8,
      "weight_sum": 8,
      "positive_weight": 6,
      "negative_weight": 2,
      "event_rate": 0.75,
      "brier_score": 0.38560000000000005,
      "log_loss": 1.045315173439001,
      "roc_auc": 0.3333333333333333,
      "brier_gap_from_overall": 0.13731666666666667,
      "log_loss_gap_from_overall": 0.3367128983412674,
      "roc_auc_gap_from_overall": -0.3273809523809524
    },
    {
      "field": "sector",
      "value": "Insurance",
      "status": "ok",
      "record_count": 8,
      "weight_sum": 8,
      "positive_weight": 2,
      "negative_weight": 6,
      "event_rate": 0.25,
      "brier_score": 0.2472,
      "log_loss": 0.7127623167847974,
      "roc_auc": 0.6666666666666666,
      "brier_gap_from_overall": -0.0010833333333333806,
      "log_loss_gap_from_overall": 0.0041600416870638535,
      "roc_auc_gap_from_overall": 0.005952380952380931
    },
    {
      "field": "sector",
      "value": "Markets",
      "status": "ok",
      "record_count": 8,
      "weight_sum": 8,
      "positive_weight": 2,
      "negative_weight": 6,
      "event_rate": 0.25,
      "brier_score": 0.11205,
      "log_loss": 0.36772933506940214,
      "roc_auc": 1,
      "brier_gap_from_overall": -0.13623333333333337,
      "log_loss_gap_from_overall": -0.3408729400283314,
      "roc_auc_gap_from_overall": 0.3392857142857143
    }
  ],
  "slice_fields": ["sector", "country", "regime"],
  "minimum_support": 4,
  "worst_slice_brier": 0.38560000000000005,
  "eligible_slice_count": 8,
  "flagged_slice_count": 0,
  "state": "slices-evaluated"
}

Other exports#

This module also exports rocCurveAndRocAuc, precisionRecallCurveAndPrAuc, brierScore, logLoss, reliabilityDiagramAndExpectedCalibrationError, gainsLiftAndDecileCapture, costSensitiveThresholdOptimization, scoreStabilityAndMigrationMatrix, rareEventBacktestAndConfidenceBounds, calculate. Every module additionally exports run as an alias of its primary function, and a meta object carrying its catalog id, domain, family, shape and article URL.