Slice-Based Validation by Sector, Country, and Regime

Install and import#

bash

npm install fintech-algorithms

import { sliceBasedValidationBySectorCountryAndRegime } from "fintech-algorithms/model-validation-and-backtesting/classification-and-score-validation/slice-based-validation-by-sector-country-and-regime";

Signature#

sliceBasedValidationBySectorCountryAndRegime(inputs)

Recomputes Brier score, log loss, and ROC-AUC on the whole population and again on every value of every requested slice field, flagging slices that are too thin or single-class and reporting each slice's gap from overall.

Parameters#

Name	Type	Notes
`inputs`	`{ records: Array<{ id: string; label: 0 \| 1; score: number; probability: number; weight?: number; score_available_at?: string; label_available_at?: string; [sliceField: string]: unknown }>; evaluation_cutoff?: string; slice_fields: string[]; minimum_support?: number; epsilon?: number }`	The scored population plus the slicing plan. Every record needs a unique nonempty `id`, a `label` that is exactly the number 0 or 1, a finite `score`, and a finite `probability` in [0,1]; `weight` defaults to 1 and must be positive. `slice_fields` lists unique nonempty field names such as `sector`, `country`, or `regime`, and every record must carry a nonempty string under each of them. `minimum_support` defaults to 4 and must be an integer of at least 2; `epsilon` defaults to 1e-15, must sit strictly inside (0,0.5), and clips probabilities for the log loss. If `evaluation_cutoff` is supplied, any `score_available_at` or `label_available_at` on a record is compared against it as a string and must not sort after it. slice_fields: unique nonempty strings, at least one · minimum_support: integer of at least 2, default 4 · epsilon: strictly between 0 and 0.5, default 1e-15

Returns#

{ overall: { record_count: number; weight_sum: number; positive_weight: number; negative_weight: number; event_rate: number; brier_score: number; log_loss: number; roc_auc: number | null }; slices: Array<{ field: string; value: string; status: "ok" | "insufficient-support" | "single-class"; record_count: number; weight_sum: number; positive_weight: number; negative_weight: number; event_rate: number; brier_score: number; log_loss: number; roc_auc: number | null; brier_gap_from_overall: number; log_loss_gap_from_overall: number; roc_auc_gap_from_overall: number | null }>; slice_fields: string[]; minimum_support: number; worst_slice_brier: number; eligible_slice_count: number; flagged_slice_count: number; state: string }

overall holds the population-level metrics; slices repeats them for every field value, sorted by field then value, with roc_auc and its gap null when that slice has only one class. status is ok, insufficient-support when the slice holds fewer records than minimum_support, or single-class. worst_slice_brier is the highest Brier score among the ok slices, and state is slices-evaluated.

Errors#

When records is absent, not an array, or empty — throws RangeError
When a record id is missing, empty, or repeats an earlier one — throws RangeError
When a label is anything other than the number 0 or 1 — throws RangeError
When a probability falls outside [0,1] — throws RangeError
When a weight is zero or negative — throws RangeError
When slice_fields is empty, holds duplicates, or holds anything but nonempty strings — throws RangeError
When minimum_support is not an integer, or is below 2 — throws RangeError
When epsilon is not strictly inside (0,0.5) — throws RangeError
When any record is missing a nonempty string value for a requested slice field — throws RangeError
When no slice clears both the support and the two-class requirement — throws RangeError
When score_available_at or label_available_at sorts after evaluation_cutoff — throws RangeError

Complexity: time O(s * n log n), space O(n).

Worked example#

verified This is the worked example published in the article, replayed by the test suite on every run. The output cannot drift.

Input#

inputs

{
  "records": [
    {
      "id": "R01",
      "label": 1,
      "score": 0.95,
      "probability": 0.92,
      "weight": 1,
      "sector": "Banking",
      "country": "Egypt",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    },
    {
      "id": "R02",
      "label": 0,
      "score": 0.9,
      "probability": 0.88,
      "weight": 1,
      "sector": "Insurance",
      "country": "Egypt",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    },
    {
      "id": "R03",
      "label": 1,
      "score": 0.9,
      "probability": 0.84,
      "weight": 1,
      "sector": "Markets",
      "country": "Saudi Arabia",
      "regime": "Expansion",
      "score_available_at": "2025-01-01T00:00:00Z",
      "label_available_at": "2026-01-01T00:00:00Z"
    }
  ],
  "evaluation_cutoff": "2026-06-30T00:00:00Z",
  "slice_fields": ["sector", "country", "regime"],
  "minimum_support": 4,
  "epsilon": 1e-15
}

Call#

sliceBasedValidationBySectorCountryAndRegime(inputs)

Returns#

object with 8 fields: overall, slices, slice_fields, minimum_support, worst_slice_brier, eligible_slice_count, flagged_slice_count, state

{
  "overall": {
    "record_count": 24,
    "weight_sum": 24,
    "positive_weight": 10,
    "negative_weight": 14,
    "event_rate": 0.4166666666666667,
    "brier_score": 0.24828333333333338,
    "log_loss": 0.7086022750977335,
    "roc_auc": 0.6607142857142857
  },
  "slices": [
    {
      "field": "sector",
      "value": "Banking",
      "status": "ok",
      "record_count": 8,
      "weight_sum": 8,
      "positive_weight": 6,
      "negative_weight": 2,
      "event_rate": 0.75,
      "brier_score": 0.38560000000000005,
      "log_loss": 1.045315173439001,
      "roc_auc": 0.3333333333333333,
      "brier_gap_from_overall": 0.13731666666666667,
      "log_loss_gap_from_overall": 0.3367128983412674,
      "roc_auc_gap_from_overall": -0.3273809523809524
    },
    {
      "field": "sector",
      "value": "Insurance",
      "status": "ok",
      "record_count": 8,
      "weight_sum": 8,
      "positive_weight": 2,
      "negative_weight": 6,
      "event_rate": 0.25,
      "brier_score": 0.2472,
      "log_loss": 0.7127623167847974,
      "roc_auc": 0.6666666666666666,
      "brier_gap_from_overall": -0.0010833333333333806,
      "log_loss_gap_from_overall": 0.0041600416870638535,
      "roc_auc_gap_from_overall": 0.005952380952380931
    },
    {
      "field": "sector",
      "value": "Markets",
      "status": "ok",
      "record_count": 8,
      "weight_sum": 8,
      "positive_weight": 2,
      "negative_weight": 6,
      "event_rate": 0.25,
      "brier_score": 0.11205,
      "log_loss": 0.36772933506940214,
      "roc_auc": 1,
      "brier_gap_from_overall": -0.13623333333333337,
      "log_loss_gap_from_overall": -0.3408729400283314,
      "roc_auc_gap_from_overall": 0.3392857142857143
    }
  ],
  "slice_fields": ["sector", "country", "regime"],
  "minimum_support": 4,
  "worst_slice_brier": 0.38560000000000005,
  "eligible_slice_count": 8,
  "flagged_slice_count": 0,
  "state": "slices-evaluated"
}

Other exports#

This module also exports rocCurveAndRocAuc, precisionRecallCurveAndPrAuc, brierScore, logLoss, reliabilityDiagramAndExpectedCalibrationError, gainsLiftAndDecileCapture, costSensitiveThresholdOptimization, scoreStabilityAndMigrationMatrix, rareEventBacktestAndConfidenceBounds, calculate. Every module additionally exports run as an alias of its primary function, and a meta object carrying its catalog id, domain, family, shape and article URL.

Diagrams#

Calculation flow#

Slice-Based Validation by Sector, Country, and Regime calculation flow

flowchart LR
    S1["Validate records clocks probabilities scores labels an"]
    S2["Calculate overall metrics"]
    S3["Enumerate observed values for each approved field"]
    S4["Filter records and calculate support"]
    S5["Assign lowsupport or singleclass states"]
    S1 --> S2
    S2 --> S3
    S3 --> S4
    S4 --> S5
    S5 --> D{"slice taxonomy availability minimum support and metric con"}
    D --> O["worst_slice_brier + diagnostics"]
    O --> A["Audit: For each slice field supports across its observed values s"]

How it works#

This page states the contract — how to call it correctly. The article explains the concept: why it works, and where it breaks.

Read the article →

References#

Revised Guidance on Model Risk Management — Board of Governors of the Federal Reserve System, OCC, and FDIC
Metrics and scoring: quantifying the quality of predictions — scikit-learn maintainers
Probability calibration — scikit-learn maintainers
Evidence boundary

Slice-Based Validation by Sector, Country, and Regime

Install and import#

Signature#

Parameters#

Returns#

Errors#

Worked example#

Input#

Call#

Returns#

Other exports#

Diagrams#

Calculation flow#

How it works#

References#

The rest of the Classification and Score Validation family#