Classification and Score Validation

10 algorithms in Model Validation and Backtesting · 10 with asserted arithmetic.

In this family#

1. ROC Curve and ROC-AUC verified

   Walks a weighted record ledger from the highest score down, grouping records that share a score into one threshold step, and emits the ROC point at every step plus the trapezoidal area under them.

   rocCurveAndRocAuc(inputs)
2. Precision-Recall Curve and PR-AUC verified

   Sweeps a weighted record ledger from the highest score down and reports precision and recall at every distinct score, together with the average precision obtained by summing precision over each recall increment.

   precisionRecallCurveAndPrAuc(inputs)
3. Brier Score verified

   Averages the weighted squared gap between each predicted probability and its realised label, and compares that average against the prevalence-only baseline to give a skill score.

   brierScore(inputs)
4. Log Loss verified

   Computes the weighted mean negative log likelihood of a set of probability forecasts, clipping each probability into [epsilon, 1 - epsilon] first and reporting how many forecasts the clip actually moved.

   logLoss(inputs)
5. Reliability Diagram and Expected Calibration Error verified

   Drops probability forecasts into equal-width bins, contrasts each bin's mean forecast with its realised event rate, and aggregates those gaps into the expected, maximum, and signed calibration errors.

   reliabilityDiagramAndExpectedCalibrationError(inputs)
6. Gains, Lift, and Decile Capture verified

   Ranks records by score, cuts the ranking into equal-count buckets, and reports how much of the total event weight each bucket and each cumulative prefix captures, as a share and as a lift over prevalence.

   gainsLiftAndDecileCapture(inputs)
7. Cost-Sensitive Threshold Optimization verified

   Evaluates the classify-nothing option and every distinct score as a cutoff, prices each one with the caller's confusion-cell costs, and returns the full candidate ledger alongside the cheapest cutoff.

   costSensitiveThresholdOptimization(inputs)
8. Score Stability and Migration Matrix verified

   Matches two score snapshots of the same population by id, bands both scores on shared edges, and reports the band-to-band migration counts, per-record moves, and the drift between the two band distributions.

   scoreStabilityAndMigrationMatrix(inputs)
9. Slice-Based Validation by Sector, Country, and Regime verified

   Recomputes Brier score, log loss, and ROC-AUC on the whole population and again on every value of every requested slice field, flagging slices that are too thin or single-class and reporting each slice's gap from overall.

   sliceBasedValidationBySectorCountryAndRegime(inputs)
10. Rare-Event Backtest and Confidence Bounds verified

    Builds a Wilson score interval around an observed event count out of a number of trials and reports whether the model's expected probability falls inside that interval, below it, or above it.

    rareEventBacktestAndConfidenceBounds(inputs)

What they share#

Every topic here is a record-transform, so once you have called one the rest follow the same shape. Import paths differ only in the final segment:

import { rocCurveAndRocAuc } from "fintech-algorithms/model-validation-and-backtesting/classification-and-score-validation/roc-curve-and-roc-auc";
import { precisionRecallCurveAndPrAuc } from "fintech-algorithms/model-validation-and-backtesting/classification-and-score-validation/precision-recall-curve-and-pr-auc";

Read them in the order above — the sequence is pedagogical, not alphabetical.

Where this sits#

Model Validation and Backtesting collects 10 algorithms across 1 family. For the concept behind this family rather than the call signatures, see the concept guides.