Separate classifier calibration from the threshold for review

Choose a narrow event and a narrow consequence

A fictional AI application uses a binary statistical classifier to prioritize human editorial review of already permitted public drafts. Positive means “requires an editorial check under rubric R-A”; negative means “not selected by this triage rule.” Human reviewers define the rubric and labels independently of the classifier.

The target is review prioritization, not secret detection, document ownership or permission to publish. The score is not an LLM’s written claim that it is confident. It belongs to a separately identified statistical classifier and a defined positive class. A familiar percentage displayed by a UI supplies none of that identity.

All examples and eight decision cards below are PROPOSED_UNRUN. No classifier was fitted, calibrated or scored. Actual labels, independent data, distributions, uncertainty, utility and deployed behavior are unknown; missing prerequisites block a real probability or operating-point claim.

Ask a scale question before an action question

The scikit-learn calibration guide describes well-calibrated binary estimates through group frequencies: cases receiving probabilities near a value should have a corresponding positive-label frequency. Its reliability diagram compares mean predicted probability with fraction positive per bin, with bin counts providing context.

The threshold guide separates probability or score estimation from the decision based on it. A changed cutoff can change predicted labels while the probability outputs remain the same. Choosing more cases for review therefore does not establish that their percentages became better calibrated.

Keep two review questions visible: “what evidence supports this scale for this event and population?” and “which review decision serves the declared objective?” A single overall accuracy number cannot answer both. This worksheet selects no estimator, calibration method, fitting API or automatic threshold tuner.

Read two invented ledgers without calling them results

Consider two alternative teaching ledgers. In each, ten symbolic cases have an assigned displayed value of 0.8. In A, eight labels are deliberately assigned positive; in B, two are. These are invented assignments, not predictions or empirical observations. They are alternatives, not groups to pool into one evaluation.

The value alone does not tell a reader which label-frequency story has support. A real reliability record needs independent labels, the actual classifier and score definition, declared bins, mean estimates and denominators. Do not replace those fields with the sentence “the model is 80% confident.”

Even A’s deliberately matching arithmetic is not evidence of population calibration. It covers one invented value and ten assigned labels. It supplies no uncertainty estimate, rare-event coverage or evidence that future drafts follow the same distribution. Do not present it as an acceptance benchmark.

Make threshold utility an explicit application choice

The threshold documentation says the objective metric must be meaningful for the use case, including the correct positive class. It also warns against training the classifier and tuning its threshold on the same data.

Our proposed team separately declares the consequences of missed review cases and unnecessary reviews, the review capacity, and the metric it will inspect. Those are application choices, not values supplied by calibration. No numeric cutoff or cost ratio is recommended here.

Record the classifier, probability mapping and threshold as separate revisions. If the mapping changes, an old numeric cutoff is not automatically the same decision rule. If the utility or review capacity changes, review the operating point without falsely describing that policy change as recalibration.

Separate four evidence roles as a proposed policy

The calibration guide explains why calibrator fitting should use evidence independent of the data used to fit the classifier. For an already fitted classifier, it explicitly assigns responsibility for keeping classifier and calibrator fitting data disjoint.

For this worksheet, adopt a stricter original governance policy with four separately controlled roles: classifier fitting, calibration fitting, threshold selection and untouched final evaluation. This is our proposed policy, not a universal four-dataset requirement imposed by the documentation. Group related drafts and their variants appropriately; do not let a changed file name disguise reuse.

The evidence ledger binds each role to case provenance, label-rubric version, permitted exposure and decision history. The holdout guide addresses the broader exposure lifecycle. Calibration does not restore independence to labels already used to select the pipeline.

Walk eight inert probability and decision cards

Keep metrics and evidence claims qualified

The calibration guide cautions that Brier loss and log loss combine reliability, discrimination and uncertainty. A lower aggregate loss does not necessarily mean better calibration. No loss, binning, plot or uncertainty calculation was executed here.

The two actual maintained bodies identified scikit-learn 1.9.1 documentation when read on 30 October 2026. That establishes no installed library, representative dataset, fitted classifier or available review capacity. Unknown prerequisites keep the positive conditional.

Boundary is prelaunch and supplies no shipped calibration or review-triage service. Its existing synthetic policy evidence covers different behavior. Neither calibrated estimates nor threshold selection promise safe release, complete confidential-data detection, production performance or a customer outcome.