Keep a model evaluation holdout clean during prompt development

Protect the decision, not just a file called test

A team can keep a separate test file while copying its difficult examples into a prompt or repeatedly selecting configurations against its scores. The file remains separate; the decision process has absorbed its answers.

Google’s dataset-splitting guidance explains why training, validation, and final testing have different roles, and why repeated decision-making against a test set weakens its value. For a prompt-based application, apply the separation to examples, retrieval settings, routing choices, and grading changes as well. This is an evaluation-governance recommendation, not a claim that prompt editing retrains model weights.

Create an issue-grouped split manifest

Consider a fictional support-labeling task with billing, technical, and other labels. Invent twelve messages from six independent seed issues, with two wording variants per issue. A reviewer writes the answer key before comparing candidates. No real tickets, personal data, or model outputs are involved.

Assign the parent issue before generating or splitting its variants. Keep the same incident, thread, copied template, and paraphrases together when they share the information the test should withhold. The scikit-learn GroupKFold documentation describes non-overlapping groups across folds; that principle motivates this simple fixed manifest. This worksheet does not run cross-validation or use the library.

The counts demonstrate bookkeeping, not a recommended split ratio. Four holdout messages represent only two seed groups, with correlated variants. They cannot establish production accuracy or rare-failure rates. This published worksheet is not a secret benchmark; a real evaluation needs separately controlled, authorized cases suitable for its decision.

Check provenance and similarity before freezing the split

Store case identifier, parent issue, derivation parent, split, corpus version, and content digest in a protected manifest. First propose checks for duplicate identifiers, parent groups crossing splits, and identical content digests. Digests detect byte equality; they do not identify paraphrases.

For example, “Invoice repeats charge INV-7” and “Please check the duplicate charge on INV-7” have different text but the same invented issue. A row-wise shuffle could separate them. Assigning both to G-A preserves their relationship. Review shared template origins and any synthetic generation lineage, not just matching words.

A similarity method can flag candidate near-duplicates for human review. Document its normalization, threshold, and limitations before scoring; no threshold proves semantic independence. Preserve distinctions that affect the expected label. If a suspicious pair crosses a split, resolve the relationship and publish a new manifest version before evaluation. Do not remove inconvenient cases after seeing failures.

Separate development feedback from final evaluation access

Let developers inspect development and validation cases. Keep holdout cases and labels under a separate evaluation custodian, with controlled access and a record of exposure. When the evaluator eventually sends a case to the candidate, send only permitted task input, never the answer key or grading rationale.

Scikit-learn’s leakage guidance warns against using test data to choose a model or fit preprocessing. If your workflow learns a normalizer, threshold, or feature selection, fit it only on the designated development data and freeze it for evaluation. An unchanged transformation can then be applied to held-out input.

Also inspect prompt examples, retrieval indexes, cached answers, demonstrations, and evaluator context for case or label leakage. Clean local separation cannot prove that a provider’s earlier training never contained similar material. Keep that uncertainty explicit, especially for public benchmarks.

Predeclare a comparison that can return no decision

Before opening this toy holdout, freeze two candidate configurations, the four case identifiers, expected labels, schema rules, one scheduled trial per case, and the treatment of errors or missing outputs. The rubric requires a valid label matching the independent key. Use the same input and grading rule for both candidates; do not omit an outage from the denominator.

Declare this fictional screening rule: invalidate the comparison if split leakage or answer-key exposure is found; return inconclusive if grading cannot be resolved. Otherwise, advance the challenger only if it passes at least as many scheduled cases as the baseline, improves at least one case, and introduces no new failing case. Any task-specific safety gate must also pass separately.

These are invented decision rules, not observed results or statistical significance thresholds. Advancement means considering a larger, representative evaluation, not deployment, purchase, or a universal model ranking. Preserve per-case outcomes and group membership so repeated paraphrases cannot look like many independent successes.

Retire exposed cases honestly

If a holdout result leads to a prompt correction or configuration choice, record that exposure. Move those cases into a regression collection where they remain useful, then obtain fresh, independently reviewed groups for a later final decision. Renaming files, changing identifiers, or lightly paraphrasing exposed examples does not restore independence.

Keep the original manifest, configuration digests, rubric version, access history, exclusions, and unresolved limitations. Changes to a wrong label need documented adjudication, not a silent edit that improves a candidate’s score. A passing small fixture remains narrow evidence about those cases and conditions.

Every split, overlap check, and comparison here is proposed and UNRUN. No model calls, scores, contamination detection, or production results are claimed. Boundary is a prelaunch commercial model router with planned request-path policies; this guide does not describe a shipped evaluation service or holdout-enforcement engine.