Evaluate AI gateway security claims before a pilot

Turn the claim into one decision

“Protects sensitive data” is too broad to accept or reject. Rewrite it as a concrete question: will this version stop a request containing a defined restricted marker before the configured provider receives it? Name the payload field, representation, destination, caller scope, and failure behavior. A narrower claim gives both buyer and supplier something observable.

OWASP AISVS supplies testable AI security requirements and supports referencing versioned requirements in procurement. It is a framework for checking controls, not a list of approved products. Choose requirements relevant to the workflow and verify ordinary application security separately.

This article provides an original buyer worksheet. It reports no vendor assessment, pilot result, vulnerability, or product recommendation.

Request a scope map before a dashboard tour

Draw the intended route from your application to the gateway and provider. Add retries, fallback providers, background jobs, file uploads, streaming, and direct-provider clients if the workflow uses them. Mark which component controls each path and which path has not been integrated.

Then separate data checks from agent actions. A gateway observing model requests may have no authority over a tool that deletes files, sends email, or charges a payment method. Ask for the enforcement point on each claimed action rather than assuming a shared brand or dashboard creates coverage.

Build a four-case synthetic acceptance packet

Use invented data and an isolated environment you are authorized to evaluate. For an example request-screening claim, agree the literal marker FAKE_TOKEN_PILOT_7 as restricted. It is a teaching string, not a credential. Set the expected decisions before looking at the candidate's response.

1. Permitted request: an ordinary public summary should reach the recording adapter and return the expected application result.
2. Restricted marker: the agreed marker in the supported message field should produce the configured rejection and zero downstream dispatches.
3. Disallowed destination: a fallback not permitted by the route policy should receive no payload, even if the primary route fails.
4. Unavailable checker: a simulated timeout should follow the agreed failure policy and produce evidence of that decision.

Record what the receiver observed, not only what the UI said. A recording adapter can establish local dispatch behavior in that harness. It does not establish that a supplier's production endpoint has the same wiring. State that distinction in the packet and identify any remaining deployment check.

Check usefulness and omissions alongside rejection

A system that blocks every request can pass a denial-only demonstration. Keep the permitted counterpart and inspect the output the application actually needs. Include a benign quoted marker if your intended workflow handles public documentation; decide whether that quotation should be permitted, transformed, or explicitly unsupported.

Ask how false positives and missed restricted cases are labeled, sampled, and reported. Four fixtures cannot establish a production detection rate. Do not infer broad confidential-information recognition from success on a literal marker. Request the supported data classes, encoding assumptions, field coverage, and limits in writing.

For a feature you need but the supplier does not support, record a gap rather than negotiating a stronger adjective. You may narrow the pilot to supported data, add your own control, or defer it.

Keep an evidence ledger with a clear status

NIST's AI RMF Measure function emphasizes documented evaluation, uncertainty, and independent review. Apply that idea with a short row for each important claim: required behavior, component/version, fixture, expected observation, actual evidence, reviewer, date, and remaining limitation.

Use statuses such as documented, locally reproduced, observed in the agreed deployed integration, failed, or untested. A screenshot of a policy editor establishes that an interface displayed a setting; it does not prove that every worker activated it. A successful administration response does not establish effective revocation on a cached path.

Attach the exact configuration and source or artifact identity needed to interpret the result. Keep real prompts, customer records, credentials, and personal identifiers out of the shared packet. A synthetic fixture is usually sufficient for the initial question.

Agree a pilot boundary and an exit

Before granting access, define the workflow, responsible operator, allowed data, accounts, duration, spending allowance if applicable, success criteria, and stop criteria. Decide who can disable the integration and how the application behaves while it is disabled. A pilot invitation is not permission to broaden access later.

Ask what happens after an error: who receives an incident report, what minimal evidence can be retained, how an uncertain provider outcome is handled, and what removal of the integration leaves behind. Confirm availability and commercial terms explicitly. Do not assume that an educational walkthrough is a free production pilot or a service commitment.

End with one decision: proceed within the agreed boundary, proceed only after a named gap is resolved, or stop. Keep quality, operational burden, and security evidence visible rather than combining them into an unexplained score.

Apply the same standard to Boundary

Boundary is a prelaunch commercial model router. Defined pattern checks, model/provider policies, and usage limits on routed requests are planned. This checklist does not turn those plans into available enforcement capabilities.

The existing eight local mock-policy tests concern a separate synthetic client-side action evaluator. The synthetic CSV exercise demonstrates that educational interface without deleting files. Neither is evidence of a gateway pilot, real model comparison, prevention rate, or customer outcome. Apply the same scoped evidence questions before treating any later implementation as ready for your workflow.