Why deleted documents can still appear in AI answers

Define what the deletion request promises

A knowledge-base owner removes a document, then an assistant quotes its old paragraph. The source interface may have completed its task while the application still has chunks, cached answers or a conversation summary. OWASP's RAG guidance calls for deletion to propagate to vector stores, derived indexes and response caches rather than assuming source removal handles them automatically.

Specify three separate states: request accepted, new application use blocked, and controlled-copy cleanup verified. Define which retained copies remain and why. Do not display “erased everywhere” because a delete endpoint returned success. This guide proposes a synthetic review exercise; no document deletion, model call, vector-store operation or recovery experiment has been run.

Build a multi-copy matrix with one fake document

Use fictional document DOC-17, revision 1, containing the unique phrase “DEMO violet return window.” At time T0, its owner requests removal from this application's knowledge workflow. A separate current document DOC-18 is a positive control: normal authorized retrieval should remain useful. The phrase is a harmless fixture marker, not a secret or a substitute for checking provenance.

This is an expected-state matrix, not evidence that these states have occurred. Give each row an owner and a completion criterion. If the application has another derivative, add a row instead of assuming the table is complete.

Stop stale reads before waiting for physical cleanup

A proposed design keeps a durable registry of document status and generation. Retrieval candidates carry document ID and source revision. Before admitting them to model context, the application checks that the current registry still permits their use. Cached answers need their source dependencies too; a cache key alone cannot explain which deleted document influenced a saved answer.

Document how this check coordinates with deletion. A status check followed by an unrelated later dispatch leaves a race: deletion may arrive between them. Define the point after which a request counts as already in flight, using an appropriate transaction or coordinated application gate. Requests beyond that point need separate handling rather than an unsupported immediate-stop promise.

If current status cannot be established, the proposed policy withholds affected context and explains that knowledge is temporarily unavailable. Keep cleanup asynchronous if necessary, but track it separately from the read decision. A denial at this gate does not prove that stored bytes are physically gone.

Prevent an old worker or restore from resurrecting the copy

Imagine an ingestion job fetched revision 1 just before T0 and finishes afterwards. Coordinate the generation check and chunk publication with deletion through an atomic, fenced or otherwise coordinated commit that the actual store supports. A separate check followed by a later write can still race deletion. If the store cannot coordinate that commit, keep the current read-time exclusion authoritative while stale records are reconciled; do not claim that a pre-write check prevents their reappearance. A stale job should be rejected even if its extraction succeeded.

Plan for backup restoration as well: restored data should not re-enter retrieval until applicable deletion records have been reconciled. Keep deletion metadata long enough for that purpose under the system's defined retention policy; avoid copying the removed paragraph into that metadata.

Storage semantics require their own verification. For example, Amazon S3 documents that a simple delete in a versioning-enabled bucket inserts a delete marker while previous versions remain. That specific behavior illustrates why disappearance from a normal read is different from version removal; it says nothing about the reader's actual storage configuration.

Check conversation context and already-dispatched content

Reopening a conversation can reintroduce the old paragraph without another search. Include retained messages, summaries and application memory in the future-context policy. Decide whether affected history is hidden, annotated or retained with restricted access, and separately decide whether it may be sent to a model again. A cosmetic citation change does not remove underlying text from a context bundle.

A request already dispatched before the boundary is a different case. Cancellation or withholding its displayed answer cannot establish that the receiver never processed the payload. Provider copies, user exports and downloaded answers need their own evidence and applicable procedures. Keep unknowns explicit; this application exercise cannot recall information from every recipient or guarantee removal from a model's learned behavior.

Verify the boundary with a small proposed exercise

Before any rollout, an independent reviewer could use local storage substitutes and a recording model stub. Propose four checks: a warm cache after T0, a delayed ingestion commit, a reopened conversation, and a simulated restore. Inspect the exact proposed context and decision for each. DOC-17 must not enter a new permitted context, while DOC-18 remains usable. None of these checks has been run here.

A missing fixture phrase in one generated answer is weak evidence: the model might simply choose different wording. Record source IDs, generations, affected cache dependencies, context admission and cleanup states. Following OWASP logging guidance, avoid placing sensitive content or credentials in ordinary evidence logs; use minimized identifiers and restricted access.

Boundary's planned request router is not a cascading deletion service, knowledge-store reconciler or erasure guarantee. The separate eight mock policy checks do not validate this lifecycle. A useful completion report names the controlled paths checked, their actual results, retained copies and unresolved scope.