Define an embedding and index contract before retrieval cutover

Choose one concrete store contract

This proposed design uses PostgreSQL 18 and pgvector 0.8.7 maintainer documentation, with separate generation tables and cosine HNSW indexes. It is an application design, not a pgvector migration feature. No database, encoder, or corpus was created.

Invent Cedar Notes generation A with a two-dimensional encoder and candidate B with a three-dimensional encoder. Their names and settings below are fictional. Changing only the column type would leave unanswered which encoder produced each document and query vector, and which source revision the replacement covers.

Pin the two encoder roles and the query recipe

B’s immutable contract names fictional family Beta, checkpoint B1, tokenizer TB1, mean pooling excluding padding, and float32 output of dimension three. Document input uses prefix passage:; query input uses query:. Each complete serialized input is limited to eighty TB1 tokens, including its role prefix and every required special token. Documents are split left-to-right into the longest fitting nonempty contiguous segments at Unicode code-point boundaries, with zero overlap; count the complete serialized input for each segment. Reject if no nonempty segment fits. Reject an over-limit query; neither role silently truncates content. Both preserve case, convert line endings to LF, and apply no character normalization or extra L2 normalization. Record the complete configuration digest, not merely the family name.

A retains its own Alpha/A1/TA1 two-dimensional contract and role-specific settings. Never relabel A’s stored vectors as B or assume that another three-dimensional family Gamma is interchangeable. The metadata gate can enforce declared provenance; it cannot establish that an encoder really implements those semantics.

For B, choose a vector(3) column and HNSW vector_cosine_ops, with recorded build settings m=16 and ef_construction=64. The query recipe sets hnsw.ef_search=80 transaction-locally and orders directly by embedding <=> $1::vector(3), ascending, with limit five. These choices are illustrative, not tuned recommendations. The maintainer documentation maps that operator and operator class to cosine distance.

Select the table and recipe from trusted immutable configuration, bind query values, and enforce current access separately. Include generation and recipe identity in cached-answer provenance. Do not substitute inner-product or similarity ordering while claiming to use the same index contract.

Use a frozen corpus manifest instead of a vague backfill percentage

For this narrow migration, deliberately pause new-document and content-update publication. Under a coordinator transaction, lock the application’s route row, enter BUILDING, and advance its epoch. Every content-writer commit must acquire that same row and enforce its mode and epoch, so work prepared before the pause cannot publish afterwards. Explain the temporary ingestion hold to operators.

After earlier admitted commits finish, capture W: the exact frozen document/chunk revisions and digests. W is a manifest, not the largest allocated sequence number or a row count. Deletions and permission restrictions remain available throughout the pause, tracked by current restriction revision R; do not postpone mandatory revocation to finish rebuilding.

Candidate-B workers may write only their declared generation. Inside their coordinated commit, check BUILDING epoch, current source revision, and tombstones. Reconcile B’s coverage against W minus currently excluded documents. Reject missing chunks, duplicate identities, stale content digests, and invalid vectors. Cosine indexes omit zero and NULL vectors, so matching table-row counts cannot establish indexed coverage.

Make the cutover transaction smaller than the rebuild

PostgreSQL 18 row-lock documentation explains that SELECT FOR UPDATE blocks conflicting row writers and lockers until transaction end. It does not automatically stop unrelated application paths. All participating publishers, restriction updates, and the coordinator must follow the designed locking protocol; ordinary reads are not blocked by that row lock.

Build and review B outside the short cutover transaction. Then lock the route row, verify the candidate evidence still matches W, current R, and the complete B configuration, and atomically change the active contract reference and epoch. If evidence is stale or incomplete, leave A active and release the lock. A lock does not manufacture missing embeddings or quality evidence.

PostgreSQL’s isolation documentation notes that Read Committed statements can see different snapshots. Read an immutable contract once per query: embed with that contract, then use its table and recipe. A query started under A must not switch to B after embedding; this design rejects release on an epoch change and requires a fresh query rather than mixing generations.

Predeclare the migration acceptance cases

These are expected routing and lifecycle decisions, not observed retrieval results. A recorder supplied with labels A/B/Gamma would test only its own comparisons.

Keep rollback and retrieval quality as separate gates

After B accepts new content, retained A becomes stale unless maintained deliberately. Rollback requires A’s own encodings for current eligible revisions plus current restrictions and configuration approval. Re-encoding with Alpha is different from copying Beta vectors into A. If that preparation cannot be established, stop affected retrieval rather than restore a stale corpus or permission snapshot.

Separately propose an authorized held-out query set with human-reviewed relevant document and citation IDs. Compare B’s approximate results with an exact scan using the same B vectors and metric, then assess relevance against the human key. Approximate-versus-exact agreement measures index behavior, not semantic correctness. Predeclare acceptable misses and access failures; filtering can reduce returned candidates, so limit five does not promise five useful hits.

All checks are PROPOSED_UNRUN: no database, embedding, model, network experiment, or ranking measurement occurred. Boundary is a prelaunch commercial model router with planned request-path policies, not a shipped vector migration coordinator or retrieval-quality service. Existing tenant and deletion controls remain necessary on both generations.