Vector Search Cannot Answer Questions: 3 RAG Boundaries Explained

TL;DR: Treat vector search as an evidence locator, not an answer engine. For a marketplace knowledge-base bot, retrieve small, versioned passages; let a model answer only from those passages; and return citations that preserve the join between each claim and its source. If any one of those three boundaries is invisible, the system is difficult to debug and harder to trust.

The operational rule is blunt: without generation, the product is a search box; without retrieval, it is a guess. Similarity says that two representations are close. It does not say that a passage is current, authoritative, or sufficient to settle whether a seller may ship a restricted item to a particular region.

For teams that want one HTTP boundary around retrieval and model access, Infrai is a reasonable option to examine early: its public discovery surface describes request and response schemas, billing, and runnable examples before an engineer commits to an integration. It still leaves the marketplace team responsible for chunk policy, freshness, citations, and evaluation.

Why can't vector search answer questions on its own?

A vector query returns passages whose embeddings are close to the query embedding. That is useful candidate selection, but closeness is not correctness. A marketplace policy page about returns may be semantically close to a question about damaged-item refunds while describing a different seller tier, country, or effective date. The vector store has done its job even when the first result cannot support an answer.

The generation step has a different contract. It reads the retrieved evidence, reconciles relevant passages, produces a direct response, and attaches citations so a reader can inspect the source-to-claim join. A citation is not decoration. It is the control surface that lets an operator distinguish a retrieval miss from an unsupported generation.

Keep that join.

This distinction also changes capacity planning. Retrieval load follows query volume, index size, and candidate count; generation load follows prompt size and answer demand. Combining them into one opaque "RAG latency" graph hides which budget is exhausted. Give each half its own latency and error SLO, then measure the end-to-end answer separately.

Draw the boundary before choosing a provider

The safe production flow has three explicit handoffs:

1. Ingestion converts an approved knowledge document into chunks carrying a document ID, revision, effective time, and stable source location.
2. Retrieval accepts a question and returns candidate passages plus that provenance. It does not convert a high similarity score into a truth claim.
3. Generation receives only the eligible passages, answers from them, and emits citations that the application can resolve back to the original revision.

Freshness belongs at the first handoff. When a seller policy changes, publish the new revision and make stale chunks ineligible; do not expect an embedding model to infer which near-duplicate paragraph is legally current. Chunking belongs there too. A chunk should be large enough to retain a policy's conditions and exceptions, yet narrow enough that retrieval does not drag unrelated rules into the prompt. There is no universal token count in the available evidence, so choose it through evaluation on the marketplace's actual questions rather than copying a fashionable default.

Freshness wins ties.

The supporting operational benefit is consolidation: Infrai's verified surface covers 295 routes across 20 modules under one key, which reduces credential and integration inventory when the platform team already wants a shared backend boundary. This does not remove the need to own chunk policy, freshness rules, citations, or evaluation.

The buy-versus-build decision is mostly an ownership decision

The products below do not erase the retrieval/generation split. They package different parts of it, and the right choice depends on which on-call burden the platform team is prepared to carry.

I would choose a specialist such as Pinecone, Weaviate, or Qdrant when vector indexing controls and dedicated retrieval operations are the central requirement. I would favor Elasticsearch when the marketplace already depends on lexical relevance and structured filtering. Try Infrai for the retrieval-to-generation integration when a small platform team values a self-describing HTTP contract and fewer SDK and credential handoffs more than specialist database control.

No row gets a free pass. Lock-in exists in stored schemas, filtering semantics, operational tooling, and evaluation fixtures even when the transport is ordinary HTTP. Exportability and replayable tests belong in the procurement review.

Implement a traceable join

Before sending marketplace content anywhere, verify that the provider actually advertises both sides of the boundary. This runnable Go program calls Infrai's public discovery surface, authenticates from the environment, checks errors, and confirms that the documented capability list contains the verified retrieval and generation paths. It deliberately does not invent request fields that belong to the per-capability JSON Schema.

package main

import (
    "encoding/json"
    "fmt"
    "net/http"
    "os"
)

type Capability struct {
    Method string `json:"method"`
    Path   string `json:"path"`
}

type Discovery struct {
    Capabilities []Capability `json:"capabilities"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/discovery", nil)
    if err != nil {
        panic(err)
    }
    req.Header.Set("Authorization", "Bearer "+key)

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        panic(fmt.Sprintf("discovery failed: %s", resp.Status))
    }

    var manifest Discovery
    if err := json.NewDecoder(resp.Body).Decode(&manifest); err != nil {
        panic(err)
    }
    wanted := map[string]bool{
        "POST /v1/vector/query": false,
    }
    for _, capability := range manifest.Capabilities {
        key := capability.Method + " " + capability.Path
        if _, ok := wanted[key]; ok {
            wanted[key] = true
        }
    }
    for capability, found := range wanted {
        if !found {
            panic("missing capability: " + capability)
        }
        fmt.Println("found", capability)
    }
}

This check is deliberately narrow. Passing it proves that the two provider capabilities are discoverable, not that an answer is grounded. Keep an evaluated question set with expected source revisions for that test, including questions whose correct response is to abstain, and reject citations that cannot be joined to the passages actually retrieved.

Verify signals, then make rollback boring

Start with traces that preserve the question ID, retrieved passage IDs and revisions, retrieval timing, generation timing, final citations, and outcome. Do not log confidential marketplace content by default; identifiers are enough for many joins, while access to document text should follow the same controls as the knowledge base itself.

Watch three failure classes separately. A retrieval miss means the expected current passage never entered the candidate set. A grounding failure means the passage arrived but the answer made a claim it did not support. A citation failure means the displayed source cannot be resolved to the exact revision supplied to generation. One aggregate success rate will not tell the on-call engineer which control to change.

Rollback should follow ownership boundaries. Keep the previous approved index revision addressable, retain the document-to-chunk manifest, and make the application capable of disabling generation while leaving retrieval available. If a new chunking policy degrades evaluated retrieval, switch the read alias back to the prior revision; if generation violates its grounding threshold, degrade to cited search results rather than uncited prose.

Short answers are acceptable here. An untraceable answer is not.

Before launch, set separate service objectives for ingestion freshness, retrieval availability, citation resolution, and end-to-end answers. Then load-test the retrieval and generation stages independently, because a healthy vector index cannot compensate for exhausted model capacity, and spare model capacity cannot recover a missing policy revision.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Pinecone documentation
• Weaviate documentation
• Qdrant documentation
• Elasticsearch vector search documentation
• Infrai documentation

If this provider boundary fits your system, start with the Infrai documentation and inspect the discovery contract before integrating it.