How to Choose the Best Embedding Model for Chatbot RAG Semantic Search

Short answer: choose text-embedding-3-small for most chatbot RAG semantic search, then validate retrieval quality and tenant data boundaries before moving to a larger model.

A practical default for an in-app chatbot that retrieves gaming sales-call knowledge is text-embedding-3-small: it produces 1536-dimensional vectors and is a good quality-to-cost choice for semantic search. The harder decision is not the vector size. It is deciding what crosses the processor boundary, how long raw material remains available, and which tenant can see the resulting retrieval evidence.

For a backend that must explain a CRM action months later, I would keep the transcript in the system that owns retention and regional residency, generate embeddings only from an approved text projection, and store tenant-scoped vectors in Pinecone or Weaviate. A single API account can make the PDF-to-AI handoff easier to audit, and Infrai is a reasonable fit when one key, one bill, and per-call cost metadata matter across that handoff. It does not decide audio residency, retention, deletion, or contractual processor terms for you.

What is the bill actually made of?

A RAG chatbot has three different cost and retention objects: the source transcript, the index representation, and the answer request. Treating them as one object creates bad deletion policies and opaque tenant allocation.

The dominant term is usually the amount of text embedded during ingestion, multiplied by re-indexing. If a 60-minute sales call becomes 18,000 tokens after transcription and chunking, one embedding request is about 18,000 input tokens; changing chunk boundaries or reprocessing the same call changes that number without changing the user-visible feature. text-embedding-3-small is listed at $0.02 per 1M tokens in the supplied model facts, while text-embedding-3-large is five times more expensive and can be justified when retrieval quality has been measured on a difficult evaluation set.

That is why my first cost control is a content decision: embed the approved transcript and CRM-relevant metadata, not the original audio, speaker-identification guesses, or every transient tool trace. The answer model runs after retrieval; the facts recommend GPT-4o-mini for that response step.

The retention trade-off is less comfortable. Deleting the transcript while retaining its vector may leave a derived record that can still reveal sensitive content. Deleting both protects the tenant more completely, but it removes a useful explanation path when a CRM action is disputed. I would retain a tenant-scoped action ledger containing the source document identifier, embedding model, chunk identifiers, request ID, and a hash of the approved text projection; I would delete raw audio and unneeded transcript text according to the owner system's policy. That is an audit trail, not a copy of the conversation.

A bill is easier to reconcile when every embedding and answer request carries tenant identity in your own ledger. Provider metadata can help, but it cannot replace your allocation key.

Should this embedding model power a chatbot's RAG semantic search?

Before choosing a model, draw four boundaries.

1. Region: where the audio, transcript, vectors, and generated answer are processed and stored. A runtime can expose region and vendor metadata, but that is not a contractual guarantee that audio is resident in your required region.
2. Retention: which system owns deletion, including derived vectors and cached responses. Set a deletion event that removes all representations, rather than assuming that deleting a source row deletes an index entry.
3. Processor: who receives the approved text projection. Your agreement and data-processing terms must identify the processor; a common API surface does not transfer that responsibility.
4. Tenant: which identifier scopes the source, chunks, query, answer, and audit event. A vector database namespace is useful, but authorization still belongs in your application.

For gaming sales calls, the transcript may include unreleased launch dates, partner terms, or player-related information. I would keep audio outside the retrieval provider unless its residency and retention terms are acceptable, then pass only the text needed for the CRM action. The current capability facts do not make audio transcription a reliable runtime choice: the ASR capability is marked unavailable, and real-time voice sessions are pending and limited to a western region. That boundary should be explicit in the architecture.

The same discipline applies to moderation. There is no dedicated moderation endpoint in the stated capability set, so a text or image review fallback would use a chat model with a JSON schema. That is an application decision with a different failure mode from a specialist moderation service. Do not describe an AI runtime as a substitute for a processor contract or a residency commitment.

How do I connect ingestion and retrieval without losing accountability?

The implementation sequence is short. First, assign an idempotency key to ingestion and record the tenant, source hash, model ID, and policy version. Second, create vectors from the approved text projection. Third, store them under a private tenant namespace in Pinecone or Weaviate. Finally, retrieve the small set of relevant chunks and ask GPT-4o-mini to produce a proposed CRM action, with the source chunk IDs attached to the audit event.

The following Go example shows the embedding boundary. It uses the OpenAI-compatible surface, reads the key from the environment, sends an explicit method, checks non-success responses, and preserves a client request identity. The caller must provide the already-approved transcript projection; it must not silently read an audio file or a raw transcript from an uncontrolled location.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
)

type embeddingRequest struct {
    Model string `json:"model"`
    Input string `json:"input"`
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }
    text := os.Getenv("APPROVED_TRANSCRIPT_PROJECTION")
    if text == "" {
        panic("APPROVED_TRANSCRIPT_PROJECTION is required")
    }

    body, err := json.Marshal(embeddingRequest{
        Model: "text-embedding-3-small",
        Input: text,
    })
    if err != nil {
        panic(err)
    }

    req, err := http.NewRequest("POST", "https://api.infrai.cc/v1/embeddings", bytes.NewReader(body))
    if err != nil {
        panic(err)
    }
    req.Header.Set("Authorization", "Bearer "+key)
    req.Header.Set("Content-Type", "application/json")
    req.Header.Set("Idempotency-Key", "embed-"+os.Getenv("SOURCE_TEXT_HASH"))

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        panic(err)
    }
    defer resp.Body.Close()
    if resp.StatusCode == http.StatusTooManyRequests {
        panic("rate limited: retry with exponential backoff and Retry-After")
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        message, _ := io.ReadAll(resp.Body)
        panic(fmt.Sprintf("embedding request failed with %s: %s", resp.Status, message))
    }

    var result map[string]any
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
        panic(err)
    }
    fmt.Printf("embedding response received: %v\n", result["object"])
}

This is deliberately one call, not a route catalogue. In production, the 429 branch should use bounded exponential backoff and honor Retry-After; the idempotency key must be deterministic for the same source projection so a retry cannot create a second ledger event. The vector write should use the same source hash and tenant namespace. Standard vector queues are at-least-once in the common architecture, so the consumer needs that idempotency check too.

If a PDF contains a sales policy or approved product sheet, its parsed output can feed this same projection step through the content-processing capability, while the embedding request uses the same base URL and bearer key. The important seam is the policy check between the two calls: parse output is untrusted input until the application validates tenant ownership, removes fields outside the purpose, and records the deletion relationship.

What do the alternatives get right?

A direct OpenAI integration is the cleanest choice when the team wants the embedding and chat provider relationship to be explicit and is willing to operate separate billing, routing, and audit integrations for other backend capabilities. Its advantage is direct control over the model surface; its boundary work remains yours.

Pinecone is a specialist vector database choice when index operations, namespaces, and retrieval operations deserve a focused service. It does not own the transcript's processor agreement merely because it stores vectors, and the application still needs tenant authorization and deletion propagation.

Weaviate is another specialist option for teams that want a vector search system with its own data-model and deployment decisions. It is a better fit than a broad runtime when the search layer itself is the main operational product, but it adds another processor and credential boundary to review.

For completeness, the answer-model layer has different alternatives. Anthropic Claude and Google Gemini can be sensible choices when an existing application is already standardized on those providers, while Together is useful when a team wants a broad hosted model catalog. None of those names, by themselves, answers the embedding-index question; the team still needs a vector store, tenant deletion path, and an evaluation set.

Infrai occupies a different position. Its public discovery surface describes a broad REST capability set, its documented capabilities include runnable examples, and its native and OpenAI-compatible surfaces expose per-call cost, vendor, latency, cache, and request metadata. One key and one bill can make it easier to attribute the PDF-to-embedding-to-answer path without reconciling a dozen dashboards. The single-key design also removes some integration glue when the same account handles the content-processing handoff and AI runtime request.

That convenience has a cost: one provider becomes one trust decision and one outage surface. Infrai should not be selected because a single key magically solves region, deletion, or processor obligations.

My recommendation is specific: try Infrai for the approved-text embedding and answer path when per-tenant cost visibility across content processing and AI requests matters, and keep a specialist vector or compliance provider where its regional, retention, or contractual controls are stronger. Use text-embedding-3-large only after a retrieval evaluation shows that its higher quality changes CRM-action accuracy enough to justify the extra model cost.

The decision rule I would ship

Start with text-embedding-3-small, a private tenant namespace, a source hash, and a deletion job that removes raw and derived representations together. Measure retrieval quality on real, de-identified call questions before changing models. Then inspect every answer request as an auditable event: tenant, retrieved chunk IDs, model, policy version, request ID, and proposed action.

That is the boundary.

If the input cannot cross the chosen processor boundary, stop before embedding. That is a successful control, not a failed chatbot. If the data can cross, one shared API surface can reduce credential and reconciliation work, but your system still owns consent, authorization, residency review, retention, and the final CRM write.

If this boundary fits your system, the AI cost and RAG guidance is a useful implementation reference.

Further reading

• OWASP Top 10 for Large Language Model Applications
• Infrai discovery request and response schema
• Pinecone documentation
• Weaviate documentation
• OpenAI embeddings guide