Using LLM for Information Retrieval

We are building a lightweight semantic search pipeline that embeds a small document corpus with Oxlo.ai embeddings, retrieves the most relevant passages for a query, and synthesizes an answer with an LLM. This pattern fits any internal wiki, product documentation, or support knowledge base where you need accurate retrieval without running a separate vector database.

What you'll need

• Python 3.10 or newer
• pip install openai numpy
• An Oxlo.ai API key from https://portal.oxlo.ai (the Free plan gives you 60 requests per day, enough to test this script)

Step 1: Configure the Oxlo.ai client

I start by initializing the OpenAI SDK to point at Oxlo.ai. Because Oxlo.ai is fully OpenAI API compatible, this is the only client we need for both embeddings and chat.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

Step 2: Prepare the document corpus

For this demo I use a hardcoded list of strings representing chunks from a fictional product manual. In production you would load these from Markdown files, a database, or a CMS.

CORPUS = [
    "Zephyr Analytics supports real-time event streaming via Kafka and WebSocket connections.",
    "The Zephyr dashboard uses React components with server-side rendering for performance.",
    "Zephyr's pricing is based on monthly active users and data retention periods.",
    "Authentication is handled through OAuth 2.0 and SAML 2.0 single sign-on providers.",
    "The REST API returns JSON responses and accepts gzip-compressed request bodies.",
]

Step 3: Embed the corpus

Oxlo.ai offers BGE-Large and E5-Large embedding models. I batch the corpus through BGE-Large to get dense vectors, then store them in memory. For larger datasets you would persist these to disk.

import numpy as np

def get_embeddings(texts):
    response = client.embeddings.create(
        model="bge-large",
        input=texts
    )
    return [np.array(item.embedding) for item in response.data]

doc_embeddings = get_embeddings(CORPUS)

Step 4: Implement retrieval

I compute cosine similarity between the query embedding and every document embedding, then return the top two matches. This keeps the LLM context short and focused.

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def retrieve(query, top_k=2):
    query_embedding = get_embeddings([query])[0]
    scores = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
    top_indices = np.argsort(scores)[-top_k:][::-1]
    return [CORPUS[i] for i in top_indices]

Step 5: Generate answers with an LLM

Now I feed the retrieved chunks to Llama 3.3 70B with a strict system prompt that forbids hallucination. Because Oxlo.ai uses request-based pricing, passing long context with multiple chunks costs the same flat rate as a short ping. See https://oxlo.ai/pricing for details.

SYSTEM_PROMPT = (
    "You are a technical support assistant. Answer the user's question using only "
    "the provided context. If the context does not contain the answer, say "
    "'I don't have that information.' Keep responses under three sentences."
)

def generate_answer(query, context_chunks):
    context = "\n".join(f"- {chunk}" for chunk in context_chunks)
    user_message = f"Context:\n{context}\n\nQuestion: {query}"

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content

Run it

Running the script below ties retrieval and generation together. I ask about authentication, which lives in the fourth document chunk.

if __name__ == "__main__":
    query = "How does Zephyr handle authentication?"
    chunks = retrieve(query)
    answer = generate_answer(query, chunks)

    print(f"Q: {query}")
    print(f"A: {answer}")
    print("\nRetrieved chunks:")
    for c in chunks:
        print(f"  - {c}")

Expected output:

Q: How does Zephyr handle authentication?
A: Zephyr handles authentication through OAuth 2.0 and SAML 2.0 single sign-on providers.

Retrieved chunks:
  - Authentication is handled through OAuth 2.0 and SAML 2.0 single sign-on providers.
  - The REST API returns JSON responses and accepts gzip-compressed request bodies.

Next steps

Cache the document embeddings in a local NumPy file or SQLite table so you do not re-embed on every restart. If you need stronger reasoning over the retrieved chunks, swap Llama 3.3 70B for Kimi K2.6 or DeepSeek V3.2 on Oxlo.ai without changing any client code.