When we were building SpendWise, one of the things we wanted was pretty simple:

The AI should know something about the user before answering them.

Not in the creepy “I know everything about you” way.

More like:

If a user has already told us their income range, savings goal, spending habits, risk preference, and financial goals, why should the AI behave as if it is meeting them for the first time every time they ask a question?

That sounds easy.

Store the information somewhere, retrieve it later, and give it to the LLM.

But this is where things got interesting.

The moment you start dealing with structured user data and LLMs, you run into a question that looks much simpler than it actually is:

How do you make structured data searchable in a way that is actually useful to an AI system?

That was one of the things I started experimenting with while working on SpendWise.

And this is what I learned.

The data wasn't really the problem

Imagine that during onboarding, SpendWise asks a user questions like:

• What's your approximate monthly income?
• What are your major expenses?
• What are you saving for?
• What's your risk preference?
• Do you have investment experience?
• What's your short-term financial goal?
• What's your long-term financial goal?

The answers are naturally structured.

Something like:

{
  "income_range": "₹75k–₹1L/month",
  "monthly_expenses": "₹45k",
  "savings_goal": "Build emergency fund",
  "investment_experience": "Beginner",
  "risk_preference": "Moderate",
  "financial_goal": "Buy a house in 5 years"
}

From a traditional backend perspective, this isn't particularly complicated.

Put it into PostgreSQL.

Create some columns.

Query it when you need it.

Done.

But an LLM doesn't necessarily ask questions in the form of:

SELECT *
FROM user_profile
WHERE user_id = 123;

A user might ask:

"I have some extra money this month. Should I invest it or keep it aside?"

Now the useful context isn't just one field.

The answer could depend on several pieces of information:

• the user's risk preference,
• their emergency-fund status,
• their financial goals,
• their investment experience,
• possibly their spending pattern.

And that's where I started looking at vector search.

The first thing I had to understand: JSON is not a vector

This is probably the most important clarification in this entire topic.

When I first thought about putting structured data into a vector database, it was tempting to think:

"I'll just store my JSON in the vector DB and then search it."

Technically, that's not really how vector search works.

A vector database doesn't magically understand JSON.

The important object is the embedding.

You take some piece of content, pass it through an embedding model, and get a numerical representation:

Text / representation
        ↓
Embedding model
        ↓
[0.021, -0.183, 0.742, ...]
        ↓
Vector database

The vector is what enables semantic similarity search.

The original information can still be stored alongside that vector as metadata, payload, or another source of truth.

So there are actually two different things happening:

Semantic representation
        ↓
     Embedding
        ↓
   Vector search


Original structured data
        ↓
 JSON / database record
        ↓
 Metadata / source of truth

That distinction matters.

JSON is a representation of structured data.

An embedding is a representation of semantic meaning.

They are not the same thing.

So what did we actually want?

Our actual requirement in SpendWise was not:

"Let's put JSON into a vector database."

The requirement was:

"Let's make previously collected user context retrievable when it is relevant to a new question."

That's a much better way to think about the problem.

For example, suppose the user asks:

"I want to start investing, but I don't want to take too much risk."

The AI might need context such as:

{
  "investment_experience": "Beginner",
  "risk_preference": "Moderate",
  "financial_goal": "Buy a house in 5 years"
}

The interesting part is that the AI doesn't necessarily need every piece of user information.

It needs the relevant information.

And that distinction becomes extremely important as the system grows.

One approach: represent the structured data semantically

One thing you can do is take the structured information and create a semantic representation of it.

For example, instead of embedding this directly:

{
  "income_range": "₹75k–₹1L/month",
  "savings_goal": "Build emergency fund",
  "investment_experience": "Beginner",
  "risk_preference": "Moderate"
}

you could represent the same information as:

The user earns approximately ₹75k–₹1L per month.
They are currently focused on building an emergency fund.
They are a beginner investor.
Their risk preference is moderate.

Then generate an embedding from that representation.

Why might this help?

Because embedding models are generally designed to capture semantic relationships in content.

Natural language gives the model more context than arbitrary field names and values.

But there's an important caveat:

This isn't a universal rule that natural language is always better than JSON.

Embedding models and data formats vary.

If you're building something serious, the correct answer is to test it.

You could compare:

Raw JSON
vs
Natural-language representation
vs
A structured + natural-language hybrid

and measure retrieval quality.

That's much more useful than blindly following the assumption that one format is always superior.

But then I ran into another problem

Let's say we successfully embed our user information.

Now imagine the user asks:

"Show me all my expenses above ₹10,000 from last month."

Should I use vector search?

Probably not.

This is where the idea of putting everything into a vector database starts falling apart.

Because some questions aren't semantic-search problems.

They're database-query problems.

If I need:

expenses > ₹10,000

SQL is extremely good at that.

Something like:

SELECT *
FROM expenses
WHERE user_id = 123
  AND amount > 10000
  AND date >= '2026-09-01'
  AND date < '2026-10-01';

There is no reason to ask an embedding model to solve something that a database can solve deterministically.

And this led me to a much more useful way of thinking about the architecture.

SQL and Vector Search aren't competitors

I think this is one of the easiest mistakes to make when building AI applications.

You hear:

"We're building an AI application, so everything should go into a vector database."

Not really.

SQL and vector databases solve different problems.

If I need an exact value:

What is my monthly income?

A structured database is perfect.

If I need a range:

Show transactions above ₹20,000.

SQL is perfect.

If I need semantic similarity:

Have I made any unusually large purchases recently?

Vector search might be useful, depending on how the data is represented.

And if I need both?

That's where things get interesting.

The approach I like more: hybrid retrieval

Instead of asking one database to do everything, we can let each system do what it is good at.

A simplified architecture looks like this:

                    User Question
                          │
                          ▼
                    Query Analysis
                          │
              ┌───────────┴───────────┐
              ▼                       ▼
        Structured Query        Semantic Search
              │                       │
              ▼                       ▼
             SQL                  Vector DB
              │                       │
              └───────────┬───────────┘
                          ▼
                  Relevant Context
                          │
                          ▼
                         LLM
                          │
                          ▼
                       Answer

For example, consider:

"I've been spending a lot lately. Based on my financial goals, am I still on track?"

There are potentially two different types of information involved.

The structured side might tell us:

Monthly income
Monthly expenses
Savings amount
Current savings goal
Transaction totals

The semantic side might contain:

The user is trying to build an emergency fund
The user prefers moderate investment risk
The user wants to buy a house within five years
The user previously expressed concern about unnecessary spending

Now the LLM receives a much more useful context window.

Not the entire database.

Not every piece of telemetry.

Not every conversation.

Just the information relevant to the current question.

And that's the real problem: relevance

This changed how I think about vector databases.

Initially, the interesting part was:

"How do I store this data?"

But that's actually not the hard part.

The harder question is:

"How do I retrieve the right data at the right time?"

You can have a beautiful vector database with millions of embeddings and still build a terrible AI system.

Because if retrieval gives the LLM the wrong context, the LLM will confidently work with the wrong context.

That gives us a simple chain:

Bad retrieval
      ↓
Bad context
      ↓
Bad reasoning
      ↓
Bad answer

And throwing a bigger model at the final step doesn't necessarily fix it.

What about storing the JSON itself?

There is still a perfectly valid reason to keep the original JSON or structured record around.

In fact, I would argue that you usually should.

For example:

{
  "user_id": "123",
  "risk_preference": "moderate",
  "investment_experience": "beginner",
  "financial_goal": "buy_house",
  "target_year": 2031
}

This structured representation is useful because your application can directly access exact fields.

Your vector record could then look conceptually like:

Vector:
[0.12, -0.03, 0.81, ...]

Metadata:
{
  "user_id": "123",
  "profile_type": "financial_profile"
}

And somewhere else, PostgreSQL can remain the authoritative source for the structured profile.

This gives you a useful separation:

SQL stores what the system knows.

Vector search helps find what is semantically relevant.

That distinction is subtle, but it makes the architecture much easier to reason about.

There are actually three approaches here

After looking at the problem this way, I think structured data + AI retrieval can broadly be approached in three ways.

1. Structured data + embeddings

Keep your structured data as structured data, but create embeddings for the parts that benefit from semantic retrieval.

For example:

PostgreSQL
    │
    ├── income
    ├── expenses
    ├── goals
    ├── risk preference
    └── investment experience

             +

Vector DB
    │
    ├── semantic profile representation
    ├── user context
    └── relevant historical information

This works well when you have a combination of exact and semantic questions.

2. SQL-first retrieval

Sometimes you don't need vectors at all.

If most of your questions look like:

How much did I spend last month?

What's my average monthly expense?

How much have I saved this year?

then SQL is probably the better tool.

Adding a vector database just because the application contains an LLM would add complexity without solving a real problem.

That's an important engineering lesson:

AI doesn't automatically make deterministic systems obsolete.

Sometimes the best AI architecture contains a lot of boring SQL.

And that's okay.

3. Hybrid retrieval

This is probably the most interesting approach for systems like SpendWise.

Use SQL for deterministic information.

Use vector search for semantic information.

Then combine the retrieved context before sending it to the LLM.

For example:

User:
"Can I afford to invest more aggressively right now?"

             │
             ▼
      Query understanding
             │
       ┌─────┴─────┐
       ▼           ▼
      SQL        Vector
       │           │
       ▼           ▼
Current       Financial
financial     goals + relevant
numbers       user context
       │           │
       └─────┬─────┘
             ▼
      Context assembly
             │
             ▼
            LLM
             │
             ▼
          Response

Now the LLM isn't responsible for retrieving everything itself.

The application does the retrieval work first.

That's a much healthier architecture.

One experiment I'd definitely run

If I were rebuilding this from scratch, there's one experiment I'd want to run properly.

Take the same set of user profiles and create three representations.

Version A — Raw JSON

{
  "risk_preference": "moderate",
  "investment_experience": "beginner",
  "goal": "buy a house"
}

Version B — Natural language

The user is a beginner investor with a moderate risk preference.
Their primary financial goal is to buy a house.

Version C — Hybrid representation

Financial profile:
- Risk preference: moderate
- Investment experience: beginner
- Primary goal: buy a house

The user is relatively new to investing and prefers moderate risk.
Their long-term financial objective is purchasing a house.

Then test the same set of queries against all three.

For example:

"I don't want to take too much risk."

"Would a high-risk investment make sense for me?"

"What should I prioritize financially?"

"How should I invest for my house?"

Then measure:

• Did the correct profile get retrieved?
• How many relevant records were retrieved?
• Were irrelevant records included?
• How much context was passed to the LLM?
• How much did latency change?
• Did the final answer actually improve?

Now you're no longer saying:

"I think this works."

You're measuring it.

And that's the point where an AI experiment starts becoming an engineering experiment.

A mistake I would avoid

One thing I wouldn't recommend is embedding every single field just because you can.

For example, imagine storing:

User ID
Age
Country
Currency
Account ID
Created At
Updated At
Risk Preference
Income
Goal
...

and embedding the entire thing.

You might end up paying the cost of vector storage and retrieval without getting much semantic value from it.

Some information is simply better represented as structured metadata.

Some information benefits from semantic embeddings.

Some information doesn't need to be retrieved at all.

The goal isn't:

"Put everything into the vector database."

The goal is:

"Make the right information available to the model when it needs it."

Those sound similar.

They're actually very different engineering decisions.

The part I found most interesting

The more I worked through this, the more I realized that vector databases aren't really the main story.

The main story is representation.

The same user can exist in multiple representations:

                User
                 │
       ┌─────────┼─────────┐
       ▼         ▼         ▼
   SQL record   JSON    Embedding
       │         │         │
       │         │         │
 exact data   structured  semantic
              context     similarity

None of these representations is inherently "the correct one."

They are useful for different jobs.

SQL is great at answering:

"What is the value?"

JSON is great at representing:

"What does this object look like?"

An embedding is useful for:

"What information is semantically similar to this query?"

Once I started thinking about the problem this way, the architecture became much clearer.

So, what would I do in a production system?

For something like SpendWise, I wouldn't choose between SQL and a vector database.

I'd probably use both.

Something along these lines:

                    ┌───────────────────┐
                    │      User         │
                    └─────────┬─────────┘
                              │
                              ▼
                         User Query
                              │
                              ▼
                    ┌───────────────────┐
                    │ Retrieval Layer   │
                    └─────────┬─────────┘
                              │
                  ┌───────────┴───────────┐
                  │                       │
                  ▼                       ▼
            Structured                 Semantic
             Retrieval                Retrieval
                  │                       │
                  ▼                       ▼
              PostgreSQL              Vector DB
                  │                       │
                  └───────────┬───────────┘
                              ▼
                       Context Builder
                              │
                              ▼
                             LLM
                              │
                              ▼
                           Answer

The exact architecture would obviously depend on the application.

But the principle is what matters.

Don't make the vector database your source of truth.

Don't make SQL responsible for semantic similarity.

Don't make the LLM responsible for retrieving everything.

Give each component a job.

What I took away from building this

The biggest lesson for me wasn't about JSON, embeddings, or even vector databases.

It was this:

AI systems are often less about how much information you store and more about how intelligently you retrieve it.

You can give an LLM a huge amount of context and still get a worse answer.

You can give it a small amount of carefully selected context and get a much better one.

That's why I think the question:

"Where should I store this data?"

is often the wrong first question.

A better sequence is:

What kind of data is this?
        ↓
What kind of questions will I ask about it?
        ↓
Do those questions require exact lookup or semantic retrieval?
        ↓
What representation works best?
        ↓
How do I retrieve only the relevant context?
        ↓
How do I measure whether retrieval actually helped?

And only then:

"Which database should I use?"

Final thought

When we started working on SpendWise, the original problem felt like a storage problem.

We had user information.

We needed to save it.

We needed the AI to use it later.

Simple enough.

But once you start building the retrieval layer, you realize that storage is the easy part.

The interesting engineering problem is deciding what information the AI actually needs, how to represent that information, how to retrieve it, and how to prevent irrelevant context from polluting the answer.

That's where SQL, JSON, metadata, embeddings, vector databases, and LLMs start fitting together.

Not as replacements for each other.

As different pieces of the same system.

And honestly, that's the part I find much more interesting about building AI applications.

It's not just about making the model smarter.

It's about making the system around the model smarter.