How I Modeled Audit History as Memory in Hindsight.?

How I Modeled Audit History as Memory in Hindsight

Most of the work in building VERIQ wasn't about the model, prompts, or user interface. It was about deciding what an "audit memory" should actually represent.

When you give a memory system your data, you're making important modelling decisions, whether you realise it or not. You need to decide what constitutes one unit of memory, how it is identified, who can access it, and how it should be labelled.

I made several of these decisions incorrectly during the early stages of development. Refining them helped me build a more structured and reliable approach to storing and retrieving historical audit information.

This article explains the technical decisions behind VERIQ's memory architecture and how Hindsight helps represent, retain, and retrieve audit history.

1. Understanding the System

VERIQ is designed to manage compliance and operational audit findings, corrective actions, and resolutions. It helps audit teams retrieve relevant information from previous audit cycles instead of repeatedly reviewing historical records manually.

The memory layer is powered by Hindsight, an open-source memory system that extracts structured facts from retained information, connects related entities, and supports retrieval through semantic, keyword, graph, and temporal search.

The main objective was to make historical audit information useful beyond the cycle in which it was originally recorded.

To achieve this, I focused on four important modelling decisions.

1. Decision One: One Memory Bank per Organisation

A Hindsight memory bank provides an isolated memory space. This is particularly important for a multi-organisation compliance platform.

For example, information belonging to Organisation A must never appear in responses generated for Organisation B.

I chose to create a separate memory bank for each organisation:

def bank_for(org_id: str) -> str:
return f"org-{org_id}"

I also considered using a single shared memory bank with tags to separate organisations. However, that approach would depend on applying the correct filtering conditions every time a recall request was made.

A missing filter could expose information belonging to another organisation.

Separate memory banks provide a stronger structural boundary. Instead of relying entirely on developers to remember a filtering rule, the application separates organisational memory at the bank level.

For a compliance-focused platform, this separation is an important architectural decision.

1. Decision Two: Using Tags for Internal Scoping

Although each organisation has its own memory bank, additional separation may be required within that organisation.

For example, an organisation may have multiple departments, branches, or operational sites. Not every auditor should necessarily access every historical finding.

I used tags to represent information that varies within an organisation, such as site and control.

During retention, each finding receives relevant tags:

client.retain(
bank_id=bank_for(org_id),
content=sentence,
context=f"audit finding, cycle {f.cycle}",
timestamp=f.reported_at.isoformat(),
document_id=f"finding-{f.id}",
tags=[f"site:{f.site_slug}", f"control:{f.control}"],
)

During retrieval, the application applies the appropriate scope:

client.recall(
bank_id=bank_for(org_id),
query="open access findings",
tags=["site:pune"],
tags_match="any",
)

The matching mode matters because different situations require different levels of filtering.

For example, organisation-wide policy information may need to remain available alongside site-specific findings. However, a restricted investigation may require stricter filtering so that unrelated or untagged memories are excluded.

I therefore treated tag filtering as an important access-control consideration rather than simply a search convenience.

I also planned tests to verify that a site-scoped recall does not return findings tagged for another site.

The key lesson was that memory retrieval must respect the same information boundaries as the application itself.

1. Decision Three: Retaining Meaningful Sentences Instead of Raw Records

One of the most important decisions was how to represent audit findings before sending them to Hindsight.

Audit applications commonly store information in structured database records or JSON objects. Initially, I considered retaining those raw records directly.

However, Hindsight processes retained content to extract facts and relationships. This means the way information is written can influence what the memory system understands.

Instead of sending raw database rows, I decided to convert each finding into a meaningful sentence.

For example:

def render(f: Finding) -> str:
return (
f"Audit finding {f.id} ({f.severity}) at {f.site} under control "
f"{f.control}, reported by {f.auditor}: {f.description} "
f"Responsible team: {f.owner_team}."
)

This representation gives the memory system more understandable context.

Rather than receiving isolated field names such as owner_team or control, the retained information describes the relationships between the finding, location, control, auditor, and responsible team.

I found this approach more suitable for representing audit information in natural language.

The context parameter also plays an important role. It provides a short description of the source and type of information being retained.

For example:

context="audit finding, cycle 2026-Q3"

This helps identify the kind of record and its audit cycle when the information is retrieved later.

It also provides useful provenance for the application when presenting historical results to an auditor.

1. Decision Four: Giving Every Retained Document a Stable Identity

Audit records can change after their initial creation.

A finding might receive an updated description, a revised severity, or a different corrective-action status. If the memory system retains every updated version without recognising the original source, historical information can become difficult to manage.

To address this, I assigned every retained item a stable document_id derived from its original database record.

For example:

document_id=f"finding-{f.id}"

This creates a consistent connection between the source record in VERIQ and the information retained in Hindsight.

When a record needs to be retained again, the same document identifier can be used to update its corresponding retained information according to Hindsight's document replacement behaviour.

For larger historical imports, I also considered batch retention:

client.retain_batch(
bank_id=bank_for(org_id),
items=[
{
"content": render(f),
"context": f"audit finding, cycle {f.cycle}",
"document_id": f"finding-{f.id}",
}
for f in findings
],
retain_async=True,
)

Asynchronous retention is useful when processing a large collection of historical records because the application does not need to wait for every item to finish processing before continuing.

For the current audit cycle, synchronous retention is preferable when the application needs to make a newly recorded finding available for retrieval immediately after processing.

This distinction helped me think about historical data ingestion and live audit activity as two different workflows.

1. What This Modelling Approach Makes Possible

The structure of the retained information determines how useful the recalled results can be.

For example, VERIQ can retrieve information and organise the returned results into application-friendly fields:

for r in client.recall(
bank_id=bank_for(org_id),
query=q
).results:
row = {
"text": r.text,
"type": r.type,
"cycle": r.context,
"when": r.occurred_start,
"source": r.document_id,
"tags": r.tags,
"entities": r.entities,
}

Each field has a purpose:

Text: Provides the recalled information.

Type: Helps identify the category of memory.

Cycle: Connects the information to its audit period.

When: Provides the associated time information.

Source: Links the memory back to its original record.

Tags: Help verify the scope of the retrieved information.

Entities: Show the people, departments, controls, or processes identified by the memory system.

This structure makes it easier to present historical findings in a way that auditors can understand and verify.

A Practical Example

Imagine an auditor working at an organisation's Pune site asks:

"What is still open regarding access control?"

VERIQ can retrieve relevant findings using the site's assigned tags.

The results can include the outstanding findings, their audit cycles, and links to their original records.

An auditor working at another site can make the same request but receive information within their permitted scope.

This demonstrates why memory modelling, organisational separation, and retrieval filtering are important parts of an audit assistant.

1. Managing Recall Based on the Question

Retaining historical information is only one part of building a useful memory system. Retrieving the right information also matters.

Not every question requires the same retrieval depth.

A quick lookup may only need a small amount of relevant information. However, identifying recurring issues across several audit cycles may require a deeper search.

I organised recall settings into different profiles:

RECALL_PROFILES = {
"lookup": {
"budget": "low",
"max_tokens": 500
},
"context": {
"budget": "mid",
"max_tokens": 3000
},
"pattern": {
"budget": "high",
"max_tokens": 4096
},
}

The intention behind these profiles is:

Lookup: For simple questions about a particular finding or control.

Context: For questions that require additional historical information.

Pattern: For identifying recurring issues and relationships across audit periods.

The retrieval depth should depend on the question rather than using the same settings for every request.

I also considered narrowing the retrieved memory types when the application already knows what kind of information it needs.

For example, identifying recurring observations may require a different retrieval approach from retrieving records about previously completed corrective actions.

This helps keep retrieval focused on the information needed by the auditor.

1. What I Would Change Next Time

One thing I would do differently is define the memory representation and document identification scheme before writing the ingestion workflow.

I initially approached these decisions in the opposite order. As the structure evolved, updating the way historical information was represented required additional work.

Having a consistent rendering function and stable identifiers from the beginning would have made the process easier.

The experience also reinforced the importance of testing memory retrieval independently. A successful retention call does not automatically guarantee that the correct information will be recalled, that the correct entities will be identified, or that access boundaries will be respected.

Those behaviours need to be tested as part of the application.

1. Lessons Learned

Building VERIQ helped me understand that persistent memory is not simply about storing more information. It is about structuring information so that it remains useful when the application needs it later.

The most important lessons were:

Use separate memory banks for organisational boundaries. Structural separation is stronger than depending entirely on filters to prevent cross-organisation retrieval.

Retain meaningful descriptions instead of raw database rows. Clear natural-language representations make relationships easier for a memory system to interpret.

Use stable document identifiers. They help connect retained information to its original source and manage updates consistently.

Treat retrieval scope as a security requirement. Tag filtering should be tested to ensure that users cannot retrieve information outside their permitted scope.

Preserve provenance from the beginning. Context, timestamps, tags, and source identifiers make historical findings easier to trace and verify.

Conclusion

The most valuable part of building a memory-enabled audit assistant is not simply connecting an LLM to a memory system. It is deciding what information should be remembered, how it should be represented, and under what conditions it should be retrieved.

With Hindsight, VERIQ can use historical audit information as more than a collection of old records. It can become useful context for understanding recurring compliance gaps, revisiting previous corrective actions, and learning from earlier audit cycles.

The modelling decisions behind retention and recall ultimately determine how reliable and useful that experience can be.

For further technical understanding, I referred to the Hindsight documentation and the Hindsight GitHub repository. The Vectorize overview of agent memory also provides useful context on why persistent memory matters when building AI-powered applications.