How Hindsight Remembered Why We Skipped Table Two

In an early browser test, our restaurant seating agent ruled out Table Three because a manager had put it on hold. The manager really had put it on hold. Staff had also cleared it by the time we asked for a recommendation.

We'd built the agent to remember why a manager made a different call on a previous shift. Somewhere along the way, we'd also taught it to carry yesterday's floor state into today's decision. The bug was easy to see once we put the old note next to the current floor. Both records were accurate. They described different moments. Live integration notes

Why the manager skipped Table Two

Restaurant Floor Copilot is a host desk for walk-in service. It tracks the waiting queue, table readiness, and staff tasks. A pretrained YOLO detector and ByteTrack supply observations from configured table regions. Staff confirm when guests have left and when a table is clean. The allocator uses the current operational state to suggest a seat. We built the shift-memory workflow on top of an existing open-source occupancy system. Project guide · Upstream attribution

Here's the case we staged. A party of four is waiting, and two four-seat tables are ready. The allocator picks T2. The manager prefers T3 because T2 sits beside the pickup area and staff had reported repeated interruptions there during a delivery rush. Later, when service is quiet, T2 is a reasonable choice again.

I like this example because the allocator's first answer is entirely reasonable. It knows the party size and which tables are ready. The manager knows something else about the room. We wanted that explanation to survive a shift change so the next manager could decide whether it still applied.

We used Hindsight's open-source memory system to retrieve past experience. SQLite already held the queue, table status, and open tasks, so we kept those records as the source of current state. The supervisor gets eligible actions from the floor and supporting evidence from memory. Staff review the proposal before anything changes.

The supervisor receives two inputs. The floor supplies what staff can do right now, and Hindsight supplies experience that might help them choose.

Keep the correction and the outcome separate

When a manager records a correction, we save the baseline table, the reported alternative, the reason, and the conditions staff confirmed for that shift. The original note goes into a SQLite ledger. A background worker delivers it to Hindsight using the local record ID as the document ID. If delivery is retried, it targets the same document.

These are the identifying fields from the SDK call:

document_id=record["id"],
timestamp=record["recorded_at"],
metadata={
    "experience_id": record["id"],
    "layout_id": record["layout_id"],
},
tags=[f"layout:{record['layout_id']}"],
retain_async=False,

The worker makes the call outside the lock that protects floor operations. The host can keep working while the source shows as pending. Once Hindsight acknowledges it, the timeline shows it as retained. If delivery fails three times, staff can see the failure and retry it. Hindsight adapter · Delivery worker

The outcome gets its own record. A manager might choose T3 at six o'clock and hear later whether the move helped. Those events answer different questions, and sometimes the answer to the second one is “unknown.” In our evaluation, the outcome was a synthetic staff report. Keeping it separate let us test how the application handles feedback without treating a seating approval as evidence of success.

An old note can be relevant and still lead you astray

The supervisor sends Hindsight the current question, floor context, and candidate actions. Each restaurant uses its own memory bank, and a strict tag limits recall to the current floor layout. The layout identity includes the configured video and the table IDs, capacities, and regions. A note about T2 loses much of its value if T2 now refers to a different part of the room.

client.recall(
    bank_id=self.bank_id,
    query=query,
    budget="mid",
    max_tokens=2400,
    types=["world", "experience"],
    tags=[f"layout:{layout_id}"],
    tags_match="all_strict",
)

Before the model sees a recalled fact, the application matches it to an active, retained source in the local ledger. A correction can influence seating only if it names the actual baseline and proposed alternative and matches the delivery conditions staff confirmed for the current shift. The preference expires after seven days. A conflicting correction or a linked report that the change didn't help can also rule it out. Hindsight documentation · Source and preference checks

We turned the old table hold into a test case. We recorded the correction while T3 was marked occupied, then made T3 ready before asking for advice. That gave the model a useful reason to prefer T3 and a historical snapshot that could trip it up. We adjusted extraction to focus on reasons and outcomes, and made the current candidate list the authority on eligibility. The test now checks whether the supervisor can use the pickup lesson while working from today's floor.

The model returns IDs for the actions and evidence it used. On approval, the backend checks the proposal's revision, rebuilds the candidates, and rechecks the cited preference. The same pattern covers task recovery. If T3 needs cleaning, the supervisor can suggest prioritizing or assigning the existing task. Staff still have to confirm the table is ready. Proposal and approval code

What happened when we removed memory

In the saved Cloud evaluation, we ran the same question through Groq's openai/gpt-oss-120b with the same floor, shift context, and candidates. One pass received the recalled evidence and the other received an empty memory list. The comparison itself couldn't seat anyone.

The Supervisor shows the proposed seat, the sources behind it, and the result with memory removed.

The retirement check mattered more than I expected. In an earlier run, the model read a note saying the pickup arrangement had changed and still recommended the old alternative. An outcome report or a later approval can also repeat an earlier lesson. The app now follows those source relationships when staff retire a correction. The timeline keeps the history, while the proposal check requires an active correction before it accepts an alternative. Recorded Cloud evaluation · Vectorize's guide to agent memory

A separate Local Docker check retrieved three cited facts for a handover after an application restart. Its open-work list matched SQLite. That gave us a way to check that the next shift could recover the reason behind a decision and still see which tasks were open now. The seating trials used synthetic inputs and measured recommendation changes. Guest interruptions and wait times would need a restaurant pilot. Local handover verification

The screen I'd show another developer is the one above. T3 is the interesting choice, but the part I care about is the source underneath it. A manager can open the note, see why the earlier shift preferred that table, and decide the advice has run its course when pickup moves. That's what I wanted from the memory in the first place.

Restaurant Floor Copilot on GitHub · Code.in

AI disclosure: This article was drafted and edited with AI using the project source and saved verification records.