Version Your AI Logs Like an API: Schema Evolution for Prompt and Review Events

Six months after launch, someone asks a reasonable question about your LLM feature: how often did reviewers override the model in March? The data is in your logs. But in April a teammate renamed reviewer to reviewed_by, and in May the model field changed from a string to an object. Every query now needs three code paths, and nobody is sure which one applies to which week.

Logs for AI features tend to outlive the code that wrote them. They feed evaluations, incident reviews and audits long after the prompt, the model and the service have changed. That makes the shape of a log event a contract with your future self, and contracts need versions.

Treat each event like a message on a bus

Event-driven systems solved this problem years ago, and their conventions transfer well. The CloudEvents v1.0.2 specification defines a small envelope that is a good model even if you never send an event over a network:

• id and source identify the event. The spec says "Producers MUST ensure that source + id is unique for each distinct event", and lets consumers treat a repeat of both as a duplicate.
• type describes what happened. Many teams also put a version in the type name, such as review.completed.v2.
• dataschema points to the schema the payload follows. The spec adds: "Incompatible changes to the schema SHOULD be reflected by a different URI."
• time records when the occurrence happened, in RFC 3339 format.

Applied to an AI feature, that means every log line says what kind of event it is, which schema version its payload follows, and which unique event it is. None of that depends on a particular logging tool.

  writer (v3 code) --> log store <-- reader / eval job
        |                 |              |
   stamps type +     keeps v1, v2,   upcasts old events
   schema version    v3 side by side to the latest shape
        |                 |              |
        +---- schema registry or a schemas/ folder ----+

Decide what "compatible" means before you change anything

The Confluent Schema Registry documentation on schema evolution (Confluent Platform 8.4 docs, read 2026-11-06) gives useful names for the choices:

• Backward compatible: "consumers using the new schema can read data produced with the last schema". Upgrade readers first.
• Forward compatible: data written with a new schema can be read by consumers using the last schema. Upgrade writers first.
• Full: both at once.

Their default is BACKWARD, which checks a new schema only against the latest one. The _TRANSITIVE variants check against every earlier version. For logs that you keep for months and replay into evaluations, the transitive question is the one that matters: can today's reader still read what was written on day one?

The documentation also shows why defaults matter. In its Avro example, a new field is backward compatible only because it has a default value; without the default, the new schema could not read old data.

A small, testable pattern

You don't need a registry to start. Stamp a version on every event, keep old readers working with "upcasters", and test them against stored samples.

import json, uuid
from datetime import datetime, timezone

CURRENT = 3

def make_event(event_type: str, payload: dict) -> dict:
    return {
        "id": str(uuid.uuid4()),
        "source": "app/review-service",
        "type": event_type,                  # e.g. "review.completed"
        "schema_version": CURRENT,
        "time": datetime.now(timezone.utc).isoformat(),
        "data": payload,
    }

def _v1_to_v2(d):                            # renamed field
    d = dict(d)
    d["reviewed_by"] = d.pop("reviewer")
    return d

def _v2_to_v3(d):                            # string -> object, with default
    d = dict(d)
    d["model"] = {"id": d["model"], "provider": d.get("provider", "unknown")}
    d.pop("provider", None)
    return d

UPCASTERS = {1: _v1_to_v2, 2: _v2_to_v3}

def read_event(raw: str) -> dict:
    event = json.loads(raw)
    version, data = event["schema_version"], event["data"]
    if version > CURRENT:
        raise ValueError(f"event written by newer code: v{version}")
    while version < CURRENT:
        data = UPCASTERS[version](data)
        version += 1
    return {**event, "schema_version": CURRENT, "data": data}

Two details are deliberate. The reader never rewrites the stored event, so the original stays available for anyone who needs to see exactly what was logged. And it refuses events from a newer version instead of guessing, which surfaces a deployment-order mistake immediately.

The tests are the real safeguard. Keep one frozen sample per version in the repository and run every sample through the current reader:

SAMPLES = {
    1: {"reviewer": "u_17", "model": "model-a", "action": "override"},
    2: {"reviewed_by": "u_17", "model": "model-a", "provider": "acme",
        "action": "override"},
}

def test_every_old_version_reads_as_current():
    for version, data in SAMPLES.items():
        raw = json.dumps({"id": "e1", "source": "t", "type": "review.completed",
                          "schema_version": version, "time": "2026-01-01T00:00:00Z",
                          "data": data})
        event = read_event(raw)
        assert event["schema_version"] == CURRENT
        assert event["data"]["reviewed_by"] == "u_17"
        assert event["data"]["model"]["id"] == "model-a"

When someone bumps CURRENT to 4, this test fails until they write the 3-to-4 upcaster. Make adding a frozen version 3 sample part of the same change.

Rules worth writing down

• Never reuse a field name with a new meaning. Add a new field and deprecate the old one.
• Give new fields defaults so old events can be upcast without inventing data. Make "unknown" explicit rather than silently empty.
• Version the payload, not just the code. A git SHA tells you which code wrote an event, not which shape it has.
• Keep evaluation jobs on the reader path. If an eval script parses raw JSON directly, it will be the first thing to break.

Where this fits

Records that stay readable as your system changes are part of the problem we work on at Obligra. Obligra Verify™ is a system of record for AI-assisted decisions, in public preview, designed to retain the evidence your application selects. The versioning habits above are worth adopting wherever your records live.

This article was generated by an AI agent (Claude) for the Obligra Verify team and checked against the linked sources.