System design for physical AI: a patch cycle that learns from every confirmed call

By Muhammad Umar. This is the engineering summary of an open reference architecture. The paper, its object model as JSON and the model register are free to reuse under CC BY 4.0: https://codeatoms.ai/ot-vulnerability-utility-us/. DOI to follow on Zenodo. The operator is described by class, never by name.

An electric utility in the United States runs substations, control centers and generating plants whose control systems fall under the critical infrastructure protection standards. Every time a vendor publishes an advisory, the compliance team has to answer one sentence: which of our devices does this touch, what does the verified baseline say, do we apply the patch or mitigate, and where is the evidence that we decided inside the window. The standards give thirty five days to review and thirty five more to act. The facts sit in seven systems of record, and the reasoning behind the last call on the same relay sits in somebody's mailbox. A handful of engineers know which vendors matter and which devices can take an outage, and two of them retire next year.

1. Inbound only, two writes

The useful signals, advisories, vulnerability records and patch artifacts, arrive from outside, while facts about the assets must never travel out. The design reads inbound feeds through verified paths and sends nothing back. Where the utility's policy demands it, OT side data leaves the electronic security perimeter through a hardware enforced one way path with no return route and no credentials inside the perimeter; a DMZ conduit is the lighter alternative. Every system of record is read only except two paths: the disposition agent drafts apply or mitigate change requests into the change management system, and the evidence agent writes confirmed evidence items into the evidence repository. Both land where the auditor already looks, which keeps the ontology a projection rather than a second system of record.

2. Eight sources, two adapter families

Seven systems of record feed the design, each read through its own API where one exists and a scheduled export where it does not: the asset inventory and configuration baseline system, the vendors' patch and advisory feeds, the national vulnerability advisories, the change management system, the work order system, the evidence repository for audits, and the network monitoring tool's device list. The eighth source is the patch artifact channel, the patches themselves, verified for source identity and integrity before any change can reference them. Two adapter families own the contracts: the integration adapter for the API and export sources, and the telemetry adapter for the OT side device list where it crosses the one way path. Every record carries its source system and its freshness, and business critical drafts refuse data older than an agreed age rather than guessing from a stale baseline.

Apache Kafka 4.4 carries everything between the adapters and the ontology, keyed on advisory identifiers and record identifiers. Delivery is at least once, made safe by idempotent upserts keyed on record identity and content hash, so a replayed export can refresh a record but never create a second applicability decision or a duplicate evidence item. There is no historian, no camera and no telemetry feed in scope: the design watches advisories, baselines and decisions.

3. Sixteen objects, three born in the design

The object model holds sixteen objects with thirteen typed links. Five anchor in the asset inventory and configuration baseline system: substation, protective relay, engineering workstation, firmware and software version, and configuration baseline. Eight more anchor in the vendor feeds, the national advisories, change management, the work order system and the evidence repository. Three anchor nowhere because the design creates them: applicability decision, disposition and cycle agent. One walk across the links answers the morning question: from the advisory through firmware and software version to the protective relays running it, onward to the substation and the outage window, and back through prior applicability decisions to their evidence items. An applicability decision moves from Drafted to Confirmed or Rejected only when a named engineer acts.

4. One node, sized by arithmetic

There is no edge tier. Both inference tiers, reasoning and embedding, sit on one node of eight GPUs of the 80 GB class in the utility's own data center. The register files gpt-oss-120b at 117 billion parameters (the paper rounds it to about 120 billion). At INT4, 0.5 bytes per parameter, the weights occupy 58.4 GB; multiplying by 1.2 to cover KV cache and activations gives 70.1 GB the GPUs must hold. The node offers 640 GB, so the weights fit with generous headroom for concurrent agent sessions. The checkpoint itself is 60.8 GiB, 65.2 GB of root safetensors; expanded fully to 16 bit it would need about 234 GB, which is what rules out a smaller host. vLLM 0.32 serves it from the node it sits on, and Harbor v2.15.2 holds signed artifacts in an air gapped registry. Nomic Embed Text v2 MoE, 475 million parameters at FP16, about 0.95 GB resident, embeds advisories, baselines and confirmed decisions on the same node. Both models carry Apache-2.0, a permissive license with no usage based trigger and no field of use restriction, and both were chosen because the requirement names open weights of United States origin on owned hardware, a GPU class with domestic supply and no export control complication for a United States end user.

5. Three agents that draft, memory that compounds

The applicability agent drafts which devices an advisory touches, for the compliance engineer to confirm. The disposition agent drafts apply or mitigate with its outage window and change request, for the asset owner to approve. The evidence agent assembles the audit package, for the compliance manager to confirm. No agent applies a patch, changes a baseline or closes a change. Nothing is trained on the fly: improvement is retrieval of confirmed decisions, with supersession and tombstone checks on every retrieval so a wrong precedent cannot confirm itself. Weights change only through a separately governed offline fine tune lane, promoted against a frozen question set built from the first cycles' confirmed calls, and never inside an active cycle week. Because engineers under a thirty five day clock can rubber stamp drafts, the acceptance rate is instrumented both ways and unflagged drafts are sampled independently each cycle.

Appendix A prices the node at about 477,000 US dollars to own over three years (437,000 to 496,000). Renting the same eight H100 80 GB cards on the deepest three year commitment in AWS US East is about 544,000, so ownership costs about seven eighths of it, and about 1.45 million on demand, against which ownership costs about a third. The region's other 80 GB class instance, eight A100 cards, rents for about 268,000 on the same plan, against which owning the H100 node costs about 1.8 times as much. Renting is a price reference only: every rented option would carry the asset inventory, the baselines and the vulnerability record outside the utility's network.

The rollout is two gated phases and 17 scope items (16, then 1): the protective relay fleet and its engineering workstations at one group of substations for two patch cycles, then an extension gate to the other asset classes and the generating plants. Phase 1 exits when the disposition agent is drafting apply or mitigate recommendations that engineers confirm on the record, measured against the utility's own baseline of engineer hours per cycle, windows kept, evidence completeness and auditor questions answered from the record.

Made with CodeNinja Praxis, the CodeNinja Atoms platform for designing physical AI systems. The object model imports into Hyper Ontology. Paper, ontology and register: https://codeatoms.ai/ot-vulnerability-utility-us/

One of the CodeNinja Atoms worked examples of a self-improving AI system, published with its object model as hyper-ontology/1 JSON under CC BY 4.0 in the cumulative dataset of vertical-driven architectures.