System design for physical AI: a site memory that answers with the link down
By Muhammad Umar. This is the engineering summary of an open industrial AI reference architecture from CodeNinja Atoms for a remote site on a satellite link, built so the industrial AI its supervisors rely on runs on har
By Muhammad Umar. This is the engineering summary of an open industrial AI reference architecture from CodeNinja Atoms for a remote site on a satellite link, built so the industrial AI its supervisors rely on runs on hardware the operator owns, inside the fence. The paper, its object model as JSON and the model register are free to reuse under CC BY 4.0: https://codeatoms.ai/remote-mine-satellite-pakistan/. DOI 10.5281/zenodo.24048581. The operator is described by class, never by name.
A mining operator at a remote open pit mine in Pakistan runs its site on its own power, reached by road and air, with one satellite link that drops in bad weather. Every shift, the supervisors need one question answered: what work is open on each pad, which equipment is overdue for service or down with a fault, and which convoy carries the parts that work needs. The data already exists, but in seven systems that each hold one true slice and none of the join, while the corporate systems sit in data centers abroad and reach the site only over the link. So when the link degrades, permits are issued on paper, faults are logged after the shift, a convoy's position is known when it arrives, and the site's state lives in the heads of people who rotate out every few weeks.
1. Read everything, write nothing below
The design is one pattern end to end: systems of record below, one governed object model in the middle as a read projection and never a second master, services and surfaces above, all in the site server room on the mine's own power. The records below keep their data, their procedures and their owners, so nothing writes into any of them. The single write path out of a draft is a named person's confirmation, recorded against the owning system's record and reconciled daily during the pilot, so no second version of permit state grows beside the permit to work system. The corporate estate abroad is a sync target, never a dependency: exactly two data class filtered flows cross the fence, mirror feeds in and chosen operational aggregates out, and the camp and workforce mirror is the one the outbound sync never touches. Store and forward with ordered, idempotent replay was picked over live bidirectional replication, because a replication stream that fails midstream on a degrading link leaves mirrors in unknown order. Artifacts and models move through Harbor, the air gapped registry, and the site keeps serving its current verified versions until a new one arrives.
2. Two adapter families, eight sources
Eight sources feed the model: seven read only systems of record on site and the corporate estate abroad. The seven are project controls, construction management and permit to work, equipment telematics and maintenance, camp and workforce management, health, safety and environment, supply and logistics tracking, and the site historian for power and water. Two adapter families are the sole path in: an integration adapter carrying record extracts from the transactional systems, and a telemetry adapter carrying the historian's utility tag streams and the telematics fault code and service hour feeds. No cameras and no new positioning hardware are added, because the requirement asks for none and the telematics and logistics systems already carry positions.
The adapter tier guarantees four things. No adapter writes to a system of record. Each adapter owns its contract: the extract or stream schema, the cadence, the mapping onto objects and the provenance stamp, so a change in a source is absorbed in one place. Replaying an event twice changes nothing in the mirror, which is what makes outage recovery safe. Every mirrored record carries a last seen time, so the command center's staleness banners can tell a supervisor that a convoy position is fresh and a work package figure is hours old.
Apache Kafka 4.3 in KRaft mode with three dedicated controllers carries every arrival to the object model, ordered per source so a permit confirmed after it was revoked never replays the other way round, delivered at least once and made safe by idempotent application, and buffered through an outage for replay in stored order when the link returns. Refresh priority is set by data class, measured against the link's real sustained rate before anything else is committed. An OCP Time Card class GNSS grandmaster with OCXO holdover, run with linuxptp and chrony, holds one clock for mirrors, telemetry and staleness stamps while the link is down. Keycloak holds identity on site so logins survive an outage, and Prometheus, Grafana and Loki watch the running system.
3. Fifteen objects, one walk
The object model holds fifteen objects with thirteen typed links on Hyper Ontology v0.9: construction area, pad, work package, permit, crew, worker, equipment unit, fault, work order, part, convoy, consignment, camp occupancy, incident and utility tag, each anchored in exactly one system of record that remains its owner. The pad sits in the focal position for the phase one gate. A permit is issued on a pad and worked by a crew; a work order belongs to a work package, traces to the fault that raised it and requires a part; a consignment is carried by a convoy, fulfills a work order and serves a work package; an incident links to the permit it happened under. So the supervisor's question, which equipment is overdue, which convoy carries the parts for the work order it needs and whether the pad's permit is still valid, is one traversal across those links, each step carrying its own last seen stamp. A document store holds each of those facts somewhere. It holds no path between them, and every surface would reimplement the joins in application code.
4. One room, sized by arithmetic
Both inference tiers sit in the site server room, and below that room the design commits no inference hardware at all: the rugged tablets in the field are thin operators to the command center and run nothing. The stack carries no edge model serving row for that reason, and no interactive answer traverses the link, so the latency budget is a shift conversation, not a satellite round trip.
The frontier tier is one node of eight 141 GB HBM GPUs serving GLM 5.3 under vLLM, orchestrated by Kubernetes. GLM 5.3's 753 billion filed parameters at FP8, one byte each, are 753 GB of weights; the node holds 1,128 GB, which leaves 375 GB beside them, and the 1.2 planning factor reserves about 151 GB of that for KV cache and activations (753 times 1.2 is 904 GB, inside the node). Two nodes of the 80 GB class were rejected: their 589 GB usable cannot hold the weights at all, and they would drag a multi node RDMA fabric onto a remote site on its own power. BGE-M3 at FP16, about 1.1 GB as published, sits beside it on the same node and keeps part numbers, fault codes and permit clauses retrievable through dense, sparse and multi vector retrieval in one pass. Export controls on high bandwidth memory accelerator classes govern what hardware can enter a D:2, D:3 and D:4 destination, so a licence determination naming the end user is the first phase one checkpoint before any card commitment, with an in country colocation fallback held for the frontier tier if the path closes. When site power fails, the enclosure and power class orders a clean shutdown of the node and the mirrors, so on restoration the mirrors resynchronize rather than corrupt.
5. Agents draft, supervisors confirm
Eight services (site ontology, sovereign platform, sync and resilience, governance and approvals, drafting agents, memory and reasoning, command center, and the phase one pilot) carry the work onto four command center views: pads by status, equipment by risk, convoys by position, and link state with staleness banners and the sync priority. Three agents draft on GLM 5.3: permits, equipment and convoys. The area supervisor confirms the permit agent's shift start draft of open permits, conflicts and expired isolations per pad, the maintenance planner approves the equipment agent's work order and parts check, and the logistics lead approves the convoy agent's re sequencing. No agent issues a permit, closes an isolation or dispatches a convoy. Every confirmed cause and accepted plan is kept on site as memory the next shift reads, the memory that survives crew rotation; nothing is trained into weights, and both models carry terms that let the operator hold them: the bespoke GLM-5.3 licence, which permits commercial use, fine tuning and redistribution, and MIT for BGE-M3.
Appendix A prices the node at about 572,000 US dollars to own over three years (548,000 to 597,000: about 408,000 for the eight card node, about 122,000 for support at 8 to 12 percent of hardware value a year, and about 42,000 for power, priced at the published industrial tariff as a reference, since the mine generates its own). Renting the same eight cards on the deepest three year commitment in AWS Asia Pacific (Mumbai) is about 751,000, so ownership costs about three quarters of it; on demand it is about 2.00 million, against which ownership costs a little over a quarter. Renting is a price reference only: AWS lists no region in Pakistan, the nearest region sits across a border, and no rented node answers with the link down.
The rollout is one phase of 17 scope items across six workstreams, closed by a gate, never a date. Phase one covers one construction area and its heavy fleet for one quarter. Its gate is a simulated link drop day with the satellite egress severed, on which the pilot area must resolve a dozer fault, carried from a raised fault code to a drafted work order whose parts the on site inventory can name; a permit clash between two crews on adjacent pads, held as a conflict rather than issued; and a late convoy, read from its last known position against plan, with the consignments it carries tied to the work packages waiting on them. Until that day is answered from local memory, nothing the build drafts is load bearing. The link's sustained rate and outage statistics are measured first, and four baselines come from the site's own records before any claim: permit conflicts found at the pad, equipment hours lost to parts not on site, convoy days late, and the time a new supervisor needs to answer what is open on a pad.
Made with CodeNinja Praxis, the CodeNinja Atoms platform for designing physical AI systems. The object model imports into Hyper Ontology. Paper, ontology and register: https://codeatoms.ai/remote-mine-satellite-pakistan/
One of the CodeNinja Atoms worked examples of physical AI system design: systems of record read and never written, a governed ontology joined by typed links, open weights on hardware the operator owns in its own country, and agents that draft while named people approve, the shape every physical AI design in the series is published in, with its object model as hyper-ontology/1 JSON under CC BY 4.0.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.