Dev.to AI ๐Ÿค– Ai ๐Ÿ‘ 0 ๐Ÿ“– 8 min read

System design for physical AI: a tailings embankment where every section has one story

By Muhammad Umar. This is the engineering summary of an open industrial AI reference architecture from CodeNinja Atoms for monitoring a high consequence tailings storage facility, one of a series of worked examples of in

By Muhammad Umar. This is the engineering summary of an open industrial AI reference architecture from CodeNinja Atoms for monitoring a high consequence tailings storage facility, one of a series of worked examples of industrial AI that get better from what named people confirm, on hardware the operator owns, without training a model on the fly. The paper, its object model as JSON and the model register are free to reuse under CC BY 4.0: https://codeatoms.ai/tailings-monitoring-us/. DOI 10.5281/zenodo.24048569. The operator is described by class, never by name.

A mining company in the United States runs a tailings storage facility classified at a high consequence level: an embankment raised in stages over decades, holding tailings behind it with a beach and a pond, inspected daily and reviewed by an engineer of record and an independent review board. It needs one question answered continuously: what is happening in one section of the embankment, read as a whole rather than instrument by instrument. The answer lives in pieces no system joins. A piezometer that rises slowly over a month is visible in the historian, but it only means something joined to the deposition that moved the beach toward it, the rain that fell, the inclinometer two hundred feet away and the radar survey from last quarter. Thresholds fire on single instruments, so the pattern that matters hides among the alarms.

1. Three layers, and the records stay put

The design is one pattern in a fixed order: systems of record below, never replaced, never modified and never loaded directly; one governed object model in the middle as a projection over them, never a copy; applications and agents above, reading only the model, with the surfaces where engineers confirm or approve each draft. The ask is a join, not a rebuild, so the design chose a governed ontology over the eight systems of record instead of a new central historian or a data lake migration.

Instrument, survey and inspection data never leave the owner's hardware, and no third party model service is called. The single instrument thresholds in the trigger action response plan keep firing in the historian exactly as they do today, and the design sits beside that alarm path with pattern drafts rather than in front of it with a replacement. The design chose against cameras and video analytics (no camera appears anywhere in the ask), against fine tuning on trigger history, and against a frontier cluster or a hosted model service; new field instruments, hardware supply, historic data migration and round the clock support stay with the owner, with evidence support only.

2. Two adapter families, eight sources

Eight read only sources of operator provenance feed the design. The instrumentation data logger network and its historian supplies minute level readings for every instrument. The satellite radar deformation reports supply quarterly surface deformation surveys. The daily inspection app holds the operators' recorded observations. The trigger action response plan supplies trigger levels and predefined actions. The engineer of record's reports and drawings supply the design basis and the section definitions the ontology anchors on. The deposition and raise plans supply the events that explain instrument behavior, the water balance model supplies the forecast that pond level divergences are read against, and the regulator's correspondence supplies submissions, notices and due dates.

Two adapter families are the single door in, and no service, agent or model reads a source system directly, which makes the no modification rule enforceable in code rather than by convention. The telemetry adapter follows the site's own standard, reading the historian's live API or OPC UA feed rather than imposing a protocol. The integration adapter takes the other seven document, plan, model and correspondence sources on a schedule suited to each; the radar surveys arrive as report files and the water balance model is read where it runs. Every source enters with its provenance class and read direction recorded, and readings carry quality flags so a degraded sensor is visible as such.

Apache Kafka 4.3.x in KRaft mode is the backbone, keyed by instrument tag so every instrument's readings keep their order within a partition even when the loggers deliver in bursts. Delivery is at least once with idempotent writes keyed on tag and timestamp, so a replay cannot double count a reading; broker retention buffers a paused consumer, and replication inside the site server room keeps the backbone alive through a node loss. Linuxptp with an OCP Time Card grandmaster, disciplined by chrony, keeps loggers, edge units and services on one clock. InfluxDB 3 Core holds the telemetry for trend queries, Harbor in offline install is the air gapped registry, identity is served on site, and Prometheus 3.13 LTS with Grafana observes the tier.

3. Fifteen objects, one walk

The object model holds fifteen objects in seven kinds with twelve typed links on Hyper Ontology v0.9. Two assets: embankment section and instrument. Three events: raise, trigger and deposition event. Two measures: reading and pond level. Four records: trigger level, inspection observation, pattern draft and response draft. Two documents: radar survey and regulator submission. One actor, the engineer of record, and one system, the water balance model. The pattern draft is the pivot, with a status vocabulary that runs drafted, confirmed, rejected.

An embankment section is raised through a raise; a reading is taken by an instrument; a trigger fires on an instrument, is explained by a pattern draft and is confirmed by the engineer of record; an inspection observation is recorded on a section; a radar survey covers a section; a pattern draft is drafted for a trigger and cites the trigger level and the deposition event; the water balance model forecasts pond level. So a slowly rising piezometer becomes one query: readings to their instrument by tag, to the instrument's section, to its raise stage, to the depositions that moved the beach, to the inspections on that section, to the radar survey that covered it and to matched past episodes in memory, returned as one argument with citations. A document store can return whatever mentions the tag, but it cannot walk the edges.

4. Three tiers on the owner's hardware

Inference runs on three tiers, all on hardware the owner holds in the United States, so no export control question arises. At the logger network, on device edge units screen readings with Granite Tiny Time Mixers (TTM r2), about 0.003 GB at 32 bit, on CPU class hardware under ONNX Runtime with K3s orchestrating containers. In the site server room beside the GPU node, Chronos-2 builds trend baselines with rainfall and pond level as covariates, and Nomic Embed Text v2 MoE, 475 million parameters at about 0.95 GB in fp16, embeds reports and inspection text for retrieval with citations. The drafting tier is gpt-oss-120b served by vLLM on one node of eight 80 GB GPUs, 640 GB in total. All four models are Apache-2.0 and US origin.

The sizing rule files 117B parameters at INT4, 0.5 bytes per parameter, which is 58.4 GB of weights; the 1.2 planning factor reserves about 12 GB for KV cache and activations, so the GPUs must hold 70.1 GB, and the node holds it with 582 GB to spare. The model card files the same model at 116.8B total and 5.1B active parameters in MXFP4, about 61 GB, which fits a single 80 GB GPU with headroom. The node draws about 10 kW, so the server room's power and cooling are surveyed before any order, with an in country colocation room as the fallback.

The latency budget follows the cadence of the work. Edge screening must finish inside the minute level reading interval, site tier baselines must be ready when a trigger fires, and the drafting agents work at the rhythm of an engineering shift, while a pattern draft is still useful and its approver is still on site. When the link between the logger network and the server room fails, the edge units keep screening and the backbone buffers and replays, so no reading is lost, only delayed. A hardware data diode is the drop in if the site segregates its logger network.

5. Agents draft, engineers confirm

Six services (the facility ontology, the on site platform, agents and memory, the command centre, the phase one gates, and Run Operate and Harden) carry the work onto five surfaces, from the trends and open triggers view through the response approval queue to the read only review and regulator views. Three agents draft: pattern, response and reporting. The tailings engineer confirms every pattern draft, the responsible tailings facility engineer approves every response draft, and the engineer of record reviews the surveillance report and approves every regulator submission before it is filed, behind an instruction gate so no draft reaches the regulator unapproved. Every confirmed pattern and its outcome is kept as memory the next trigger is read against. Nothing is trained into weights.

Appendix A prices the node at about 456,000 US dollars to own over three years (437,000 to 496,000): a configured eight card server at about 320,000, support at 8 to 12 percent of hardware value a year, and power taken as a ceiling at 10 kW around the clock at the United States average industrial price. Renting the same eight cards on the deepest three year commitment in a US AWS region is about 544,000, so ownership costs about five sixths of it; on demand it is about 1.45 million, against which ownership costs a little under a third. Renting is a price reference only, because every rented option would carry instrument, survey and inspection data off the owner's hardware.

The rollout is three phases carrying 15 scope items, 12, 2 and 1, each closed by a gate, never a date. Phase one binds one embankment section for two quarters of record across five workstreams, and its gate is the ontology itself: that section's records bound into the object model before the agents are trusted. The run phase hardens the platform, trains the tailings team and hands over. Phase two scales the ontology to every section and opens only after the run phase. The owner's four measures judge it: triggers explained with a confirmed context within a day, false trigger rate, days to assemble a section's story, and findings raised from data the team had not seen, with a false trigger rate gate and an alert budget per engineer per shift beside them.

Made with CodeNinja Praxis, the CodeNinja Atoms platform for designing physical AI systems. The object model imports into Hyper Ontology. Paper, ontology and register: https://codeatoms.ai/tailings-monitoring-us/

One of the CodeNinja Atoms worked examples of physical AI system design: systems of record read and never written, a governed ontology joined by typed links, open weights on hardware the operator owns in its own country, and agents that draft while named people approve, the shape every physical AI design in the series is published in, with its object model as hyper-ontology/1 JSON under CC BY 4.0.

๐Ÿ“ฐ Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ€” full credit and traffic to the original publisher.