An offline AI pilot needs a network test, not a label

Before I would describe a proposed assistant as "offline", I would write down what that promise includes. Does it cover only model inference, or also document import, search, speech, authentication, logs, updates and recovery?

This is a proposed acceptance-test plan. It is not a client case study, a benchmark result or a claim that a particular product has passed these tests.

Define the boundary in plain language

For a hypothetical internal document assistant, my first draft of the requirement would be:

During normal use, prompts, retrieved passages and generated answers must stay inside the approved environment. Any exception needs to be named and tested separately.

That statement is deliberately narrower than "nothing ever leaves the machine". An update process or a remote support session needs its own decision. Do not let a broad marketing label settle those choices.

Then create a dependency inventory:

Treat "unknown" as an open item, not as "local".

Separate installation from normal operation

A useful test plan distinguishes at least two modes:

1. Provisioning: downloading approved software and model files, verifying their origin, configuring access and loading the initial corpus.
2. Normal operation: asking questions, adding permitted documents, searching, generating answers and reviewing errors.

A system that needs a download during installation may still meet a defined disconnected-use requirement. A system that silently sends a prompt to a remote fallback during normal use would not meet the requirement above.

The acceptance criteria should say which mode is being tested.

Test with synthetic data first

I would start with invented documents that contain unique, harmless marker phrases. Do not use real customer records just to find out where a new pipeline sends data.

Run the same small task set in an approved connected test environment and then with outbound traffic blocked by the test environment's network controls. Observe connection attempts and component errors. Record the tool versions and configuration used so the test can be repeated.

A proposed test log:

case: retrieve a policy passage and answer with a citation
input: synthetic-policy-01
network mode: outbound blocked
expected: answer from approved local sources, or a clear unavailable state
unexpected: remote fallback, silent data export, fabricated citation
review: pending

This log is a fixture, not a report of an executed test.

Test the awkward paths too

The happy path is only one part of the pilot. I would include:

• An empty index and a document the parser cannot read.
• A restart while the network remains disconnected.
• An expired session or unavailable identity service.
• A missing model file and a failed update.
• A request that the approved sources cannot answer.
• A support or diagnostics feature that might attach logs.

For each case, define the allowed failure behavior. A visible error is preferable to quietly changing the data boundary or inventing an answer.

Keep privacy and answer quality on separate scorecards

Passing a network-boundary check would not prove that an answer is correct. Passing an answer-quality check would not prove that the data stayed inside the required boundary.

I would review these separately, along with access control, retention and recovery. A pilot decision should name any remaining gaps instead of compressing them into a single "secure" tick.

Make the final claim match the evidence

After the tests, write a scoped statement: which version, which environment, which workflows and which exceptions were checked. Avoid promising that every future feature has the same behavior.

Disclosure: I run IT Impress, whose primary website describes private, custom AI and offline or embedded AI among its services. This article sets out a proposed evaluation approach; it does not claim measured deployment results or certify any system.

By Keshavan Thiagaraja, IT Impress.