Your AI Agent Passed Last Month. Does It Know the Procedure Changed?

An AI agent can return a fluent, cited answer and still be operationally wrong because the cited procedure was superseded yesterday. For a Swiss SME, the useful control is not “re-index everything and hope.” It is a small, repeatable release gate that proves a change travels from an approved source to retrieval, respects permissions, invalidates stale copies, and fails closed while that path is uncertain. The Swiss SMB AI projects discussion is a helpful starting point for the broader industrial use case; this article narrows the question to what happens after an approved work instruction, price list, safety note, or customer rule changes.

The practical answer is to treat freshness as a release property. Give each governed document a stable identity, an explicit effective interval, an approval state, a version, and an access policy. When an approved change arrives, create a change event, invalidate or supersede the old retrieval representation, process the new version, then run tests against both the new and old facts. Keep the agent read-only until the test suite proves that current answers use the new approved version and that restricted or withdrawn information is no longer retrievable. If the refresh state cannot be confirmed, the system should say it cannot verify the current procedure or route to a person. It should not confidently answer from yesterday’s cache.

This is a deployment control, not a claim that every small company needs a sophisticated vector database. The same idea applies to a shared drive search, a CRM knowledge assistant, a SharePoint copilot, or an agent that drafts service replies. The control can be a spreadsheet and a handful of test prompts at first. The important distinction is between a document file changing and the answer path becoming current. Those are separate events, and each needs evidence.

The reader decision: can this agent answer from the rule in force now?

The decision is not whether retrieval augmented generation is generally useful. It is whether a particular agent may answer a particular class of operational questions after a source has changed. A procedure may be edited, approved, scheduled for next week, withdrawn, restricted to a group, renamed, duplicated, or partially ingested. Each change produces a different safe answer. A “last modified” timestamp alone cannot distinguish an approved instruction from an unreviewed draft or a future-effective revision.

For example, a maintenance checklist is approved today but takes effect Monday. On Friday, an agent that always chooses the newest file may prematurely apply the Monday rule. Conversely, if it keeps selecting the current Friday version after Monday, it is stale. The system needs to represent both approval and effective time. The query should resolve the version valid for the question’s date, not blindly retrieve the newest object.

For a narrow, low-risk FAQ, a delayed refresh may be tolerable if the answer displays its source and the user verifies it. For machine safety, contractual commitments, pricing, payroll, medical handling, or access rights, unresolved freshness should block an answer or require a qualified reviewer. This is an inference from the consequence of a wrong answer, not a universal legal classification. A company should classify its own workflows and obtain legal or safety advice where required.

The proposed gate is designed to answer a concrete release question: “After this approved change, can we demonstrate that the agent selects the correct effective version, does not expose a withdrawn or unauthorized version, and admits uncertainty when indexing is incomplete?” If the evidence is missing, the deployment remains in observation or human-reviewed mode. That is a decision rule, not a ranking or performance promise.

What the baseline sources already say

Official platform guidance already treats content governance, lifecycle, permissions, and stale material as related concerns. Microsoft’s SharePoint Admin Agent overview describes assessing content risks, managing content lifecycle, archiving stale content, and preventing oversharing in search and AI responses. Its page also includes an overview video for the administrative agent. That is a useful baseline for what a platform vendor exposes; it does not prove that every update is indexed instantly or that a specific business workflow is safe.

The European Commission’s AI Act FAQ explains that deployers of high-risk systems have duties including monitoring operation, acting on identified risks, and assigning human oversight. Those requirements apply in the legal context described by the Act; they should not be stretched into a claim that every internal SME assistant is high-risk. They do support a broader operational principle: when a system is used in a regulated or high-risk context, oversight and monitoring cannot be replaced by trusting the initial configuration.

The Stanford AI Index 2026 report notes that responsible-AI benchmark reporting remains sparse compared with capability benchmark reporting. The report concerns model-level evaluations, not the retrieval freshness of a company’s own documents. The connection here is an inference: generic model benchmarks cannot tell an SME whether its local corpus was updated, whether a permission filter is respected, or whether an old version remains searchable. Those properties need local tests.

Gartner’s 2026 guidance on AI agent information governance recommends processes to keep agent-accessible information current, manage permissions, and archive obsolete information. This is analyst guidance, not a field measurement of the test protocol below. It reinforces the governance problem but does not validate any particular indexing architecture.

A real unresolved question: what does “updated” mean end to end?

Practitioners often describe a familiar symptom: a source file has changed, but an assistant still returns a previous answer. The Power Platform Community discussion about a changed SharePoint source is one example of users asking whether an agent refreshes after its underlying file changes. It is a single support discussion, not a representative survey and not evidence about all SharePoint tenants. It points to a concrete ambiguity: where in the pipeline did the update stop?

“Updated” can mean the file system saved new bytes; a connector noticed a change; extraction completed; chunks were recreated; an embedding job completed; the retrieval index became queryable; a cache expired; and a response path selected the new version. A green status for one stage does not prove the others. A user may have edited the file correctly while a failed connector, stale cache, permission lag, duplicate source, or version filter continued to serve old content.

The useful response is to expose stage-specific status and an end-to-end acceptance check. In a small deployment, this can be a manually maintained change ledger with timestamps and test results. In a managed platform, use its ingestion logs and lifecycle controls, but still test the actual questions users ask. Avoid claiming a fixed refresh time unless the platform contract and your own observations support that number. Latency can vary by connector, file type, load, permission change, and service behavior.

The question signal justifies making a test that distinguishes “file saved” from “agent answer updated.” It does not justify asserting that stale answers are common or that one vendor is uniquely affected. Our test cases are designed scenarios, not an incident-frequency dataset.

A version model small enough for an SME

Start with a source register. Each governed item receives a stable document_id that does not change when its filename or folder changes. Record the source owner, version identifier, approval status, valid_from, optional valid_until, access scope, content hash, source location, and the identifier of the version it supersedes. Keep a separate ingestion record for extracted text, chunk set, index version, and last successful end-to-end test. Do not use file modification time as a substitute for approval or validity.

The answer resolver should first determine which approved version is valid for the question’s business date and user identity. Only then should retrieval rank content within that eligible set. A reranker cannot repair a candidate set that includes a future draft or excludes the current approved procedure. Similarly, a relevance score does not override an access rule. If the user lacks permission to read a document, the answer path should not surface it through a derived chunk or summary.

Keep a distinction between operational availability and historical retention. A withdrawn policy may need to stop influencing current answers while remaining stored for audit, legal retention, or investigation. Deleting all history can destroy evidence; leaving all historical chunks active can create stale answers. The safer pattern is to mark old versions as not eligible for current retrieval, retain them in a controlled archive if required, and test that archive access separately.

Use explicit states such as draft, approved_future, current, superseded, withdrawn, ingestion_pending, indexed_verified, and quarantined. The names are examples, not a mandated standard. The state transitions matter more than the vocabulary. A single document should not be simultaneously active and superseded for the same purpose and effective date unless the system can explain which version wins.

The freshness release gate

The release gate has five steps. First, capture the change event and identify the affected document and downstream answers. Second, verify the source state: the intended version exists, its approval is recorded, its effective date is correct, and its permissions are defined. Third, wait for the ingestion pipeline to report completion, while recording each stage’s evidence. Fourth, execute the targeted acceptance suite against the real answer interface using a test account with the relevant role. Fifth, release the agent only when current-version, stale-version, access, and failure behavior all match the expected results.

If an ingestion job fails, times out, or reports an unknown state, do not translate “not sure” into “probably fine.” Keep the previous version available only if policy permits and the business rule says it remains valid. If the old procedure has already been revoked, return a clear “I cannot verify the current approved procedure” response and route to the owner. If a future version is approved but not yet effective, preserve the current answer until the effective date.

The release can be proportional. A public, low-impact FAQ might use a short delay notice and a daily spot check. A safety-related instruction needs a stronger approval trail, explicit version selection, and a human owner who can stop the agent. A pricing assistant may need a same-day cutover test and a review of downstream quotes. The gate scales with impact, not with whether the backend happens to use vectors.

flowchart TD
  A[Approved change event] --> B[Resolve owner, scope, effective time]
  B --> C{Approval and permissions valid?}
  C -- no or unknown --> H[HOLD and route to owner]
  C -- yes --> D[Extract and update derived index]
  D --> E{Pipeline complete and cache checked?}
  E -- no --> H
  E -- yes --> F[Run current, stale, draft, and access tests]
  F --> G{Every required case passes?}
  G -- no --> H
  G -- yes --> I[Scoped read-only release with receipt]

Record who approved the change, who ran the test, which identity was used, the expected and actual source version, the result, and the time. Keep a rollback path to the last known-good configuration, but do not roll back to content that is no longer valid. A rollback is an application or configuration recovery mechanism, not permission to restore a revoked business rule.

A reusable 12-case synthetic acceptance suite

The matrix below is an original, hand-authored test asset. Method: on 26 October 2026, we specified twelve synthetic change scenarios and their expected safe behavior by reasoning from source state, effective time, access scope, and pipeline state. Sample size is 12 designed cases. No customer documents, production agents, connector logs, or deployment outcomes were measured. The cases are a starting fixture to adapt, not a pass-rate statistic or proof of system reliability.

The matrix is intentionally small. Add cases for languages, file formats, business units, service accounts, and source types that actually exist in your environment. If a procedure has multiple valid versions for different locations, the expected result must include the location context. If an answer can trigger a write action, extend the suite to cover confirmation, idempotency, and a human approval boundary. Do not claim the 12 cases cover every threat; they make a repeatable minimum visible.

A practical change ledger

Use a row per change, not a vague ticket comment. Suggested fields are change_id, document_id, old_version, new_version, approval_ref, valid_from, withdrawal_time, permission_diff, ingestion_job_id, extraction_status, chunk_set_id, index_status, cache_invalidation_status, test_fixture_version, test_identity, result, operator, and release_decision. The fields can live in a spreadsheet at first. Restrict access to the ledger if it contains document names, security roles, or customer references.

Keep timestamps and states separate. source_saved_at is not approved_at; index_job_finished_at is not answer_verified_at. Those distinctions let an operator locate a delay without guessing. A value such as unknown is better than filling a blank with a guessed time. If a test is rerun after a permission or content change, retain the earlier result and append the new one rather than overwriting history.

For each high-impact source, assign an accountable owner and a backup. If the owner cannot confirm which version is in force, the agent should not invent that decision. The owner can approve a new version, mark a withdrawal, or tell the system that two documents govern different scopes. This is a governance decision, not a prompt-engineering trick.

An evidence row might read: change_id=WI-042, old_version=7, new_version=8, approval=approved, valid_from=2026-10-26T08:00+02:00, index_status=complete, current_version_test=pass, old_version_trap=pass, role_test=pass, release=read-only. The identifiers are illustrative, not real records. Do not put confidential document contents into a public prompt or a published article to make the example feel concrete.

A small executable gate for teams with a JSON export

The following Node.js snippet is deliberately a policy checker, not an indexing integration. It expects one JSON object per source with approval, effective date, access scope, and retrieval state. It fails closed if the source is unapproved, not yet valid, withdrawn, not indexed and verified, or missing an explicit permission match. In production, obtain these values from trusted systems; never accept model-generated values as authoritative control state.

function eligible(source, { now, userGroups }) {
  const instant = new Date(now).getTime();
  const start = Date.parse(source.validFrom);
  const end = source.validUntil ? Date.parse(source.validUntil) : Infinity;
  const approved = source.approvalStatus === 'approved';
  const notWithdrawn = !source.withdrawnAt || Date.parse(source.withdrawnAt) > instant;
  const indexed = source.indexStatus === 'indexed_verified';
  const inWindow = Number.isFinite(start) && start <= instant && instant < end;
  const allowed = Array.isArray(source.allowedGroups) &&
    source.allowedGroups.some(group => userGroups.includes(group));
  return approved && notWithdrawn && indexed && inWindow && allowed;
}

function releaseDecision(change, tests) {
  const required = ['currentVersion', 'oldVersionTrap', 'permissions', 'failureClosed'];
  const failures = required.filter(name => tests[name] !== 'pass');
  if (change.pipelineState !== 'complete') failures.push('pipelineState');
  if (change.approvalStatus !== 'approved') failures.push('approvalStatus');
  return failures.length
    ? { state: 'HOLD', failures }
    : { state: 'READ_ONLY_RELEASE', failures: [] };
}

The code is intentionally conservative and incomplete. It does not validate timezone ambiguity, cryptographic signatures, overlapping scopes, group membership freshness, document hashes, retention rules, or race conditions between the test and release. It assumes an upstream system has verified the source values. A real implementation should evaluate permissions at request time, record the selected source identifiers, and prevent a stale answer cache from bypassing current access checks. Do not copy this snippet into production without tests and review.

Where the check should run in the pipeline

The minimum path is source register → approval and effective-date resolution → extraction → chunk or document representation → index update → cache invalidation → permission-aware retrieval → answer citation → release record. Put a status signal at every boundary. If a connector says “complete” but the agent still cites the old version, compare the source identifier returned by retrieval with the expected identifier. If that identifier is absent from the interface, instrument or export a trace before using the agent for consequential answers.

Do not confuse a vector index with the source of truth. The authoritative source remains the approved document or structured record. The index is a derived representation. If rebuilding the representation takes time, the system needs a defined transition behavior: use the still-valid old version, abstain, or route to a person. It should never silently combine fragments from a superseded version and a new draft.

The same principle applies to metadata filters. A robust filter can exclude drafts, expired versions, and unauthorized records before similarity search. A post-retrieval filter may be too late if a forbidden chunk already influenced ranking or an answer cache is shared. The exact risk depends on architecture; test the deployed system instead of assuming one filter position is always correct.

For small teams using a vendor-managed assistant, you may not control chunking or refresh scheduling. You can still maintain a source-of-truth register, submit an update through the supported UI, run the 12 prompt fixtures, inspect citations, and keep the agent in human-review mode until the results pass. If the product exposes no usable evidence or no way to remove stale material, that limitation itself belongs in the deployment decision.

Contradictions and trade-offs

Aggressive deletion reduces the chance that an obsolete instruction is retrieved, but it can remove records needed for audit, dispute handling, or historical questions. The answer is usually separate lifecycle states for current retrieval and retained archive, with different access controls and tests. Do not let a historical archive remain a default answer source simply because it is retained.

Immediate refresh is attractive, but “re-index on every save” can be wasteful and can promote unapproved drafts. A better trigger is an approved change event, with future effective dates represented explicitly. For a genuinely urgent revocation, a separate emergency block may remove the old source from current retrieval before the replacement is fully indexed; during that gap, the agent should abstain or route to a human.

Human review is safer in some situations but can create queues and encourage users to bypass the process. Use review where impact warrants it, and make the safe path usable. Low-impact, stable FAQs can be released with lighter controls and sampled checks. High-consequence decisions need stronger controls, documented ownership, and an explicit stop mechanism. Governance should be proportional to what the agent can do and what harm a stale answer could cause.

Freshness testing also cannot establish truth. A newly approved document may itself be wrong, incomplete, or inconsistent with another valid source. The gate proves that the answer path uses the intended source state; it does not validate the underlying business rule. The source owner remains responsible for approval, and the agent’s citation should help the user verify the content.

Finally, not every question requires current data. A historical query such as “what did our procedure say last March?” should be answered from the historical version if the user is authorized and the date is explicit. A current operational question should resolve to the version valid today. Store the question’s temporal intent or ask a clarification rather than forcing every request through a single “latest document” rule.

Failure modes to rehearse

The file changed but the index did not. A connector may queue work, fail extraction, or report a partial completion. Expected control: keep the change in ingestion_pending, make the test fail, and route to the owner. A green file-save indicator is not a release receipt.

The new draft wins because it is newest. The metadata has a recent timestamp but lacks approval or has a future effective date. Expected control: eligibility is based on approval and valid time, not recency alone. Test both a future approved version and an unapproved recent draft.

The old version remains a strong semantic match. New and old documents share similar language, so vector similarity returns both. Expected control: filter by document state and validity before ranking, then run the old-version trap prompt. If the interface cannot expose source identifiers, escalate instrumentation rather than treating fluent citation text as proof.

A revoked user still sees a cached answer. Retrieval permissions may have changed while a shared cache still holds a previous result. Expected control: re-evaluate identity and access before serving a cached response, or invalidate the relevant cache. Test with the user whose access was removed, not only an administrator.

A duplicate file has a different path. Renaming or copying a file creates a second identity, leaving the old version active. Expected control: stable document identifiers, content hashes or duplicate review, and a conflicting-copy fixture. Do not assume filenames uniquely identify policy.

The status dashboard is green but the answer is wrong. The dashboard covers an ingestion stage, not the user-facing query path. Expected control: run the actual prompt through the deployed agent and inspect its source IDs, effective version, and permissions. Pipeline metrics and answer acceptance tests complement each other; neither replaces the other.

The agent invents a resolution for conflicting owners. Two departments approve different procedures for different scopes. Expected control: record the scope and owner, ask for missing location or role, or abstain. Similarity should not decide organizational authority.

A workaround restores an obsolete rule. An operator rolls back the index to repair a failed deployment but also restores a withdrawn safety note. Expected control: separate software rollback from content validity. A rollback must preserve the current blocklist and effective-date rules.

How to measure a pilot without inventing a success rate

For a pilot, choose a bounded workflow and record baseline behavior before turning the agent loose. Count eligible source changes, time from approved change to verified answer, test case results, stale-version retrievals, permission failures, abstentions, and human corrections. Report numerator and denominator for each measure. For example, write “the test suite had 12 designed cases; 11 passed on the first run” only if that actually happened in your environment. In this article, no such run occurred, so no pass rate is claimed.

Separate operating metrics from outcome metrics. Ingestion delay describes the pipeline. Stale-answer count describes the tested answer path. Escalation rate describes how often the system abstains or routes. A business outcome such as reduced handling time requires a comparison design that accounts for case mix, training, and human review; it cannot be inferred from the number of indexed files.

Set thresholds before the test. A high-impact workflow might require zero unauthorized retrievals and zero use of withdrawn content in the release suite. A low-impact internal search tool may accept a stated refresh window if users see a clear “content last verified” marker and can report stale answers. These are recommendations to adapt, not thresholds prescribed by the cited sources.

Keep the sample visible. If you test only one file format, one user role, one language, and one connector, say so. Do not generalize to other teams or content types. Add a regression case for every real freshness incident, and preserve the expected answer and source version in a controlled fixture. If the underlying policy changes again, update the fixture with its owner’s approval rather than leaving an old expected answer in the suite.

A rollout sequence that can be reversed

Begin with observe-only mode. The agent retrieves and cites but does not change a record, send a message, or instruct a worker to perform a safety-sensitive action. Keep an owner beside the source register. Run the 12 synthetic cases against the chosen platform, then replace or extend them with actual approved procedures and test identities. Record where the system exposes source IDs, timestamps, and permission context.

Next, run a shadow period: users follow the approved source as usual while the agent produces a parallel answer for review. Compare the answer’s source version with the expected current version. Treat mismatches as defects even when the wording looks plausible. Use a documented change set so the test has a known old version, new version, approval record, and effective date.

Only after the source-path tests pass should the agent assist a real workflow. Start with read-only recommendations or drafts. Keep write actions behind human approval and retain a stop switch. If a freshness check fails, suspend the affected knowledge source or the entire agent according to the risk boundary. Do not continue because “the model usually cites the right page.”

After release, monitor source changes, index status, permissions, stale answer reports, and test regressions. Re-run the relevant suite after a connector upgrade, schema change, permission change, model change, prompt change, or source migration. A stable prompt does not guarantee a stable source path. Make the review cadence proportional to change rate and impact, then revise it when real evidence shows the cadence is too slow or unnecessarily costly.

The release receipt

A release receipt should let another operator reproduce the decision. Include the article or source IDs tested, old and new version identifiers, approval and effective-time evidence, identity and groups used, ingestion job and status, cache behavior, test fixture version, expected versus actual citations, failures, reviewer, and release scope. Redact document contents and personal data where they are not needed. Link to the internal audit record rather than copying confidential text into a public issue.

Use a release state that cannot be confused with a draft. HOLD means a required check failed or is unknown. READ_ONLY_RELEASE means the defined source and retrieval tests passed for the stated scope, not that the model is universally reliable. HUMAN_APPROVAL_REQUIRED means the output may be prepared but not executed. REVOKED means current answers must not use that source. These labels are examples; consistency and clear ownership matter more than the exact strings.

The receipt should state exclusions. If you did not test multilingual copies, image-only PDFs, external guests, offline clients, or cached mobile responses, list them. The result applies only to the tested configuration and time. A later connector update or permission change can invalidate the receipt, so record a review trigger rather than treating it as permanent certification.

If the business owner cannot explain the intended version or effective date, stop. The engineering team can diagnose a pipeline but cannot decide which operating rule the organization has approved. Resolve that question first, then run the test. This boundary prevents a technical deployment from quietly becoming a policy-making process.

Checklist for the next procedure change

• Identify the stable document or record ID and the owner accountable for its content.
• Record the old version, proposed version, approval state, effective date, and any withdrawal time.
• Confirm whether the new version applies to all locations, roles, languages, and customer classes.
• Record the permission difference and test with identities representing allowed and denied users.
• Track extraction, indexing, version filter, cache invalidation, and retrieval verification separately.
• Run a new-version answer, an old-version trap, a draft trap, a future-date test, and a withdrawn-source test.
• Test partial failure and confirm the agent abstains or routes to a named human owner.
• Inspect the cited source identifier and effective version through the actual user interface.
• Keep retained historical versions out of current retrieval while preserving required archive access.
• Write the release receipt, including the test date, case sample, exclusions, and unresolved risks.
• Release only within the tested scope; keep write actions behind the approval boundary.
• Schedule a regression run after source, connector, permission, model, or retrieval configuration changes.

Recommendation

Do not ask whether your AI agent’s documents are “up to date” as a single yes-or-no question. Ask which approved version is valid for this user, this scope, and this date; what evidence shows it reached the actual answer path; and what the agent does while that evidence is missing. The reusable 12-case suite makes those questions testable without pretending to measure a live system.

For a Swiss SME, the sensible starting point is a small, read-only pilot over a narrow set of procedures with named owners and an explicit release receipt. Keep current retrieval separate from archival retention. Treat approval, valid time, access, indexing, and caching as independent controls. If your vendor hides the answer’s source version or cannot make a withdrawn document unavailable, record that as a product limitation and keep a human in the loop.

The article’s case count is the size of a designed fixture, not evidence that twelve prompts guarantee safety. The European Commission, Microsoft, Stanford HAI, and Gartner sources cited above support different parts of the governance context; none certifies this protocol or your implementation. The protocol’s value is that it turns a vague freshness promise into a repeatable, scoped release decision with a clear failure path.

Sources and limits

• European Commission: Navigating the AI Act — deployer duties described for high-risk AI systems; applicability depends on context.
• Microsoft Learn: SharePoint Admin Agent — product documentation for lifecycle, stale content, permissions, and governance features; it is not an independent validation of refresh latency.
• Stanford HAI: 2026 AI Index Report — model-level responsible-AI evaluation context, not an SME retrieval benchmark.
• Gartner: AI agent information governance steps — analyst recommendation; not treated here as a measured comparison of vendors.
• Power Platform Community: discussion of a changed SharePoint source — one user discussion, used only as a question signal.
• Microsoft Learn video: Overview of the SharePoint Admin Agent — the page embeds a short product overview video. Its accessible documentation describes lifecycle and access controls; we did not obtain a separate video transcript, so no claim here depends on unseen spoken content.

Failure mode summary

The most dangerous failure is a pipeline that reports “updated” while the user-facing answer remains based on an old or unauthorized version. Catch it by testing the deployed query path with the actual identity, inspecting the returned source version, and deliberately asking prompts that should retrieve the old, future, withdrawn, or restricted material. A second failure is to overcorrect: deleting history, blocking every agent, or forcing human approval for every low-impact search can make a safe workflow unusable. Separate current eligibility from retention, and apply controls according to scope and consequence.

The recommendation fails when a business has no reliable owner or approval record for its procedures. No retrieval filter can resolve an organization’s uncertainty about which rule is in force. It also fails when platform access does not expose enough telemetry to verify source IDs, access decisions, or cache behavior. In that case, use a narrower source set, a human-reviewed answer path, or a different tool until the observable controls exist.

Final validation checklist

Before releasing a changed knowledge source, confirm that a person can answer all of these questions from the receipt: What changed? Who approved it? When does it apply? Which user identity was tested? Which version did the agent cite? What happened to the former version? Were permissions checked after the change? Did a failed or partial refresh block release? What did the suite not test? Who can stop the agent? When does the evidence expire or require a rerun?

If any answer is unknown, the right result is not a green status invented by the model. It is a hold, a scoped fallback, or a human route. This gives a small team a practical way to use an assistant while preserving the distinction between a useful answer and an approved operational instruction.