Dev.to Security 🔐 Cybersecurity 👁 0 📖 8 min read

Internal Documentation Retrieval: Enforcing Access Rules Before Context Assembly

Internal engineering documentation retrieval architecture has one hard boundary: a page-change alert is not permission to expose the changed page. Authorization has to survive the entire path from a watched document to t

Internal engineering documentation retrieval architecture has one hard boundary: a page-change alert is not permission to expose the changed page. Authorization has to survive the entire path from a watched document to the context sent to a model.

Short answer: enforce access rules before ranking and context assembly, carry document identity and policy revision on every chunk, and treat content changes and permission changes as separate, replayable events.

I've been paged for missed jobs and duplicate deliveries in cron and queue infrastructure. The same failure shape appears in documentation retrieval: a missed update leaves stale content searchable, while a duplicate update can race with a permission change. The invariant is more important than the trigger: every candidate chunk must be authorized for the requesting principal at read time, even when ingestion, indexing, and alerts are delayed or replayed.

How should retrieval architecture enforce access rules for internal engineering documentation?

Put an authorization boundary between candidate generation and ranking. The retriever may search an index partition that is already narrowed by tenant or security domain, but it still must evaluate each candidate against the current access policy before the chunk can influence a score, prompt, citation, log, or cache entry. Filtering after context assembly is too late because unauthorized text has already crossed the boundary.

That gives the request path a useful shape: authenticate the caller, resolve a stable principal and group set, generate candidates inside an allowed coarse partition, apply the current policy to document IDs, rank only the surviving chunks, and assemble citations from those same authorized records. A caller should receive no clue that a restricted runbook matched the query.

Keep denial quiet.

The access decision belongs to the document, not to an arbitrary chunk. Each chunk should carry a document ID, content revision, policy ID, policy revision, and source location. If a document's policy cannot be resolved, fail closed for that document. A 403 may be appropriate at a direct document endpoint, but a search response normally omits denied candidates rather than returning a per-result authorization error.

This is a practical implementation boundary in Go. The interfaces are deliberately generic; the important part is the order of operations and the data that remain attached to a candidate.

package retrieval

import (
    "context"
    "errors"
    "sort"
)

type Principal struct {
    ID     string
    Groups []string
}

type Candidate struct {
    DocumentID    string
    ChunkID       string
    Content       string
    ContentRev    string
    PolicyID      string
    PolicyRev     string
    SourceLocator string
    Score         float64
}

type Index interface {
    Candidates(context.Context, string, string) ([]Candidate, error)
}

type Authorizer interface {
    CanRead(context.Context, Principal, string, string, string) (bool, error)
}

func Retrieve(ctx context.Context, idx Index, auth Authorizer, p Principal, domain, query string, limit int) ([]Candidate, error) {
    candidates, err := idx.Candidates(ctx, domain, query)
    if err != nil {
        return nil, err
    }

    allowed := make([]Candidate, 0, len(candidates))
    for _, c := range candidates {
        if c.DocumentID == "" || c.PolicyID == "" || c.PolicyRev == "" {
            continue // Missing policy metadata is a denial, not an indexing default.
        }

        ok, err := auth.CanRead(ctx, p, c.DocumentID, c.PolicyID, c.PolicyRev)
        if err != nil {
            return nil, errors.New("authorization decision unavailable")
        }
        if ok {
            allowed = append(allowed, c)
        }
    }

    sort.SliceStable(allowed, func(i, j int) bool {
        return allowed[i].Score > allowed[j].Score
    })
    if len(allowed) > limit {
        allowed = allowed[:limit]
    }
    return allowed, nil
}

There is a catch: per-candidate authorization adds work to the read path. Coarse partitions and batched policy checks can reduce that work, but a cached allow decision needs a key that includes the principal's effective membership and the policy revision. A time-only cache key can outlive an urgent access revocation. Don't make that trade without measuring revocation latency as an explicit security objective.

The incident invariant: replay content, recheck permission

Consider a developer-tools service that watches internal web pages, computes a diff, updates a retrieval index, and alerts an owning team. The page watcher can use HTTP validators such as ETag or Last-Modified; an unchanged representation can return 304, avoiding an unnecessary download. A changed representation creates a content event. A group-membership edit or page ACL edit creates a policy event. They are related, but they are not interchangeable.

I would make both event types idempotent and independently replayable. A content event identifies the document and immutable content revision. A policy event identifies the document, policy revision, and effective time. Consumers record the highest applied revision per stream, ignore an exact duplicate, and reject an older event that arrives after a newer one. The index write goes to a new document revision, then an atomic pointer or equivalent visibility marker makes it searchable. Until that switch, readers stay on the prior complete revision.

This is where queue instincts matter. Acknowledging the content job before its index revision is durable creates a missed-update window. Assuming the job runs once creates duplicate side effects, including repeated alerts. The alert should therefore have its own deterministic delivery key, derived from document identity and the transition between content revisions, while the retrieval authorization check remains live and independent of whether the alert was delivered.

One more guard is worth the storage: keep citations bound to the exact content revision returned by retrieval. A citation that points only to a mutable page URL can appear to support text that has since changed. The user-facing link may remain friendly, but the evidence record needs the document ID, revision, and source span used for the answer. Retrieval-augmented generation was introduced as a way to combine generated text with retrieved external memory; that only helps grounding when the retrieved evidence remains identifiable.

Small rule. Big blast radius.

What must change when a watched page or its access policy changes?

Content and policy need different invalidation paths because their correctness deadlines differ. A normal content edit can tolerate the indexing service's documented freshness objective. An access revocation generally needs a tighter path: update the policy source, invalidate authorization caches, and prevent stale index metadata from granting access. Read-time authorization is the backstop while asynchronous index maintenance catches up.

The failure modes are easier to reason about as state transitions than as a list of components:

Change Required action Safe read behavior during propagation Evidence to retain
Page body changes Fetch, parse, chunk, and publish a new content revision Serve the prior complete revision if it is still authorized Validator, content revision, chunk source spans
Reader gains access Publish the new policy revision and invalidate the relevant denial cache Omission is acceptable until the new decision is visible Actor, policy revision, decision time
Reader loses access Publish the new policy revision and invalidate allow decisions Deny or omit immediately when policy cannot be confirmed Actor, policy revision, revocation time
Page moves or is deleted Tombstone the old identity and update citations Do not silently attach old chunks to a new page Old and new identity, terminal revision
Group membership changes Recompute effective membership or invalidate its version Treat an unresolved membership version as denied Principal, group-set version, decision reason

Notice that the index is not the policy authority. Encoding groups into index filters can be a useful first pass, especially when it prevents broad cross-domain candidate generation, but denormalized ACL metadata will lag sometime. The authoritative decision still uses the current document identity and policy revision. I'm not sure a single propagation target fits every organization; the right number depends on the identity provider, policy store, and severity of the documentation involved. A measured revocation test settles that question better than an architecture diagram.

For a page watcher, conditional requests also need careful interpretation. 304 Not Modified says the selected representation has not changed under the request's validators; it says nothing about who may retrieve a previously indexed copy.

Authorization still runs.

Preventative checks belong in deployment and the runbook

Unit tests should exercise allow, deny, missing-policy, stale-policy, and authorizer-unavailable paths. Integration tests should then force ordering problems: publish content revision B before policy revision B, replay content revision A, revoke a group while a retrieval cache is warm, and deliver the same change event twice. The expected result is boring: no unauthorized candidate reaches ranking, no duplicate alert is emitted for the same revision transition, and citations name the content revision actually used.

Deployment needs a shadow phase for policy changes. Evaluate the new policy beside the current one, record decision differences without recording query or document text, review unexpected expansions, and only then make the revision authoritative. Logs should contain stable IDs, decision reason codes, policy revisions, and latency. They should not contain chunk bodies or assembled prompts by default. Search telemetry can become a second copy of the sensitive documentation it was meant to protect.

The runbook starts with two questions: "Can an unauthorized principal retrieve any restricted chunk?" and "Which revision is authoritative?" If the first answer is unknown, disable retrieval for the affected security domain or fail closed at authorization. If content freshness is degraded but policy decisions remain current, serving the last complete, authorized revision may be acceptable. Those are different incidents and should page different owners.

Track at least authorization decision latency, denied-candidate counts, policy revision lag, content revision lag, duplicate event suppression, and citation resolution failures. Avoid a raw denial-count alert with no baseline; normal queries can generate denied candidates in a shared coarse partition. Alert on a change in rate, a violated lag objective, or any canary retrieval that crosses a seeded boundary.

A reconciliation job completes the queue design. It periodically compares source validators, visible index revisions, and policy revisions, then republishes missing work with the same idempotency keys. This catches the job that was never delivered without turning every periodic scan into a fresh alert. The schedule is not the correctness boundary — read-time authorization is — but reconciliation shortens stale-content and stale-metadata windows.

Where this architecture is not the right fit

This design is not suitable when the retrieval system cannot evaluate authorization before model context is assembled. In that case, keep restricted documents out of that system and use separate indexes or a retrieval service with an enforceable policy boundary. Post-generation redaction is not an equivalent control.

It may also be excessive for a corpus that is intentionally public and has no per-document access rules. A simpler change watcher, one shared index, and revision-bound citations can be enough there. Conversely, highly regulated material may require physical or cryptographic isolation between security domains rather than a shared candidate index plus logical filtering. Stick with isolated retrieval paths when policy or threat modeling requires that stronger boundary.

The operational decision is straightforward: choose the least complex architecture that can prove an unauthorized chunk never reaches ranking or context, can revoke access within the required window, can replay page changes without duplicate effects, and can trace every answer citation to an immutable source revision. Everything else is tuning.

Sources

📰 Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.