Your AI Cites Sources. Can It Survive a Correction!
Your RAG System Has Citations. Does It Have Provenance? The answer is correct. The citation opens. The quoted passage supports the claim. Then the source changes. Your system continues serving the old answer. Was the
Your RAG System Has Citations. Does It Have Provenance?
The answer is correct. The citation opens. The quoted passage supports the claim.
Then the source changes.
Your system continues serving the old answer.
Was the citation feature working?
In a narrow sense, yes: it connected a statement to supporting evidence. But it did not necessarily preserve the relationship between the answer, the exact source version, the retrieval event, and the sourceโs subsequent lifecycle.
That distinction is the subject of this article.
A citation is a visible reference. Provenance is a record of relationships.
When building retrieval-augmented generation systems, we need bothโand we need to stop treating one as proof of the other.
The architecture and metrics below are engineering proposals, not a new standard or reported benchmark. Examples are hypothetical unless explicitly attributed.
1. โWhere did this answer come from?โ is four different questions
Imagine this pipeline:
Original research
โ
โผ
A technical explainer
โ
โผ
Your document index
โ
โผ
A retrieved passage
โ
โผ
A generated answer
The answer cites the explainer.
Is that the right source?
It depends on what you are asking.
| Question | Required relationship |
|---|---|
| What did the system retrieve? | Answer to retrieval record |
| What supports this claim? | Claim to supporting evidence |
| Who introduced this idea? | Idea to intellectual origin |
| Which artifacts must change if the source changes? | Source version to dependent artifacts |
The explainer might be the correct answer to the first question, an adequate answer to the second, an incomplete answer to the third, and insufficient information for the fourth.
These are not interchangeable requirements.
The W3C PROV data model makes a useful distinction between entities, activities, and agents. It also distinguishes relationships such as derivation, revision, quotation, and attribution. That vocabulary is richer than a flat list of URLs because it can describe how something came into existenceโnot merely where a reader can click. w3
For a RAG application, that suggests a useful starting point:
Source revision โ entity
Chunking run โ activity
Chunk โ entity
Retrieval run โ activity
Generation run โ activity
Answer โ entity
Responsible service โ agent
You do not have to adopt RDF or a graph database to benefit from this model.
You do have to preserve the distinctions.
2. A citation can be correct without being a causal explanation
Consider this answer:
The recommended retry policy uses exponential backoff with jitter. [Source A]
Suppose Source A supports the statement.
That establishes a support relationship.
It does not, by itself, establish that:
- Source A was retrieved before generation.
- Its relevant passage was included in the modelโs context.
- The model depended on that passage.
- Source A introduced the recommendation.
- The answer would differ without it.
These are separate claims requiring separate evidence.
Citation benchmarks such as ALCE explicitly evaluate dimensions including answer correctness and citation quality. Those are valuable targets: they ask whether answers are accurate and whether citations provide support. They do not make a displayed citation a complete causal history of generation. arxiv
A useful engineering distinction is:
Supports(answer, source)
versus:
ObservedInContext(source, generation_run)
versus:
DependsOn(answer, source)
The first is a semantic judgment.
The second is an execution fact you can log.
The third is a stronger claim that usually requires intervention to investigate.
Do not store all three as source_used: true.
Retrieval is not exposure
Even execution records need precision:
Retrieved
โ
Selected by reranker
โ
Included after truncation
โ
Cited
โ
Necessary for the answer
If you log the initial top-20 results but only pass five chunks to the model, your retrieval log is not an exact record of the modelโs context.
If the context builder truncates one of those chunks, even the selected-chunk list is incomplete evidence.
The unit worth recording is the actual context delivered to the generation call, subject to your privacy and retention constraints.
3. Model provenance as a typed graphโnot an unqualified source list
A practical internal representation might look like this:
source:policy
โ has_revision
โผ
revision:r17
โ chunked_into
โผ
chunk:c91
โ included_in
โผ
context:ctx44
โ supplied_to
โผ
generation:g28
โ produced
โผ
answer:a63
โ served_as
โผ
cache_entry:k12
Add a separate relationship for a verified claim-level citation:
claim:a63:2 โโ supported_by โโ> chunk:c91
Why separate it?
Because inclusion in a context does not prove support. A chunk can be present and irrelevant. Conversely, an answer may contain a true claim that none of the supplied chunks supports.
This graph is operational lineage. It records observed processing and assessed relationships.
It is not a map of everything that influenced the modelโs parameters.
Distinguish source identity from source revision
A URL identifies a location. It does not uniquely identify the content served there over time.
Use separate identifiers:
{
"source_id": "src_policy",
"revision_id": "rev_r17",
"canonical_url": "https://example.org/policy",
"content_sha256": "<digest-of-normalized-content>",
"retrieved_at": "<UTC timestamp>",
"status": "active"
}
The hash helps establish which bytes or normalized representation you recorded.
It does not prove authorship, priority, truth, or permission.
Also record the normalization policy. Two pipelines can produce different hashes from the same page because one removes navigation and the other does not.
Version your transformations too
Chunks depend on more than source text.
They also depend on:
- Parser version.
- Normalization rules.
- Chunking strategy.
- Chunk size and overlap.
- Embedding model version.
If the source stays unchanged but your parser drops a warning paragraph, your downstream representation has still changed.
A useful chunk record therefore includes:
{
"chunk_id": "chunk_c91",
"revision_id": "rev_r17",
"chunker_version": "chunker_v3",
"parser_version": "parser_v2",
"text_sha256": "<digest-of-chunk-text>",
"embedding_model": "<configured-model-version>"
}
This is not administrative decoration. It is what lets you explain why two builds of โthe same corpusโ produced different answers.
4. Source changes are dependency events
Here is the operational consequence of the graph:
A source update is not merely an ingestion event.
It may invalidate downstream artifacts.
Source revision superseded
โ
โผ
Dependent chunks become stale
โ
โผ
Retrieval materializations change
โ
โผ
Dependent answer caches are invalidated
โ
โผ
Affected evaluations are rerun
OWASPโs RAG guidance explicitly calls for lifecycle handling across source documents and derived data, including removal when documents are deleted, permissions change, or retention expires. This makes provenance relevant to security and deletionโnot just citation presentation. cheatsheetseries.owasp
Start with conservative invalidation
For a cached answer, record every source revision present in its generation context.
That may overestimate dependency: some chunks were probably irrelevant.
But it gives you a conservative invalidation rule:
If any context revision becomes unavailable or stale, invalidate the cached answer.
Later, you can optimize with claim-level dependency assessments.
Do not start by assuming that only displayed citations matter. An answer may have been shaped by an uncited passage.
Separate the event types
These events should not all trigger identical handling:
| Event | Possible response |
|---|---|
| Content revised | Reindex and mark dependent answers stale |
| Source withdrawn | Stop serving affected derived answers pending review |
| Access revoked | Recheck authorization and invalidate affected caches |
| Erasure required | Purge governed derivatives and verify completion |
| URL moved | Update location while preserving revision identity where justified |
A historical answer may remain useful for an audit after a policy revision.
The same retention may be unacceptable after a valid erasure requirement.
Lineage tells you what is related. Policy determines what action is allowed.
5. Deletion is a workflow, not one database command
Avoid both extremes:
โDeleting the vector removes everything.โ
and:
โOnce embedded, content can never be removed.โ
Neither is a useful architectural rule.
In a controlled RAG system, you can design deletion across source-linked artifacts. The hard part is locating and handling all relevant derivatives, not merely issuing a delete against one index. OWASP specifically treats derived-data lifecycle handling as part of RAG security. cheatsheetseries.owasp
A deletion workflow might cover:
Document store
Vector index
Keyword index
Retrieval cache
Answer cache
Temporary processing files
Evaluation copies
Governed logs
Backups under their retention policy
This is a proposed inventory, not a claim that every application needs every store.
A useful deletion record distinguishes:
Requested
Access blocked
Active-store purge completed
Verification completed
Backup expiry pending
Closed under policy
โNot retrievable through the applicationโ and โphysically removed from every retained copyโ are different completion claims.
Your logs should say which one you have verified.
6. Test attribution under controlled exposure
An open-web prompt such as โExplain my frameworkโ cannot cleanly diagnose attribution failure.
If your work is not cited, several explanations remain:
- It was never retrieved.
- It was retrieved but excluded.
- It was included but ignored.
- Another source supported the same answer.
- The response did not actually depend on your contribution.
Start with controlled exposure instead.
Build a small, private test corpus
Create synthetic technical documents containing distinctive, harmless facts.
For example:
The fictional Oriole queue uses a retry ceiling of 13 attempts.
Keep these documents out of public discovery during the experiment.
Prepare matched conditions:
| Condition | Available material |
|---|---|
| A | Original document with source identity |
| B | Same document without identity fields |
| C | Derived explainer with an explicit link to the original |
| D | Original plus derived explainer |
| E | No document containing the test fact |
Ask substantive questions:
What retry ceiling should the Oriole queue use?
Do not only ask:
Who wrote the Oriole queue document?
The first tests answer generation and attribution together. The second mainly tests identification.
Log the actual exposure
For each run, save:
- Corpus snapshot.
- Prompt.
- Retrieved chunk IDs.
- Reranking results.
- Final context IDs and revisions.
- Model configuration.
- Raw answer.
- Extracted citations.
- Evaluation judgments.
Use repeated trials and report the counts.
Do not treat responses from the same prompt and corpus as a collection of entirely unrelated observations. The shared setup matters when interpreting uncertainty.
7. Use intervention to investigate dependence
To examine whether a source matters, change its availability.
Run the same question under:
Condition 1: original source available
Condition 2: original removed, derivative available
Condition 3: both removed
Then compare:
- Presence of the distinctive test fact.
- Correctness of the answer.
- Attribution to the original.
- Attribution to the derivative.
- Abstention behavior.
You can define an experimental contrast:
[
\Delta =
P(\text{target fact appears}\mid \text{source available})
P(\text{target fact appears}\mid \text{source unavailable})
]
This estimates an availability effect under your experimental setup.
It does not identify every historical influence on a pretrained model.
Also decide what you want to intervene on:
- Removing a document from the corpus tests the end-to-end retrieval-and-generation pipeline.
- Removing a passage from a fixed context tests generation under controlled context changes.
Those are different experiments.
If an alternate source contains the same fact, removing one document may leave the answer unchanged. That does not prove the removed document was irrelevant in the original run.
8. Measure a vector, not a magic score
A single โtrust scoreโ is tempting.
It also hides different failure modes.
I would start with separate measurements:
| Dimension | Question |
|---|---|
| Evidence coverage | How many externally checkable claims have supporting evidence? |
| Citation support | How many evaluated claimโcitation pairs actually support the claim? |
| Attribution accuracy | When origin is known and relevant, is it correctly identified? |
| Lineage coverage | How many served answers have a complete operational dependency record? |
| Freshness | How often do answers depend on superseded revisions? |
| Correction latency | How long does a source change take to affect served behavior? |
| Purge verification | How many required artifact stores have verified completion? |
The first two are related to the citation-quality concerns evaluated by benchmarks such as ALCE. The operational dimensions extend the evaluation target to system behavior over time. arxiv
For each metric, declare:
- The unit being counted.
- The denominator.
- The eligibility rule.
- The treatment of unknown cases.
- Whether judgment is automatic, human, or mixed.
For example:
[
\text{Citation support rate}
\frac{\text{supported claimโcitation pairs}}
{\text{evaluated claimโcitation pairs}}
]
This is not the same as the proportion of all claims that have support.
A system could cite only one easy claim and score perfectly on citation support while leaving the rest unsupported.
Report both dimensions.
Do not hide unknowns in the denominator
Suppose you cannot establish an ideaโs original author.
That is not automatically incorrect attribution. It is unresolved attribution.
Use an explicit status:
correct
incorrect
not_applicable
unresolved
Report unresolved cases separately rather than quietly converting them into successes or failures.
9. Correction propagation is the most revealing integration test
Here is the test I would run before polishing citation chips.
- Ingest a synthetic policy revision.
- Ask a question and cache the answer.
- Publish a corrected revision.
- Trigger the update workflow.
- Ask the same question through every serving path.
- Verify that obsolete cached answers are not served as current guidance.
Test paths separately:
Fresh retrieval
Exact-match answer cache
Semantic answer cache
Existing conversation state
Background-generated summary
The important property is not merely that the new document is searchable.
It is that the old representation no longer silently controls current answers.
Define correction latency as:
[
L = t_{\text{verified serving change}} - t_{\text{accepted source update}}
]
Be precise about the start time.
If your ingestion job runs hourly, measuring from ingestion completion hides discovery delay. You may need two measurements:
- Detection latency.
- Downstream propagation latency.
Likewise, a finite test suite cannot prove that no stale answer exists anywhere. It can establish that your declared serving paths passed specified checks.
That is still much stronger than โthe index updated successfully.โ
10. Provenance records can also be wrong
A graph is not a truth machine.
A copied page can falsely claim to be an original.
A compromised ingestion worker can emit incorrect relationships.
A document can place misleading instructions beside authentic source metadata.
The provenance record itself therefore needs an evidence model.
For each relationship, distinguish:
Observed by our pipeline
Declared by the source
Inferred by a detector
Reviewed by a person
For example:
{
"subject": "revision_r17",
"predicate": "declared_primary_source",
"object": "source_original",
"basis": "publisher_metadata",
"verification_status": "unverified"
}
Do not silently promote that into:
Verified intellectual origin
This follows a broader principle in provenance modeling: the record describes relationships that can inform assessments of trustworthiness; recording a relationship does not itself establish that the underlying content is true. w3
The same applies to cryptographic hashes.
A hash can help detect a change in recorded content. It cannot establish that the unchanged content is honest.
11. Technical attribution does not settle economic fairness
One final boundary matters.
A system can:
- Cite the right author.
- Use the correct revision.
- Preserve its processing history.
- Apply corrections promptly.
- Still keep the entire user interaction inside its own interface.
That is not necessarily an attribution bug.
It is a separate question about how attention, relationships, and economic value are distributed.
Keep those outcomes separate:
Was the author named?
Was the idea correctly attributed?
Did the user recognize the author?
Did the user visit the source?
Did the creator receive any benefit?
An unclicked citation is not automatically worthless.
A clicked page is not automatically meaningful recognition.
If you want to measure creator outcomes, measure them directly rather than assuming that citation count or click-through rate represents everything.
This distinction makes the technical discussion cleanerโand the economic discussion more honest.
12. What I would build first
For a small RAG application, I would prioritize these capabilities:
- Stable source IDs and immutable revision IDs.
- Chunk records linked to source revisions.
- Exact records of the context supplied to generation.
- Answer and cache dependency records.
- Revision, access-change, and deletion events.
- Conservative invalidation of affected answers.
- Claim-level citation evaluation on a fixed test set.
- A correction-propagation integration test.
- A verified deletion workflow.
- Explicit labels for observed, declared, and inferred lineage.
You do not need to solve training-data attribution to implement these.
You do need to be honest about the boundary of what your application can observe.
The real engineering question
โDoes the answer have citations?โ is a useful UI question.
It is not a complete systems question.
The deeper questions are:
- What evidence supports each claim?
- Which exact source revisions entered this execution?
- Which relationships were observed, and which were inferred?
- What must change when a source is corrected?
- What must stop being served when access is revoked?
- What can we verify after a deletion request?
A reliable knowledge system should not merely tell us where an answer points.
It should let us inspect what the answer depends onโand act when those dependencies change.
Author: Seyed Alireza Alhosseini Almodarresieh
my site:https://almodarresieh.github.io/
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes โ full credit and traffic to the original publisher.