I Went Looking for the Dead Internet in My RAG. I Found a Bad Experiment.

I removed duplicate sources from a RAG benchmark and watched held-out accuracy fall from 10 of 20 tasks to 5 of 20. The tempting conclusion was that source deduplication made retrieval worse.

The audit found another change hiding in the comparison. Lexical top-k supplied roughly five documents; the collapse policy often supplied one or two. The run changed source diversity and the evidence budget together, so the accuracy drop could not identify a provenance effect.

With a fixed-budget control, root-aware retrieval recovered the primary record and accuracy returned to 10 of 20. It did not improve further. That correction became the point of this investigation: retrieval correctness and answer correctness are different measurements, and an apparent result can disappear when the control is fair.

TL;DR

• Removing duplicate sources appeared to halve accuracy, but the comparison also reduced the evidence budget.
• Under the corrected control, root-aware retrieval recovered primary records without improving answer accuracy.
• A 15-case real-world pilot validated the provenance pipeline, not retrieval effectiveness. V2 will test adversarial cases.

Part 1 corrects a misleading synthetic result and validates a provenance-capture and evaluation pipeline on real sources. Part 2 will test whether provenance-aware retrieval changes answers when the evidence actually conflicts.

The part of the "dead internet" idea worth testing

The phrase "dead internet" is associated with a 2021 forum post, Dead Internet Theory: Most of the Internet Is Fake. Contemporary reporting traces the popular term to that thread. Its broad version bundles together bots, manufactured engagement, algorithms, and a claim that real human activity has become marginal.

That claim is too broad for this experiment. A traffic statistic cannot establish who wrote a page, whether a claim is true, or whether people have stopped participating online.

For RAG, a systems problem remains. Imperva's 2025 report estimated that automation represented 51% of the web traffic it observed in 2024. The category includes useful crawlers and malicious bots, so it cannot count a "fake internet."

For retrieval, a smaller question is enough: how many independent source roots support the evidence in context?

Repeated documents can reach an index through a documented syndication chain, a press release, a scrape, a summary, or a generated rewrite. The causes differ. The failure is the same when repetition reaches the model as apparent corroboration.

The wider theory also tends to mix four questions:

• Did a human or an automated client make the request?
• Was the page written by a person, copied, or generated?
• Where did the claim originate?
• Does the claim have enough support to warrant an answer?

They require different evidence. This experiment addresses the third question, provenance. It does not diagnose the web as a whole.

What I mean by a source root

A source root is an information lineage, not a hostname. One root can include a public primary record and several documented syndications. One publisher can contain several independent roots.

I assign a root only when the relationship has evidence: a canonical link, a syndication notice, an explicit citation, or a documented editorial relationship. Near-identical phrasing is useful for discovery, but it does not prove copying. Treating a similarity score as provenance would create a confident-looking field from an unverified assumption.

The system

The implementation keeps evidence capture, retrieval, generation, and scoring separate. Otherwise one provenance_aware switch can quietly change the candidate pool, context length, answer rule, and source metadata at once.

Evidence capture saves the public and canonical URLs, timestamp, HTTP status, extracted text, hash, and documented relation. Retrieval selects documents under a defined budget. Generation receives document text, IDs, roots, and roles, but never an expected stance or fixture label. Evaluation checks the returned stance, citation IDs, and retrieval trace.

candidates = lexical_retrieve(question, pool=50)
context = root_aware_fill(candidates, k=5)
response = generate(question, context, fixed_prompt, seed)
audit(response, context, expected_stance)

The pool=50 is important. Collapsing sources only after a top-five cutoff cannot recover a primary record that was ranked sixth. Candidate grouping has to happen before the context budget is spent.

The result that fooled me

I built a 60-task synthetic fixture with known source roots. The first comparison looked clean: lexical top-k got 10 of 20 held-out tasks right; a root-collapse policy got 5 of 20.

The comparison was not clean. Root collapse sometimes returned one or two documents where lexical top-k returned five. Both variables changed, so the first run cannot attribute its regression to source grouping alone.

The original collapse policy spent fewer evidence slots. Root-aware fill chooses roots from a larger pool, then restores the intended document budget.

With the budget fixed, root-aware fill recovered the primary record and retained the deterministic accuracy of lexical top-k. It did not improve accuracy. The remaining slots still let the answer rule count repeated stances.

The original drop measured a policy that also shrank the prompt. It did not isolate the effect of source grouping.

Why keep the synthetic fixture?

The fixture isolates a retrieval mechanism that is hard to control on public pages. It contains 60 tasks across duplicate dilution, true repetition, conflicting roots, and satire stripped of its context. Each category has ten development tasks and five held-out tasks.

The publishers and documents are fictional because the fixture needs declared roots and known answers without claiming to measure the open web. The first experiment uses a deterministic stance aggregator, making the intervention inspectable. It also exposes a limitation: recovering a primary record does not automatically improve the final answer if the aggregator still counts every retained document equally.

A concrete held-out fixture trace. It demonstrates a retrieval change, not a real-world effectiveness result.

Fixtures cannot show whether the pipeline works on public pages, so I collected a separate real-data pilot.

The 15-case real pilot

The pilot contains 15 NASA-to-Phys.org syndication chains. Every accepted case has:

• a public NASA record;
• a Phys.org page whose snapshot explicitly says Provided by NASA;
• timestamped HTML, extracted text, a canonical URL, HTTP status, and a SHA-256 content hash;
• a graph edge supported by the documented syndication marker.

Similar language never counted as proof of dependence. If the source relationship was undocumented, the case did not enter the slice.

Gemma 3 4B and Qwen 2.5 7B independently labelled document roles. They disagreed enough that I did not use their votes as ground truth. Final roles came from the explicit record type and syndication marker; model annotations remain in the corpus as machine annotations. This is a machine-adjudicated pilot, not a human-verified dataset.

I ran both models on the same 15 questions under four conditions:

1. primary-only;
2. primary plus the documented dependent copy;
3. root-aware;
4. conflict-preserving root-aware.

The prompt, seed, decoding settings, evidence ordering, retrieval trace, and raw responses were saved for every run. Neither model received the expected answer or a stance label in its prompt.

What V1 actually found

15 source chains · 2 local models · 120 retained responses

Gemma 3 4B and Qwen 2.5 7B each returned the correct stance for all 15 cases in every condition. Citation-ID validity changed when the dependent copy was present. The slice has one root and positive support in every case, so it validates provenance capture and evaluation plumbing. It does not test whether root-aware retrieval prevents a real answer error.

Both models answered all 15 questions correctly in every condition.

Every accepted chain has one source root and supports one positive claim, so root-aware retrieval reduces to primary-only. The slice has no false assertion or contradictory root to resolve.

The citation trace moved:

Those are exploratory counts, not a model ranking. Citation-ID validity only says whether the returned ID was present in supplied context. It does not establish that the cited document supports the answer.

The V1 confound, stated plainly

There is a second limit on the V1 comparison. Adding the dependent page changes the context from one document to two, and Ollama's recorded prompt-token count changes in every model-case pair.

The citation results are descriptive because the comparison cannot isolate source dependence from the extra context. V2 will block an evaluation unless each paired case has equal document and prompt-token budgets.

The real corpus also has a ceiling effect. Primary-reference recall is 100% by construction in the one-root conditions, and root concentration is 1.0 because every selected context belongs to one lineage. These measurements confirm that the policy did what it was asked to do. They also explain why the pilot cannot support an accuracy claim.

Metrics worth keeping separate

Correctness is useful, but it can hide the failure that changed. A model may preserve a correct stance while losing the primary record, naming a citation ID that was never in context, or becoming impossible to audit after a page changes.

I would report at least these outcomes for a larger corpus:

• answer stance;
• primary-reference recall;
• root concentration;
• citation validity;
• contradiction coverage;
• false-claim adoption.

A single combined score would hide which part failed. The unit for a future comparison is the source case, not the generated response. Paired accuracy deltas, bootstrap intervals, and an exact sign test make sense only after enough independent held-out cases survive the admission gate.

For production, I would retain this trace beside every retrieved chunk:

document_id · canonical_url · publisher · publication_date
retrieved_at · content_hash · source_root · source_role
claim_relation · dependency_evidence

Candidate selection, source weighting, and conflict handling should stay separate. A source root can come from a documented origin or explicit citation evidence. It cannot come from cosine similarity alone.

What a reader can verify today

The public release supports artifact verification, not a byte-for-byte end-to-end rerun. It contains public URLs, hashes, annotations, the run manifest, the paired-budget audit, and 120 raw model responses. It deliberately excludes third-party HTML and extracted page text.

git clone https://github.com/icesurf666/rag-provenance-lab.git
cd rag-provenance-lab
python3 -m venv .venv && . .venv/bin/activate
pip install -e '.[test]' && PYTHONPATH=src pytest -q

The reproducibility guide explains that boundary. The paired-budget audit records all 30 unequal model-case pairs; the raw responses, run manifest, and recorded results are public.

A real retrieval failure outside this pilot

In May 2024, Google AI Overviews advised users to add glue to pizza sauce. Associated Press reported that the answer traced back to a satirical Reddit comment, and Google later published its response.

The incident shows a retrieval failure: the words reached the system while the sentence that explained their status did not. It cannot estimate a dead web or the prevalence of AI-generated content.

What comes next: V2 — when provenance changes the answer

V1 is complete as a pipeline validation. V2 will test the effectiveness question it leaves open: can source provenance prevent a model from accepting a repeated false claim when a correct primary record is available?

V1 and V2 answer different questions. This diagram describes a planned V2 protocol, not results.

The next corpus will include corrections alongside persistent stale copies, satire retold as fact, contradictory evidence, genuinely independent corroboration, and conflicts within one source lineage. The target is 40 to 60 quality-gated real cases with documented provenance, snapshots, hashes, independent machine annotations, and adjudication.

Every case that passes the gate will stay in the evaluation set. V2 will include cases where both policies are correct, both are wrong, lexical retrieval wins, and root-aware retrieval wins. Selecting only later success stories would turn the corpus into a demonstration instead of an experiment.

Gemma and Qwen will run pre-specified paired conditions with equal document and prompt-token budgets. Document-order permutations will be assigned before outcomes are inspected. The result may show an improvement, a regression, or no material effect.

Until V2 exists, a claim that root-aware retrieval improves real-world answer accuracy would be premature.

The RAG Provenance Lab repository contains the public V1 code, artifact-verification materials, model outputs, run manifests, and the protocol. It distinguishes artifact verification from end-to-end reproduction: third-party page snapshots and extracted text are not distributed in the public release.