I Wanted Documentation to Evolve With Code: Building DocDrift

There is a small problem that is easy to overlook when working on software: the code changes, but the explanation of that code does not always change with it.

A developer renames a function parameter. Someone updates a command-line option. A default value changes after a bug fix. The Pull Request is reviewed, the code is merged, and the project moves forward. But the README may still describe the previous version.

That mismatch can be confusing for the next person who tries to use the project. They may follow an old example, pass an argument that no longer exists, or misunderstand what a function returns. The implementation has moved on, while the documentation is still describing an earlier state of the software.

This is known as documentation drift.

For our hackathon, we chose Track 1, Problem Statement 5: Documentation Drift Detector. We wanted to explore whether this problem could be checked automatically as part of the normal Pull Request workflow, instead of depending entirely on a contributor remembering to update every related document.

Our project became DocDrift: a tool designed to inspect code changes, compare the affected code with Markdown documentation, identify inconsistencies, and report them during development. The design combines Git diff analysis, Python's Abstract Syntax Tree, Markdown scanning, an open-weight language-model evaluator, deterministic checks, and a CI/CD quality gate.

We did not want the project to be only a script that searches for old words in a README. We wanted to understand how code changes could be connected to the documentation that describes them, how a mismatch could be explained with evidence, and how a developer could correct it without manually searching through the whole repository.

This is the story of how we approached that problem.

1. Why Documentation Drift? When a software project is small, updating its documentation may seem straightforward. A contributor changes a function and then edits the corresponding paragraph or parameter table.

The difficulty grows as a repository becomes larger and more people contribute to it. A single Pull Request can alter a function signature, a default value, a command-line flag, or a return annotation. The documentation connected to that code may be located in a README, an API guide, a usage example, or another Markdown file.

The code change and the documentation change are related, but they are not necessarily made in the same place or by the same person.

Consider a function that originally accepts a timeout in seconds:

1. Why Documentation Drift? When a software project is small, updating its documentation may seem straightforward. A contributor changes a function and then edits the corresponding paragraph or parameter table.

The difficulty grows as a repository becomes larger and more people contribute to it. A single Pull Request can alter a function signature, a default value, a command-line flag, or a return annotation. The documentation connected to that code may be located in a README, an API guide, a usage example, or another Markdown file.

The code change and the documentation change are related, but they are not necessarily made in the same place or by the same person.

Consider a function that originally accepts a timeout in seconds:
def connect(host, timeout_secs=30):
...
A contributor later changes the interface to use milliseconds:
def connect(host, timeout_ms=30000):
...
The implementation now expects timeout_ms, but the README might still contain a row such as:
| timeout_secs | Connection timeout in seconds |
The function and the documentation no longer agree. A reader who trusts the README may use the old parameter and encounter an error or unexpected behaviour.

This example helped us define the central responsibility of DocDrift: detect when a meaningful code change makes the corresponding documentation outdated.

We chose this problem because it sits at the intersection of software engineering and the day-to-day work of maintaining open-source projects. It is not enough for code to work on the developer's machine. Other people need accurate explanations, examples, and interfaces so that they can use and contribute to the project.

The hackathon requirements also influenced how we approached the solution. We needed a public, open-source project, a documented and reproducible example, an automated CI/CD quality pipeline, and an open-weight model component. We also had to respect the contribution workflow: after the initial hackathon.md commit, subsequent development changes were expected to go through Pull Requests rather than being pushed directly to the main branch.

That made DocDrift a useful problem to investigate because its purpose and its development workflow were closely connected. We were building a tool that checks Pull Requests while following a Pull Request-based workflow ourselves.

1. Before Building the Detector, We Wanted to Understand the Architecture Before discussing individual features, we needed a clear picture of how information would move through the system.

A Pull Request provides the starting point. It contains a set of code changes and may also contain documentation changes. DocDrift needs to identify the relevant changes, understand the affected code structures, find the associated documentation, evaluate whether the two still agree, and produce a useful result.

Our architecture is organised around five main stages:

1. GitHub Pull Request and Git diff extraction.
2. Python AST diff analysis.
3. Markdown documentation scanning.
4. Open-weight AI evaluation with validation and a deterministic fallback.
5. CI/CD audit reporting and quality-gate behaviour. The intended flow can be represented as follows:

The architecture has two information paths before evaluation. The AST parser extracts structured information from the source code. The Markdown scanner extracts the documentation that may describe that code. The evaluator then receives the relevant information and determines whether there is a discrepancy.

This separation is important. The code parser and the Markdown scanner have different responsibilities. The parser should not have to understand the layout of a README, and the Markdown scanner should not have to interpret Python syntax. Keeping those tasks separate makes the overall process easier to reason about and test.
Creating an architecture illustration
We also wanted the architecture to be readable in the technical material we prepared. The following screenshot is from our Python ReportLab document-generation code. It shows the imports and the beginning of the PDF creation function. This code prepares the document layout; it is not part of the DocDrift detection engine itself.

The code imports SimpleDocTemplate to create the document, Paragraph for formatted text, Table and TableStyle for structured layouts, Preformatted for content that must preserve spacing, and styling helpers for typography. The letter page size is selected, and the four margins are set to 36 points each. Since 72 points make one inch, each margin is half an inch.

The function accepts output_path: str, indicating where the generated PDF should be saved. It then creates a SimpleDocTemplate using that path and loads ReportLab's sample stylesheet. The rest of the PDF-building process is outside the visible portion of this screenshot, so this excerpt should be understood as the document setup rather than the complete PDF-generation function.

The next screenshot shows how we prepared the architecture section itself. The ASCII diagram is stored in a string, wrapped in a Preformatted object, placed inside a one-cell table, and styled with a background, padding, and border.

A few details are worth understanding. story.append(Paragraph(...)) adds the section heading to the document's story. The arch_ascii variable stores the diagram, while \n creates line breaks. Preformatted(arch_ascii, code_style) preserves the layout of the diagram instead of treating it as ordinary paragraph text. Table places that block inside a cell with a width of 540 points. TableStyle applies the background, eight-point padding, and border. Finally, story.append(arch_table) adds the diagram to the document, and Spacer(1, 10) adds vertical space after it.

This was a useful reminder that technical communication is also part of engineering. A diagram can make a multi-stage workflow easier to understand, but only when its layout remains readable and its labels match the actual implementation.

1. Moving From a Pull Request to Structured Code Changes The first technical problem is deciding what DocDrift should inspect.

A repository may contain many files and functions, but a Pull Request usually changes only a portion of them. We wanted the detector to focus on the changes made by the contributor instead of treating every existing line in the repository as a new problem.

That is the purpose of the PR diff mode, exposed through the --diff workflow described for DocDrift. Git diffs show additions and removals between versions of a file. They provide the starting evidence for identifying which source files and line ranges were modified.

A simplified change might look like this:

The diff tells us that the function definition changed. However, a raw diff is still text. It does not, by itself, explain the structure of the Python function or identify every parameter as a distinct object.

For that reason, DocDrift uses Python's Abstract Syntax Tree, commonly called the AST. Python can parse source code into a structured representation. From a function definition, an AST-based parser can inspect information such as the function name, arguments, default expressions, and return annotation.

The project description identifies the AST parser as the component responsible for extracting changed symbols, function parameters, CLI flags, and type information. The important design idea is to combine the changed-line information from Git with structured information from the source code, so the detector can focus its analysis on the relevant code.

This creates an engineering detail that matters: Git diff hunks and AST nodes are not the same representation. A diff identifies changed text ranges; the AST represents parsed source structures. The implementation must map the changed ranges to the relevant functions or symbols. That mapping is what allows a change-scoped analysis instead of an unrestricted scan.

The AST approach is also useful because it is more structured than searching for parameter names with plain text matching. A name may occur in comments, strings, examples, or unrelated functions. Parsing the source gives the detector a way to reason about actual Python definitions.

1. Reading the Documentation Alongside the Code After extracting information from the source, DocDrift needs to find the documentation that describes it.

A Markdown file can contain headings, paragraphs, code examples, lists, and parameter tables. The same function may be documented in a README or in a separate guide. The Markdown scanner is intended to locate relevant sections and extract details such as parameter names, default values, headings, and line locations.

For example, a function might be documented like this:

If the code now defines horizon=30, the documentation contains an outdated default. If the code has renamed or removed a parameter, the old table entry may also need attention.

The scanner's role is not simply to read the whole README and send it to the model. It is to identify the relevant documentation content so that the comparison has useful context. Preserving line numbers is especially important because a report that points to README.md:82 is more actionable than a message that only says that the README is outdated.

The parser and scanner therefore produce two different views of the same feature: what the source code currently defines and what the documentation tells a reader.

1. The Types of Documentation Drift We Wanted to Detect We organised the detector around several concrete forms of mismatch. Each one represents a different way in which documentation can stop matching the implementation. Parameter mismatch A parameter mismatch occurs when a function argument is renamed, removed, or otherwise changed in the source code while the documentation continues to mention the old name.

• timeout_secs
• timeout_ms

If a Markdown table still lists timeout_secs, DocDrift can report a PARAM_MISMATCH for the affected symbol. This is useful because function parameters are part of the interface that users and other developers depend on.
CLI flag mismatch
Command-line tools have their own public interfaces. A developer might rename an option from --timeout to --deadline, while the README still instructs readers to use the old flag.

The project description specifies extraction of command-line options declared with argparse's add_argument() and Click's @click.option decorator. The detector can then compare those declarations with the command examples or descriptions in the documentation.

A stale --timeout reference can be reported as a CLI_FLAG_MISMATCH. This matters because documentation drift is not limited to Python function signatures; command-line options are also interfaces.
Default-value mismatch
A default-value mismatch occurs when the implementation changes a default but the documentation continues to show the previous value.

def simulate(history, horizon=30):
...
If the README says that horizon defaults to 60, the two sources disagree. DocDrift is intended to report this as a DEFAULT_VALUE_MISMATCH and show both values.
Return-type mismatch
A function's return annotation communicates what callers should expect. For example:
def is_admin(user_id) -> bool:
...
If the documentation says that the function returns a string, the documented type and the annotation conflict. The project description includes generalized return-type checking for types such as bool, int, str, dict, list, tuple, and DataFrame.

The exact coverage depends on how the implementation extracts annotations and interprets documented type descriptions. We should distinguish annotation-based checks from runtime type inference: a return annotation describes the declared interface, but does not automatically prove what every execution path returns.
Exact location reporting
Detection is more useful when the result tells a contributor where to look. DocDrift is designed to report the source-code location, the documentation location, the relevant code change, and the exact outdated sentence where available.

A report can therefore communicate more than a generic warning:

The value of this report is its traceability. A maintainer can see which symbol was affected, where the source changed, where the documentation is stale, and which text needs review.

1. Why We Chose a Hybrid AI and Deterministic Approach One of the most interesting design questions was deciding how much of the comparison should be handled by programmatic checks and how much should be handled by a language model.

A language model can help interpret whether a sentence still describes a changed function or feature. But asking a model to inspect an entire repository without structured context would be inefficient and could produce unsupported conclusions.

At the same time, deterministic parsing is well suited to facts that can be extracted directly from code, such as declared parameter names, default expressions, and return annotations.

DocDrift's described architecture combines these approaches. The AST parser and other deterministic checks provide structured code information. The Markdown scanner provides relevant documentation. The open-weight AI evaluator is used for contextual comparison, with structured output and validation intended to make its results usable by the rest of the system.

The project overview identifies Gemma 2 and Llama 3 as the open-weight model options, with Ollama as the local model runtime. The configuration described for the evaluator includes a low temperature of 0.1, a 30-second timeout, JSON validation, enum validation, and retries for invalid model responses.

Each setting has a purpose:

• A low temperature reduces variation in generated responses, although it does not guarantee perfect determinism.
• A timeout prevents a model request from waiting indefinitely.
• Structured JSON gives the program a predictable response format to parse.
• Enum validation restricts categorical values to the labels the application expects.
• Retries provide another attempt when a response cannot be parsed or validated.
• The deterministic AST fallback is intended to preserve supported checks when the local model service is unavailable. The fallback is not necessarily identical to the semantic evaluator. It should be described according to the exact checks it can perform. A structured parameter comparison can be deterministic; understanding whether a paragraph still explains a feature correctly may require contextual evaluation.

This hybrid approach reflects an important principle we learned while working on the project: the model should assist the analysis, but the application still needs clear rules, validation, and evidence around its output.

1. Testing the Case Where Nothing Is Wrong A detector should not only find problems. It should also avoid reporting a problem when the code and documentation are already consistent.

We included a clean-code test for this purpose. The test defines a simple Python function named greet, creates documentation for that function, parses the source, scans the Markdown, and passes the extracted information to the drift evaluator.

Figure 7.1 — The test_clean_code_no_drift() test fixture, showing matching source code and Markdown documentation before invoking the evaluator.

The visible test can be understood step by step.

First, it declares the test function:
def test_clean_code_no_drift():
The name communicates the scenario being checked: clean code with no documentation drift.

Next, it defines the source code as a multiline string:
code = """
def greet(name: str) -> str:
return f"Hello, {name}"
"""
The example function accepts a name parameter annotated as str and has a return annotation of str. Its body returns a formatted greeting.

The test then defines a Markdown string:
doc = """

greet()

| Parameter | Description |
| name | Person name |
"""
This documentation names the same function and includes the same parameter, name. The fixture is deliberately simple so that the test can focus on the no-drift condition.

The next lines import the parser and scanner:
from docdrift.ast_parser import ASTParser
from docdrift.markdown_scanner import MarkdownScanner
ASTParser is responsible for extracting structured information from the Python source. MarkdownScanner is responsible for extracting sections from the documentation.

The test creates instances of these two components:
parser = ASTParser()
scanner = MarkdownScanner()
Then it extracts the code symbols and documentation sections:
symbols = {s.name: s for s in parser.parse_source(code)}
sections = scanner.scan_content(doc)
The dictionary comprehension uses each extracted symbol's name as a key. In this example, the intended key is greet. The scanner returns the sections it finds in the Markdown content.

The test then creates the drift detector and evaluates the extracted function against the first documentation section:
detector = DriftDetector()
issue = detector.llm_evaluator.evaluate_drift(
symbols["greet"],
sections[0],
{}
)
The evaluator receives the greet symbol, the first scanned documentation section, and an empty dictionary as its third argument. The screenshot does not show the evaluator's method definition, so the exact purpose of that third argument must be explained from the implementation rather than guessed.

Finally, the test contains its key assertion:
assert issue is None
This asserts that the evaluator should return no issue for this clean example. If the test passes, it demonstrates that this particular matching source-and-documentation fixture did not produce a drift issue under the test conditions.

This is a useful negative test. It helps check that the detector does not raise a warning every time it sees a function and a Markdown section. A reliable detector needs both sides of the problem: examples where drift is present and examples where it is absent.

The screenshot shows the test code, not the terminal result of running it. For that reason, the code establishes the intended assertion, while a separate test-run screenshot or log is needed to establish that it passed in the project's environment.

1. From Detecting Drift to Helping Repair It Finding an outdated sentence is useful, but it still leaves the developer with the work of correcting it. We therefore included a repair-oriented workflow in the project description.

For local development, the CLI exposes a fix command. The supplied invocation is:
python3 -m docdrift.cli fix --code client.py --docs README.md
The intended sequence is straightforward. DocDrift reads the source, extracts the current function information, scans the Markdown file, locates an obsolete entry, prepares a replacement, and writes the corrected documentation. The developer can then inspect the change with Git:
git diff -- README.md
Reviewing the diff is important. An automatic edit should still be visible and reviewable before it is committed. A maintainer should be able to confirm that the replacement is accurate and that surrounding documentation has not been damaged.

The project description also includes a GitHub Pull Request suggestion workflow. GitHub supports suggestion blocks in review comments, allowing a maintainer to apply a proposed replacement through the Pull Request interface. In the DocDrift scenario, the old Markdown line can be shown alongside the proposed corrected line, and the maintainer can choose to commit the suggestion.

This creates two repair paths: one for a developer working locally and another for a maintainer reviewing a Pull Request. The actual GitHub integration should be demonstrated with a real suggestion and a resulting commit; a normal text comment that merely contains proposed Markdown is not the same as a native, applicable GitHub suggestion.

1. Making Documentation Checks Part of CI/CD A tool that works only when someone remembers to run it manually cannot reliably protect every Pull Request. We wanted documentation validation to become part of the automated review process.

DocDrift is designed to run through GitHub Actions. When a relevant Pull Request event triggers the workflow, the action runs the detector against the proposed changes. The result determines whether the check succeeds or fails.

The CLI behaviour described for the project uses a successful exit code when the audit passes and exit code 1 when breaking drift is detected. A failing workflow check can prevent a Pull Request from being merged when the repository rules require that status check to pass. The required-check configuration is important: a failed job by itself does not automatically block every repository's merge button unless branch protection or a ruleset requires it.

The project also includes a reusable GitHub Action through action.yml, intended to make adoption easier for other repositories. Instead of asking every maintainer to recreate the workflow, the action provides a reusable integration point with the project's documented inputs and behaviour.

This was important to the original problem. Documentation drift is a maintenance issue that can appear in any Pull Request. The goal is to catch it at the point where code changes are reviewed, before the outdated explanation becomes part of the accepted project state.

The CI environment also raises a practical deployment question for a locally hosted model. A GitHub-hosted runner does not automatically share the localhost of a developer's laptop. The model must be available in the runner environment, through a configured remote service, through a self-hosted runner, or the workflow must use the supported deterministic fallback. The exact route used by the final workflow should be documented from its configuration.

1. Evaluating DocDrift Against Historical Changes A detector should be evaluated against examples that represent both the problem it is supposed to catch and changes that should not trigger warnings.

The benchmark described for DocDrift uses historical changes from four repositories:
Repository Intended drift scenario
psf/requests Parameter drift
pallets/flask CLI flag drift
tiangolo/fastapi Return-type drift
rudras2048-arch/medicoool Teammate Pull Request #7 drift

The benchmark also includes four clean control Pull Requests covering examples such as internal refactoring, comment changes, and typo fixes. These controls are important because a detector that flags every code modification would be noisy and difficult to trust.

The results supplied for this selected benchmark were:
Metric Reported result
Drift cases detected 4 out of 4
Recall 100.0%
Precision 100.0%
False alarms on clean controls 0 out of 4
False-alarm rate on clean controls 0.0%

Under the stated eight-case setup, the corresponding counts are four true positives, zero false negatives, zero false positives, and four true negatives. The metric calculations are:
Recall = TP / (TP + FN) = 4 / (4 + 0) = 100%

Precision = TP / (TP + FP) = 4 / (4 + 0) = 100%

False-alarm rate = FP / (FP + TN) = 0 / (0 + 4) = 0%
These results are encouraging for a proof-of-concept evaluation, but the sample is small. They describe performance on the selected historical cases; they do not establish universal accuracy across all languages, repositories, documentation formats, or kinds of code change. A larger benchmark with more projects, more drift categories, ambiguous examples, and independent labels would provide stronger evidence about generalisation.

The benchmark also needs to be reproducible. For each historical case, the project should record the repository, commit or Pull Request reference, expected outcome, actual DocDrift output, and the method used to label the case. That makes the result inspectable rather than just a number in a presentation.

1. What We Learned Beyond the Code The work on DocDrift brought together several concepts that are easy to study separately but more interesting when they have to operate as one system. A diff is not the same thing as an understanding of code Git can show what text changed. An AST can represent the structure of Python source. Connecting the two requires a deliberate mapping. This helped us think more carefully about the difference between changed lines and changed program elements. Documentation is not just plain text A Markdown file has headings, tables, paragraphs, examples, and relationships between sections. Finding the right documentation for a function is a separate problem from extracting the function's parameters. The scanner and parser therefore need clearly defined responsibilities. AI needs boundaries and verification A language model can help interpret context, but its output still needs a schema, validation, error handling, and evidence. We learned to treat the model as one component in a larger system rather than as the entire system. A good detector must also know when not to warn The clean-code test illustrates this. If the source and documentation agree, the expected result is no issue. Positive detection tests and negative tests are both necessary to understand the behaviour of a quality-checking tool. A useful result should lead to an action A warning with no location leaves the maintainer searching. A report that identifies the symbol, source line, documentation line, stale sentence, and suggested correction is much easier to investigate. The local fixer and GitHub suggestion workflow extend that idea from detection toward repair. Automation belongs in the normal development workflow The CI/CD quality gate is intended to make documentation checks repeatable. It changes the process from “remember to check the README” to “the Pull Request receives an automated documentation check.”

The larger lesson was that the project was not only about comparing two files. It was about building a chain of evidence from a source change to a documentation decision and then making that decision useful to a contributor.

1. What We Would Explore Next The current design provides a foundation, but there are several areas that would benefit from broader evaluation and further development.

First, we would expand the historical benchmark. Eight selected cases are useful for demonstrating the idea, but a larger dataset would help measure how the detector behaves across different repositories, coding styles, documentation structures, and ambiguous changes.

Second, we would expand the range of source and documentation patterns. Python AST analysis provides a structured starting point for Python code. Other languages would require their own parsers or language-specific analysis components. Markdown support could also be tested against nested headings, multi-line tables, examples, links, and documentation spread across several files.

Third, we would evaluate the relationship between deterministic checks and model-based evaluation more systematically. We would measure which categories can be resolved reliably with structural comparisons, which benefit from semantic reasoning, and how often the model requires retries or fallback handling.

Fourth, we would test the CI/CD integration in different environments. A locally hosted Ollama model and a GitHub-hosted runner are separate machines. The deployment strategy, runtime, permissions, timeout, and offline behaviour should all be tested as part of the workflow rather than assumed.

Finally, we would continue improving the repair process. Automatically generated changes should preserve useful descriptions, formatting, and surrounding context. A correction should be easy to review, and maintainers should remain in control of what gets committed.

These improvements would move the project from a focused hackathon implementation toward a more extensively evaluated tool for repository maintenance.
Conclusion: Documentation Should Evolve With Code
We began with a simple observation: a code change can be complete while the documentation describing it remains outdated.

That observation led us to Problem Statement 5 and to the idea behind DocDrift. We designed a workflow that begins with Pull Request changes, uses Git diff information to focus the analysis, extracts structured information from Python source with an AST parser, scans Markdown documentation, and evaluates whether the two still agree.

We brought deterministic checks together with an open-weight AI evaluation approach, added validation and fallback considerations, and designed reports that point to the code and documentation involved. We also developed the idea beyond detection, with a local repair command, GitHub review suggestions, and a CI/CD quality gate intended to make documentation validation part of the regular contribution process.

The clean-code test reminded us that a detector must also recognise when there is no problem. The historical benchmark gave us an initial way to evaluate both drift cases and clean controls. The ReportLab work helped us organise the architecture and technical explanation into a readable document.

DocDrift is still a focused engineering project, not a claim that every documentation problem can be solved automatically. Its value lies in making a neglected maintenance task more visible, structured, and testable.

The question that started the project remains useful: when code changes, why should its documentation stay behind?

For us, the answer was to make documentation consistency part of the development workflow itself.
Project Evidence
The project work described in this article includes the following components and evidence areas:

• Problem statement: Track 1, Problem Statement 5 — Documentation Drift Detector.
• Project: DocDrift v0.2.0.
• Team: BIG 4 — Krrish Raj (Team Lead), Kushal Singla, Saksham Bist, and Rudra Sharma.
• Code analysis: Git diff-based change identification and Python AST analysis.
• Documentation analysis: Markdown scanning for headings, parameter information, defaults, and line locations.
• Drift categories: Parameter mismatch, CLI flag mismatch, default-value mismatch, and return-type mismatch.
• AI evaluation: Open-weight Gemma 2 / Llama 3 model options through the configured Ollama workflow, with structured output and validation as described in the project design.
• Reliability: Timeout, retry, validation, and deterministic AST fallback mechanisms as described for the project.
• Reporting: Severity, source location, documentation location, quoted stale text, and suggested corrections.
• Repair workflow: Local CLI fixing and GitHub Pull Request suggestion workflow.
• CI/CD: GitHub Actions integration and reusable action.yml workflow component.
• Evaluation: Four historical drift cases and four clean controls, with the reported results stated in Section 10.
• Testing example: test_clean_code_no_drift() using ASTParser, MarkdownScanner, and the drift evaluator.
• Technical document preparation: ReportLab code for PDF page setup and architecture-diagram layout. Scope note: The benchmark figures in this article refer to the selected cases described in the project brief. The clean-code screenshot shows the test definition and assertion, not a test-run log. The ReportLab screenshots show document-generation code, not DocDrift's detection engine. Before publication, attach the corresponding repository files, actual test output, CI run links, model configuration, and benchmark records wherever a claim needs independently inspectable evidence.