description: "How we built an offline tool, GitHub Action, and web dashboard that catches outdated documentation before a Pull Request gets merged."

Every year during Hacktoberfest, developers spend hours exploring open source repositories. You find a cool project, read the setup guide, copy a command into your terminal, and immediately hit an error:

bash
error: unrecognized arguments: --timeout-secs

You check git blame and find out someone renamed --timeout-secs to --timeout in the code four pull requests ago, but forgot to update the documentation.

Code changes constantly, but documentation is usually updated by hand. When code and documentation don't match anymore, it creates a roadblock for new contributors and extra work for maintainers.

At Hacktoberfest Hack Day at Rishihood University, our team wanted to solve this problem. We built Docs-Drive, a tool that checks code changes against documentation automatically and tells you exactly what needs to be fixed.

The Problem with Just Using an LLM

When people first think about solving this problem, they often suggest throwing the whole code diff and the entire documentation folder into an AI model. In practice, that approach runs into three big issues:

1. Too Much Noise: A normal code cleanup or internal rename will trigger dozens of false alarms.
2. Speed and Cost: Reading hundreds of markdown files on every single code push is slow and expensive.
3. Missing API Keys: Many open source contributors work on forks where cloud API keys (like OpenAI or Anthropic) are not available in CI secrets.

To fix this, we designed Docs-Drive to work locally and offline first, using simple text matching and structural code analysis before involving any AI.

How Docs-Drive Works (Step by Step)

Our system runs across four main parts. Here is how code flows from a pull request down to a clean error report.

1. Reading Code Changes and Guarding Against False Alarms

The system reads unified diffs from local git repositories, patch files, or GitHub pull requests. It automatically ignores build artifacts, lock files, and binary files.

It also includes a PR Doc-Update Guard: if a developer changes a command name and updates the documentation in the same pull request, the tool notices that the docs are already fixed and suppresses any warnings.

1. Extracting Code Changes using AST (Abstract Syntax Trees)

Instead of treating code as plain text, the engine parses Python and TypeScript code structurally to spot real changes:

Renamed options or flags: For example, changing --timeout-secs to --timeout.
Changed default values: For example, changing a default timeout from 30 to 60.
Modified parameters: Adding, removing, or changing function arguments.
Removed features: Deleting functions or endpoints that are still mentioned in guides.

Internal code cleanups that don't affect public behavior are automatically ignored.

1. Parsing Documentation into Sentences

The documentation parser breaks Markdown (.md, .mdx) and reStructuredText (.rst) files down into individual sentences. It keeps track of the exact line numbers, heading titles, and table structures so it can point right back to the exact sentence that needs fixing.

1. Matching Code Changes to Documentation

To figure out which documentation sentence relates to a specific code change, the engine uses a mix of:

Exact symbol matching: Looking for direct mentions of variable or flag names.
Keyword search (BM25): Matching technical terms.
Local embeddings: Finding conceptual links using local embedding models or a simple offline fallback when no local AI is running.

Turning it into a GitHub Action and Web Dashboard

A local script is helpful, but open source happens on GitHub. We wrapped our core engine into a complete product:

GitHub Action: Runs automatically on pull requests. Instead of spamming a pull request with 10 separate comments, it posts a single summary table and updates it cleanly using an invisible HTML signature (<!-- docs-drift-bot-summary -->). It also leaves native GitHub suggestion blocks so maintainers can click one button to apply the fix.
FastAPI Backend & Next.js Dashboard: We built a full-stack web interface where you can view side-by-side comparisons of code diffs and outdated documentation sentences.
AI Fix Generator: If you want help rewriting a sentence, you can click a button to generate a clean fix, paragraph update, or a downloadable.patch` file.

We also designed the backend to run smoothly on serverless environments like Vercel by switching to synchronous execution when background workers aren't available.

Testing and Results

We built a historical testing harness that replays real-world code changes and subsequent documentation fixes. Across our benchmark dataset, the tool achieved 1.00 Precision, 1.00 Recall, and a 1.00 F1 score, successfully catching true documentation drift without false alarms on internal refactors.

Backing all of this up, our entire test suite currently sits at 75/75 passing unit tests:

bash
.venv/bin/python3 -m pytest tests/ -v
======================== 75 passed in 1.42s========================

Final Thoughts

Open source thrives when projects are easy to understand and contribute to. Documentation is the first handshake a new developer gets when visiting a repository. By automating how we keep code and text aligned, we can save maintainers hours of review time and keep documentation accurate automatically.

If you are exploring open source this month, check out how your favorite projects handle documentation—and happy Hacktoberfest! 🚀

HacktoberfestAtRishihood #NewtonSchoolOfTechnology #OpenSource #Hacktoberfest

Written by Sahaj Choukhani (Team HisenBugs) for Hacktoberfest Hack Day at Rishihood University