How to convert PDFs to Markdown for RAG with Python (tables, columns and scanned pages)
Disclosure: I built the Apify actor used in the second half of this post. It is a paid tool (pay per PDF and page). The first half is free and uses only open-source Python libraries. If you feed PDFs to an LLM, the qual
Disclosure: I built the Apify actor used in the second half of this post. It is a paid tool (pay per PDF and page). The first half is free and uses only open-source Python libraries.
If you feed PDFs to an LLM, the quality of the text you extract decides the quality of the answers. A retrieval pipeline that splits a sentence across two columns, or puts the same running header into every chunk, will retrieve the wrong things no matter how good the model is.
This post shows the simple way to extract PDF text with Python, the problems you will hit with real documents, and a way to get structured Markdown with page-cited chunks.
The simple way: pdfplumber
pdfplumber (MIT licence) reads the text layer of a PDF and can find tables drawn with lines:
import io
import pdfplumber
import requests
url = "https://www.irs.gov/pub/irs-pdf/i1040gi.pdf"
pdf_bytes = requests.get(url, timeout=120).content
with pdfplumber.open(io.BytesIO(pdf_bytes)) as pdf:
print(len(pdf.pages), "pages")
page = pdf.pages[8] # page 9
print(page.extract_text()[:600])
for table in page.extract_tables():
print(table[:3])
For a single-column page this is often enough. Ruled tables come back as lists of rows, which you can turn into Markdown in a few lines:
def to_markdown_table(rows):
rows = [[(c or "").replace("\n", " ").strip() for c in row] for row in rows]
lines = ["| " + " | ".join(rows[0]) + " |", "|" + " --- |" * len(rows[0])]
return "\n".join(lines + ["| " + " | ".join(r) + " |" for r in rows[1:]])
with pdfplumber.open(io.BytesIO(pdf_bytes)) as pdf:
print(to_markdown_table(pdf.pages[8].extract_tables()[0]))
What goes wrong with real documents
Run extract_text() on a research paper or a government booklet and you will meet these problems:
- Columns get interleaved. Two- and three-column pages come out line by line across the whole page width, so each "line" joins half a sentence from the left column with half from the right.
- No structure. Headings, lists and paragraphs are flattened into lines. Your chunker cannot split at section boundaries, and an LLM cannot tell a heading from body text.
- Repeated noise. Running headers, footers and page numbers appear on every page, and stamps in the margin (arXiv prints one sideways on every paper) are spliced into the text as single letters.
- Broken words. "repre-" at the end of one line and "sentation" at the start of the next stay apart. Font changes inside a word ("Bidirectional") can split it into "B idirectional".
- Scanned pages are empty. A page with no text layer returns nothing at all; you need OCR.
- Figures pretend to be tables. Diagrams drawn with lines are often detected as tables full of label fragments.
Fixing these means working with each word's position, font size and weight: find column gutters, group words into lines and paragraphs, detect headings from font size and section numbering, and drop text that repeats in the same margin on many pages. It is doable, but it is a few hundred lines of heuristics plus testing on many layouts. Open-source projects such as Docling and marker take this on with machine-learning models, at the cost of heavier installs (and, for some, licences to check before commercial use).
The hosted way: PDF to Markdown on Apify
PDF to Markdown is the Apify actor I built for this. It does the layout work above (columns, headings, lists, ruled tables, running headers, hyphenation), reads scanned pages with OCR in 16 languages, and returns one row per document, per page or per chunk.
You need a free Apify account and your API token (Console, then Settings, then API & Integrations). Then pip install "apify-client>=3". This converts a two-column paper and writes it to a Markdown file:
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("sonic_scraper/pdf-to-markdown").call(run_input={
"pdfUrls": ["https://arxiv.org/abs/1810.04805"],
})
for doc in client.dataset(run.default_dataset_id).iterate_items():
print(doc["status"], doc["title"], doc["pageCount"], "pages,", doc["tableCount"], "tables")
with open("bert.md", "w", encoding="utf-8") as f:
f.write(doc["markdown"])
An arXiv /abs/ link works (the actor follows it to the PDF), as do Google Drive and Dropbox share links. The output starts like this (the bold author line is read as a heading, one of the imperfections you will see on paper title pages):
# BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
## Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
Google AI Language { jacobdevlin,mingweichang,kentonl,kristout } @google.com
## Abstract
We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. ...
## 1 Introduction
Chunks with page numbers for a vector database
Set splitBy to chunk and each row becomes a piece of about chunkSize tokens that starts at a section boundary, with its heading path and page range. This writes them as JSON Lines, ready to embed:
import json
run = client.actor("sonic_scraper/pdf-to-markdown").call(run_input={
"pdfUrls": ["https://arxiv.org/abs/1706.03762"],
"splitBy": "chunk",
"chunkSize": 500,
"chunkOverlap": 50,
})
with open("chunks.jsonl", "w", encoding="utf-8") as f:
for c in client.dataset(run.default_dataset_id).iterate_items():
f.write(json.dumps({
"id": f"{c['fileName']}#{c['chunkIndex']}",
"text": c["markdown"],
"metadata": {"title": c["title"], "section": " > ".join(c["headingPath"]),
"pages": f"{c['pageStart']}-{c['pageEnd']}", "source": c["url"]},
}) + "\n")
A chunk from that run looks like this:
{
"chunkIndex": 6,
"headingPath": ["Attention Is All You Need", "3 Model Architecture"],
"pageStart": 2,
"pageEnd": 3,
"tokenEstimate": 194,
"markdown": "## 3 Model Architecture\n\nMost competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35]. ..."
}
Storing the section and pages as metadata lets your application answer with "see section 3, page 2" instead of an unsourced paragraph.
Every PDF on a page, and scanned documents
Put a web page in pageUrls and every PDF it links to is converted, for example a company's reports page or a regulator's publications page. pageRange keeps a first test cheap:
run = client.actor("sonic_scraper/pdf-to-markdown").call(run_input={
"pageUrls": ["https://www.irs.gov/forms-pubs/about-form-w-9"],
"maxPdfsPerPage": 5,
"pageRange": "1-3",
"splitBy": "page",
})
for row in client.dataset(run.default_dataset_id).iterate_items():
print(row["status"], row.get("fileName"), "page", row.get("pageNumber"), "found on", row.get("foundOn"))
Scanned pages are read with OCR automatically (ocrMode: "auto"); set ocrLanguages, for example ["fra", "eng"], for documents in other languages. Pages with a text layer are never billed as OCR.
What it costs
On the free plan: $2 per 1,000 PDFs, $0.30 per 1,000 text pages and $4 per 1,000 scanned (OCR) pages, with lower rates on paid Apify plans. Failed downloads and files that are not PDFs are not charged. A 20-page report costs $0.008; 1,000 of them cost $8. A text page takes about half a second, a scanned page a few seconds.
Honest limits
- Tables need ruling lines. Tables drawn with lines become Markdown tables; tables laid out only with spacing (common in academic papers) come out as text lines.
-
Images and charts are skipped, and math stays plain text (
x 1for x₁). - Forms and posters with text boxes scattered over the page convert in a readable but imperfect order.
- OCR follows scan quality. Clean scans read well; photos and handwriting less so.
- If your PDFs are simple single-column documents, the free pdfplumber code above may be all you need.
Links
- Actor: apify.com/sonic_scraper/pdf-to-markdown
- pdfplumber: github.com/jsvine/pdfplumber
- Using it from Claude, Cursor or another MCP client:
https://mcp.apify.com?tools=sonic_scraper/pdf-to-markdown
Questions or a PDF that converts badly? Leave a comment or open an issue on the actor's Issues tab with the PDF link. I read them daily.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.