Perusal: the internet without all the 💩

I've just released Perusal, a small Python tool for reading the web without most of the web getting in the way.

Modern web pages contain an extraordinary amount of stuff that isn't actually the content:

• navigation
• cookie banners
• footers
• related articles
• scripts
• styles
• tracking
• repeated links
• UI chrome
• adverts
• framework scaffolding

Sometimes you don't want a web page.

You just want what the page says.

That's what Perusal does.

pip install perusal

Then:

perusal https://en.wikipedia.org/wiki/Domino

And instead of a giant HTML document, you get clean Markdown on stdout.

HTML in, useful text out

Under the hood the basic pipeline is deliberately simple:

fetch
  ↓
DOM
  ↓
Readability
  ↓
Turndown
  ↓
clean Markdown

Perusal uses the Python ports of Readability and Turndown I've been working on in the domonic ecosystem.

That means it isn't just stripping tags.

It first tries to identify the meaningful article content, throws away most of the surrounding page furniture, and then converts what remains into Markdown.

You can use a URL:

perusal https://example.com/article

A local HTML file:

perusal ./saved-page.html

Or pipe HTML straight into it:

curl -s https://example.com | perusal -

See how much rubbish disappeared

There's also a report command:

perusal report https://example.com

This compares the original HTML with the cleaned Markdown and gives you an indication of the size reduction and approximate token saving.

That becomes particularly interesting when the consumer isn't a human.

It's an LLM.

If I'm passing a web page into an AI system, I generally don't want to pay for it to read navigation menus, tracking markup and 400 links from the footer.

I want the useful bit.

Give it a reading goal

Perusal can also rank sections of a page against a query:

perusal https://example.com/docs \
    --query "installation instructions" \
    --max-chars 4000

The query isn't a web search.

It's more like telling the reader:

this is what I'm trying to find on this page

Perusal ranks the headings and content chunks against that goal and then applies the character budget.

So instead of blindly chopping the first 4,000 characters from a document, it can preferentially return the relevant sections while keeping them in their original page order.

That makes it useful as a very small preprocessing layer for RAG, agents and other AI tooling.

Read several pages at once

You can also throw several URLs at it:

perusal \
    https://example.com/page-one \
    https://example.com/page-two \
    https://example.com/page-three

Perusal fetches the pages and produces one Markdown digest:

# Digest of 3 pages

## Page One

...

## Page Two

...

## Page Three

...

For simple research tasks I've found this much nicer than juggling multiple blobs of HTML.

What about JavaScript sites?

By default Perusal reads the HTML returned by the server.

That's intentional. It's fast and covers a huge amount of the web.

But obviously some sites return little more than:

<div id="app"></div>
<script src="everything.js"></script>

For those there's an optional JavaScript mode:

pip install 'perusal[js]'

Then:

perusal https://example.com/app --js

This uses myjs, my Python JavaScript interpreter, to execute the page's scripts against a live domonic DOM before handing the result to Readability.

No Node process and no full browser involved.

It's obviously heavier than simply fetching HTML, which is why it's opt-in.

It accidentally became a browser too

Once I had a clean page reader I thought:

why shouldn't humans be able to use the same thing?

So Perusal also has an optional reader browser.

pip install 'perusal[browser]'
perusal browse

Or open a page directly:

perusal browse https://en.wikipedia.org/wiki/Domino

It's a deliberately small tabbed reader with things like:

• tabs and history
• bookmarks
• incognito tabs
• adjustable typography
• save page as Markdown
• page source
• resource inspection
• DNS / WHOIS information
• certificate-transparency subdomain discovery

The DevTools are intentionally fairly old-school.

They're there to answer things like:

What did this page actually return?

What resources does it reference?

What domain am I talking to?

rather than pretending to be a replacement for Chrome DevTools.

And an MCP server

The other obvious consumer of clean web content is an agent.

So Perusal can expose the exact same reader through MCP:

pip install 'perusal[mcp]'
perusal mcp

It intentionally exposes just one read-only tool:

web_read(url, query?, max_chars?, timeout?)

That's it.

I didn't particularly want another MCP server with seventeen overlapping tools for searching, crawling, scraping, clicking and browsing.

Perusal does one thing:

read this web page for me.

That makes it quite easy to give an AI client controlled access to cleaned web content without handing it an entire browser.

One pipeline, three interfaces

So Perusal has effectively ended up with three ways of using the same idea:

perusal <url>

for scripts and terminals,

perusal browse

for humans,

and:

perusal mcp

for agents.

I'm still treating it as alpha software and there are plenty of rough edges, but I think the underlying idea is useful:

the web page and the information on the web page are not the same thing.

A lot of the time we can throw away an enormous amount of the former while keeping the latter.

If you want to try it:

pip install perusal

PyPI:

https://pypi.org/project/perusal/

Feedback, bug reports and weird pages that break it are very welcome.

python #opensource #ai #webdev