How to Build an Autonomous AI Web Research Agent with CrewAI & Smart Markdown API
When building autonomous AI agents (like CrewAI, AutoGen, or LangGraph) that search and read web documentation, feeding raw HTML causes context truncation errors. Here is how to equip your CrewAI agent with clean Markdo
When building autonomous AI agents (like CrewAI, AutoGen, or LangGraph) that search and read web documentation, feeding raw HTML causes context truncation errors.
Here is how to equip your CrewAI agent with clean Markdown scraping:
Step 1: Install Dependencies
pip install crewai requests
Step 2: Define the Scraping Tool
import requests
from crewai.tools import tool
@tool("Smart Web Scraper")
def scrape_webpage(url: str) -> str:
"""Scrapes a URL and returns clean Markdown optimized for LLMs."""
endpoint = "https://smart-markdown-web-scraper.p.rapidapi.com/scrape"
headers = {
"x-rapidapi-key": "YOUR_RAPIDAPI_KEY",
"x-rapidapi-host": "smart-markdown-web-scraper.p.rapidapi.com",
"Content-Type": "application/json"
}
response = requests.post(endpoint, json={"url": url}, headers=headers)
return response.json().get("markdown", "")
Step 3: Run Your AI Agent
Now your CrewAI researcher can read technical documentation sites or news articles with zero HTML noise!
👉 Try the API on RapidAPI Hub: Smart Markdown Web Scraper
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.