Custom Tools for LangChain and CrewAI: Wrapping Proxy Rotation into Agentic Functions

The promise of autonomous agents is total liberation: you define the goal, provide the tools, and step back while the LLM orchestrates the labor. But the moment your agents step outside the clean, gated gardens of official APIs and start interacting with the raw web, they hit a wall. Rate limits, IP bans, and geo-fencing turn a sophisticated CrewAI workflow into a graveyard of "403 Forbidden" errors.

If you are building production-grade scrapers, market intelligence bots, or automated researchers, you quickly realize that an agent is only as good as its ability to remain undetected. Managing proxy rotation manually inside an agent’s logic is a recipe for high latency and token waste. The solution is to treat proxy management not as a network setting, but as a Custom Tool.

In this guide, we will dive deep into architecting proxy-aware tools for LangChain and CrewAI, ensuring your agents move through the web like ghosts rather than bulldozers.

Why does your agent keep getting blocked?

Every time an agent uses a standard RequestsWrapper or a Selenium tool, it leaves a digital fingerprint. If your agent is performing a recursive search—say, analyzing twenty different LinkedIn profiles or price-checking fifty Amazon listings—it originates all those requests from a single IP address in a very short window.

To a server’s firewall, this looks like a classic scraping bot. Even if you use high-quality residential proxies, hardcoding them into your environment variables isn't enough. You need dynamic rotation logic that can handle failures gracefully, switch regions on the fly, and maintain session persistence when needed.

The Architecture of a Proxy-Aware Tool

When we talk about "Custom Tools" in LangChain or CrewAI, we are essentially creating an interface that the LLM can understand. A well-designed proxy tool should hide the complexity of the networking layer while giving the agent a clean API.

1. The Strategy: Functional Wrapping vs. Class Inheritance

In LangChain, you have two primary ways to build tools: using the @tool decorator for simple functions or subclassing BaseTool for complex states. For proxy rotation, subclassing is superior because it allows you to maintain a connection pool or a rotation counter without polluting the global namespace.

2. The Implementation Logic

A robust tool needs to handle three things:

• Authentication: Passing credentials to the proxy provider.
• Rotation: Selecting a new IP for each request or task.
• Validation: Checking if the proxy is actually working before the agent wastes a prompt.

Step-by-Step: Building a Rotating Proxy Tool for LangChain

Let’s look at how to wrap a proxy rotation service into a tool that a LangChain agent can use to perform "stealth" web requests.

The Foundation: Defining the Request Logic

We start by creating a utility that handles the actual HTTP request with proxy injection.

import requests
from langchain.tools import tool

def fetch_with_proxy(url: str, proxy_list: list):
    # Select a proxy (this could be random or sequential)
    proxy = {"http": proxy_list[0], "https": proxy_list[0]}

    try:
        response = requests.get(url, proxies=proxy, timeout=10)
        response.raise_for_status()
        return response.text[:2000] # Truncate to save tokens
    except Exception as e:
        return f"Error fetching {url}: {str(e)}"

The Custom Tool Interface

Now, we wrap this logic so CrewAI or LangChain agents know how to use it. The key here is the Docstring. The LLM uses this description to decide when to trigger the tool.

from langchain.pydantic_v1 import BaseModel, Field

class ProxySearchInput(BaseModel):
    url: str = Field(description="The full URL of the website to scrape or analyze")

@tool("stealth_web_browser", args_schema=ProxySearchInput)
def stealth_web_browser(url: str):
    """
    Use this tool to access websites that have strict bot protection. 
    It uses a rotating proxy network to mask the agent's identity.
    """
    # In a real scenario, pull these from an environment variable or a managed service
    proxies = ["http://user:[email protected]:8000", "http://user:[email protected]:8001"]
    return fetch_with_proxy(url, proxies)

Integrating with CrewAI: Multi-Agent Orchestration

CrewAI excels at role-playing. You can designate a "Web Researcher" agent whose sole purpose is to use your stealth_web_browser tool. By isolating the proxy logic to a specific agent, you can save costs—using expensive residential proxies only for the Research Agent, while the "Analyst Agent" works on the local machine using cached data.

The Manager-Worker Pattern

In a complex CrewAI setup, you might have a "Proxy Manager" agent. This isn't just a technical layer; it’s a strategic choice.

• Researcher Agent: Uses the proxy tool to gather data.
• Validator Agent: Checks the data for completeness.
• Writer Agent: Synthesizes the final report.

This separation of concerns ensures that if a proxy fails, only one agent's task is retried, rather than the entire pipeline crashing.

Framework: The "RESILIENT" Proxy Checklist

When moving from a script to a production agent tool, use the RESILIENT framework to evaluate your custom tool:

1. Rotation: Does the tool cycle IPs automatically?
2. Error Handling: Does it return a clear message if blocked?
3. Session Management: Can it maintain cookies if the agent needs to "log in"?
4. Intelligence: Does the agent know why a request failed (e.g., a 403 vs. a 404)?
5. Latency: Are you using data-center proxies for speed or residential for stealth?
6. Integration: Does it play nice with Pydantic schemas?
7. Efficiency: Does it truncate HTML to avoid blowing the LLM's context window?
8. Network Isolation: Is the proxy logic hidden from the LLM's reasoning chain?
9. Traceability: Are you logging which proxy was used for which task?

The Hidden Insight: Why "Simple" Proxying Fails Agentic Workflows

Most developers stop at rotating the IP. However, sophisticated websites look at the User-Agent and TLS Fingerprint.

If your Custom Tool uses Python's requests library, the TLS handshake looks distinctly "Pythonic." Advanced firewalls will block you even if your IP is clean. To make your LangChain tools truly "Senior" level, your custom tool should use libraries like curl_cffi or playwright which can mimic the TLS fingerprint of a real Chrome browser.

Advanced Implementation Snippet:

from curl_cffi import requests as stealth_requests

def advanced_stealth_fetch(url, proxy):
    # This mimics a Chrome browser TLS fingerprint
    res = stealth_requests.get(
        url, 
        proxies={"http": proxy, "https": proxy}, 
        impersonate="chrome110"
    )
    return res.text

Wrapping this into a LangChain tool gives your agents a "cloaking device" that standard scrapers lack.

Conclusion: Turning Infrastructure into Intelligence

Building Custom Tools for LangChain and CrewAI is more than just writing Python wrappers. It is about translating low-level infrastructure—like proxy rotation and header spoofing—into high-level capabilities that an LLM can strategically deploy.

When you wrap proxy rotation into a tool, you stop treating your agent like a script and start treating it like a professional operative. You provide it with the means to navigate the complexities of the modern web without being sidelined by technical friction.