Building a WhatsApp AI Agent with Gemini Using Gemini as Your Copilot

What you'll build: A WhatsApp number that replies with an AI agent powered by Google Gemini. You'll use Gemini (via the Gemini CLI or Gemini Code Assist in your IDE) as your pair-programming copilot to scaffold, debug, and extend the code, so Gemini is both the agent's runtime brain and the assistant building it.

How the pieces fit together

WhatsApp user
     │  message
     ▼
Meta WhatsApp Cloud API  ──webhook POST──►  Your server (Flask)
     ▲                                            │
     │  send reply (Graph API)                    ▼
     └──────────────────────────────────  Gemini API (the "brain")

Two moving parts:

• WhatsApp Cloud API (Meta) — receives inbound messages via a webhook and sends replies via the Graph API. Free tier available.
• Gemini API (Google) — generates the agent's responses.

Two hats for Gemini: at runtime, the Gemini API generates replies to WhatsApp users. While building, you use Gemini as your coding copilot — the Gemini CLI (npm install -g @google/gemini-cli, then run gemini) in your terminal, or Gemini Code Assist inside VS Code / JetBrains. You describe what you want in plain English, and it writes, explains, and fixes the code below.

Prerequisites

• Python 3.10+
• A Meta for Developers account (free)
• A Google AI Studio API key for Gemini (free tier available)
• ngrok or similar, to expose your local server to Meta's webhook
• The Gemini CLI installed (npm install -g @google/gemini-cli) or Gemini Code Assist enabled in your IDE, open alongside your editor

Step 1 — Get your Gemini API key

1. Go to aistudio.google.com → Get API key.
2. Copy the key somewhere safe.

Ask Gemini: "Explain what a Gemini API key can access and how to store it safely as an environment variable."

Step 2 — Set up WhatsApp Cloud API

1. In Meta for Developers, create an app → type Business.
2. Add the WhatsApp product. Meta gives you a test phone number and a temporary access token.
3. Note these three values from the WhatsApp → API Setup page:
   • Phone number ID
   • Temporary access token (valid 24h; swap for a permanent System User token later)
   • App secret (App Settings → Basic)
4. Add your own number as a recipient so you can test.

Ask Gemini: "Walk me through creating a permanent WhatsApp access token with a System User in Meta Business Suite."

Step 3 — Project setup

mkdir whatsapp-gemini-agent && cd whatsapp-gemini-agent
python -m venv .venv && source .venv/bin/activate
pip install flask google-genai requests python-dotenv

Create a .env file:

GEMINI_API_KEY=your_gemini_key
WHATSAPP_TOKEN=your_whatsapp_access_token
WHATSAPP_PHONE_NUMBER_ID=your_phone_number_id
VERIFY_TOKEN=pick_any_random_string

Ask Gemini: "Generate a .gitignore for a Python project and make sure .env is excluded."

Step 4 — The webhook server

Meta requires two things from your endpoint:

1. GET /webhook — a one-time verification handshake.
2. POST /webhook — where inbound messages arrive.

Create app.py:

import os
import requests
from flask import Flask, request
from google import genai
from dotenv import load_dotenv

load_dotenv()

GEMINI_API_KEY = os.environ["GEMINI_API_KEY"]
WHATSAPP_TOKEN = os.environ["WHATSAPP_TOKEN"]
PHONE_NUMBER_ID = os.environ["WHATSAPP_PHONE_NUMBER_ID"]
VERIFY_TOKEN = os.environ["VERIFY_TOKEN"]

app = Flask(__name__)
client = genai.Client(api_key=GEMINI_API_KEY)

SYSTEM_PROMPT = (
    "You are a friendly, concise WhatsApp assistant. "
    "Keep replies short and clear — this is a chat app, not email."
)


def ask_gemini(user_text: str) -> str:
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=user_text,
        config=genai.types.GenerateContentConfig(system_instruction=SYSTEM_PROMPT),
    )
    return response.text


def send_whatsapp_message(to: str, body: str) -> None:
    url = f"https://graph.facebook.com/v22.0/{PHONE_NUMBER_ID}/messages"
    headers = {"Authorization": f"Bearer {WHATSAPP_TOKEN}"}
    payload = {
        "messaging_product": "whatsapp",
        "to": to,
        "type": "text",
        "text": {"body": body},
    }
    requests.post(url, headers=headers, json=payload, timeout=20)


@app.get("/webhook")
def verify():
    # Meta's verification handshake
    if (request.args.get("hub.mode") == "subscribe"
            and request.args.get("hub.verify_token") == VERIFY_TOKEN):
        return request.args.get("hub.challenge"), 200
    return "Forbidden", 403


@app.post("/webhook")
def incoming():
    data = request.get_json()
    try:
        change = data["entry"][0]["changes"][0]["value"]
        message = change["messages"][0]          # inbound message
        sender = message["from"]                  # user's phone number
        text = message["text"]["body"]            # message text

        reply = ask_gemini(text)
        send_whatsapp_message(sender, reply)
    except (KeyError, IndexError):
        # Status updates and non-text messages land here — ignore them
        pass
    return "OK", 200


if __name__ == "__main__":
    app.run(port=5000)

Ask Gemini: "Paste this file and explain each function line by line, then suggest error handling I'm missing."

Step 5 — Expose your server and connect the webhook

Run the server, then tunnel it:

python app.py            # terminal 1
ngrok http 5000          # terminal 2  → copy the https URL

In Meta's WhatsApp → Configuration:

• Callback URL: https://<your-ngrok-id>.ngrok.io/webhook
• Verify token: the same VERIFY_TOKEN from your .env
• Click Verify and save (this triggers the GET handshake).
• Under Webhook fields, subscribe to messages.

Ask Gemini: "My webhook verification is returning 403. Here's my code and the ngrok logs — what's wrong?"

Step 6 — Test it

Send a WhatsApp message from your registered number to the test number. Within a second or two you should get a Gemini-generated reply.

If nothing comes back, ask Claude to help you read the logs:

Ask Gemini: "The webhook receives a POST but no reply is sent. Here's the JSON payload and my server log — trace where it breaks."

Step 7 — Turn the chatbot into an agent

A chatbot answers. An agent takes actions. Gemini supports function calling — you declare tools, and the model decides when to call them.

Example: give the agent a check_reservation tool.

def check_reservation(confirmation_code: str) -> dict:
    # Replace with a real lookup (DB, internal API, etc.)
    return {"code": confirmation_code, "status": "confirmed", "checkin": "2026-08-14"}


def ask_gemini_agent(user_text: str) -> str:
    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=user_text,
        config=genai.types.GenerateContentConfig(
            system_instruction=SYSTEM_PROMPT,
            tools=[check_reservation],   # SDK auto-generates the schema from the function
        ),
    )
    return response.text

The Gemini SDK reads the function's signature and docstring to build the tool schema, calls it when the user asks about a reservation, and folds the result into its reply.

Ask Gemini: "Add a second tool that cancels a reservation, and add conversation memory so the agent remembers earlier messages in the same chat."

Good next tools to build with Claude's help:

• Conversation memory — store recent turns per phone number (a dict, Redis, or a DB) and pass them as contents.
• Handoff to a human — detect frustration or keywords and notify a live agent.
• Domain tools — bookings, order status, FAQs backed by your own data.

Step 8 — Before you go to production

• Swap the temporary WhatsApp token for a permanent System User token.
• Verify the X-Hub-Signature-256 header on every POST using your App Secret — reject anything unsigned.
• Return 200 fast and process Gemini calls in a background task/queue (Meta retries if you're slow).
• Add rate limiting and logging.
• Move off ngrok to real hosting (Cloud Run, Render, Fly.io, a VM).

Ask Gemini: "Write the X-Hub-Signature-256 verification middleware for my Flask app, and refactor the Gemini call to run in a background thread so the webhook returns 200 immediately."

How to work with Gemini effectively

• Give context, not just the error. Paste the code and the log/payload.
• Ask for explanations, not only fixes — you own this code afterward.
• Iterate in small steps. One tool, one feature, one bug at a time.
• Ask Gemini to review for security and edge cases before you ship. In the Gemini CLI you can point it right at a file (gemini "review app.py for security issues").

Quick reference

Model note: gemini-2.5-flash is fast and cheap for chat; switch to gemini-2.5-pro for harder reasoning. Check Google's model list for the latest IDs.