I Blocked the Attacker's IP. The Attack Didn't Stop. Here's Why.
It's 2:14 AM and your phone won't stop buzzing. The alert says your /reset-password endpoint is getting 40,000 requests per second. Normal traffic is maybe a few requests per minute. Your dashboard shows it all coming f
It's 2:14 AM and your phone won't stop buzzing.
The alert says your /reset-password endpoint is getting 40,000 requests per second. Normal traffic is maybe a few requests per minute. Your dashboard shows it all coming from one IP address, somewhere in India.
Easy fix, right? You add that IP to the block list, refresh the dashboard, and wait for the graph to drop.
It doesn't drop.
It's still 40,000 requests per second.
I came across this as a system design interview question, and it stuck with me, because the "obvious" answer is the wrong one. I'm a Business Analyst who's learning system design, not a security engineer. So in this post I'm working through it the way I'd want someone to explain it to me: what's actually going on, why the block didn't work, and what a calmer, layered response looks like.
Let's take it apart.
Understand the Problem First
Before touching any config, it helps to slow down and ask: what is this endpoint, and why would anyone hammer it?
What /reset-password actually does
When you click "Forgot password?", the app usually:
- Takes the email or phone number you typed
- Looks it up in the database
- Creates a one-time reset token and saves it
- Sends you an email or SMS with a link or code
That's four steps, and some of them cost real money. Every SMS costs you something. Every email counts against your sending limits and your domain's reputation.
So this endpoint is unusual. Most endpoints just read data. This one does things in the outside world on behalf of someone who isn't logged in.
Why it's dangerous
At 40,000 requests per second, a few different things could be going wrong at once:
- Your bill. If even a fraction of those requests send SMS, the cost adds up quickly. There's a known fraud pattern called SMS pumping (or toll fraud), where attackers trigger texts to premium numbers they profit from.
- Your email reputation. Email providers notice when you suddenly send huge volumes. Your real emails, like order receipts, might start landing in spam.
- Your users. Real people may get flooded with reset emails they never asked for. That's confusing and a bit scary for them.
- Account enumeration. If your endpoint answers "email not found" for unknown addresses, the attacker can use it to learn which emails have accounts.
- Your system. 40,000 database lookups and token writes per second can slow down everything else that shares that database, including logins and checkout.
The business impact
Put together, this isn't only a "server is busy" problem. It's money leaking, trust eroding, and possibly the whole product slowing down for everyone.
Reasonable assumptions
Since the question gives us very little, I'll assume:
- The app sits behind something like a load balancer or CDN (more on those below)
- Reset messages are sent through a third-party email or SMS provider
- We have basic logs and metrics, but no fancy bot protection yet
- The block was added somewhere sensible, like a firewall rule or app config
If your setup is different, some of the reasoning below will shift. That's fine. The thinking process matters more than the exact tool.
So Why Didn't Blocking the IP Work?
This is the heart of the question. The honest answer is "it depends," but there's a short list of usual suspects. Any one of them (or several together) explains what we're seeing.
Here's an analogy I found helpful. Imagine a shop being flooded with prank phone calls. You block the number shown on caller ID. The calls keep coming. Why? Maybe the caller ID was fake. Maybe they switched phones. Maybe you blocked the number on the wrong phone line. Or maybe the calls are blocked, but your phone still rings before the block kicks in.
Each of those has a real-world equivalent.
1. The "IP" on your dashboard wasn't the real IP
Most apps don't talk to users directly. Requests usually pass through a load balancer (a traffic cop that spreads requests across your servers) or a CDN (a network of servers around the world that sits in front of your app).
Because of that, your app often learns the user's IP from an HTTP header called X-Forwarded-For. The proxy adds it to say "this request originally came from X."
Here's the catch: a client can put anything it likes in that header. If your app trusts it blindly, the attacker can write any IP they want.
So a very plausible story is:
- Your dashboard reads
X-Forwarded-Forand shows one Indian IP - Your firewall blocks that IP based on the actual network connection
- The actual connections come from completely different addresses
- The block matches nothing, and the flood continues
Or the reverse: you blocked the header value, and the attacker simply changed it.
2. You blocked a proxy, not the attacker
CDNs have servers in many countries, including India. If an attacker sits near a CDN server in Mumbai, all their traffic might arrive at your app from that CDN server's IP.
Block that IP at your origin, and two things happen. The CDN routes traffic through a different one of its servers, so the attack continues. And real users who happened to use that CDN server may get blocked too.
3. The attacker just moved
Renting lots of IP addresses is cheap. Attackers use botnets (networks of hacked devices) and residential proxy services (traffic routed through ordinary home internet connections). When one address gets blocked, the bot operator simply moves to new IP addresses and carries on.
IPv6 makes this even easier. Home connections usually get a whole block of addresses, not just one. The IETF's guidance suggests something like a /56 for home users. Even a single /64 block contains about 18 quintillion addresses (2 to the power of 64). Blocking one address out of that barely matters.
4. The attacker skipped your front door
If your block lives on the CDN or a WAF (Web Application Firewall, a filter that inspects incoming requests), it only works for traffic that goes through them.
If your origin server's IP address has leaked (old DNS records, a subdomain that doesn't use the CDN, your mail server setup), the attacker can send requests straight to your server. Your shiny CDN rule never sees them.
5. The block was in the wrong layer, or didn't spread
A few more boring but common causes:
- The rule was added in app code, so every request still reaches your servers before being rejected
- You have servers in several regions, and the rule only went live in one
- Another rule earlier in the list allowed the traffic before the block rule ran
6. The block did work. The graph just looks the same.
This one surprised me. Blocking an IP doesn't stop someone from sending requests. It only changes how you respond.
If your request-count graph is measured at the load balancer, it will still show 40,000 requests per second. They're just being rejected now. The real question isn't "did the number go down?" It's "are we still sending emails, writing tokens, and hitting the database?"
That distinction changes everything about how you respond.
The First 5 Minutes: Stop the Bleeding
The goal right now is not to catch the attacker. It's not to find the perfect fix either. It's to stop the damage so you have time to think.
Think of a burst pipe. First you shut off the water. You figure out why the pipe burst later.
Step 1: Find out what's actually being harmed
Remember point 6 above. Before doing anything else, check:
- Are emails or SMS still going out at a high rate? (Your provider's dashboard usually shows this.)
- Is the database under pressure?
- Are other features, like login and checkout, getting slower?
This tells you whether you're in "money and users are being hurt" mode or "the graph looks scary but we're coping" mode.
Step 2: Protect the expensive part
If messages are still being sent, that's the most urgent leak. Some fast options:
- Pause or cap sending in your email or SMS provider. Many providers let you set spending or volume limits.
- Turn off SMS resets temporarily with a feature flag (a switch that turns a feature on or off without redeploying code), and keep email only.
- Stop sending duplicates. If the same account already got a reset message in the last few minutes, don't send another.
The trade-off: a few real users might not get their reset message for a while. That's annoying, but it's much better than a five-figure SMS bill or your whole email domain getting flagged as spam.
Step 3: Put a hard limit on the endpoint itself
Add a blunt rate limit on /reset-password as a whole. A rate limiter is simply a rule like "no more than N requests per second for this endpoint." Anything over that gets rejected quickly, with an HTTP 429 ("Too Many Requests") response.
Pick a number well above your normal traffic, but far below 40,000. Some real users will be caught in it. That's the cost of stabilizing.
Step 4: Tell people
Post a quick note in your incident channel. Let support know that users might mention odd reset emails. Communication feels like a soft step, but it prevents five people from "fixing" the same thing in five different ways.
The Next 15 Minutes: Get Smarter About Who You Block
Now that the bleeding has slowed, you can move from blunt tools to sharper ones. The big shift here is this:
Stop asking "which IP is attacking?" and start asking "what does this attack have in common?"
Fix how you read the client IP
Make sure your app only trusts X-Forwarded-For values added by your own load balancer or CDN, not whatever the client sent. MDN describes two ways to do this: tell your app how many proxies sit in front of it, or give it a list of your proxies' addresses. Either way, it reads the header from the right-hand end and ignores what the client wrote.
Once that's right, your dashboard shows the real source addresses. You may suddenly see that "one IP in India" is really thousands of IPs all over the world. That alone often explains the whole mystery.
Rate limit by the thing being attacked, not just by IP
This was the biggest "aha" for me. An attacker can change their IP easily. They can't easily change which accounts they're targeting.
So add limits like:
- Per email or phone number: at most a few reset requests per hour for the same account (OWASP recommends this kind of per-account limit)
- Per IP and per network range: limit a whole range (say, an IPv6 /64) instead of single addresses
- Global: a ceiling for the endpoint as a whole, like the one from Step 3
Each layer catches something the others miss. Per-account limits protect your users' inboxes. Range limits catch rotating addresses. The global limit protects your system no matter what.
The trade-off: per-account limits can be abused too. Someone could deliberately use up the limit for your email, so you can't reset your own password for an hour. Keep the window short and the limit sensible.
Move the block to the edge
The "edge" means the outermost layer of your system, usually a CDN or WAF. Blocking there means bad requests never reach your servers at all, so they cost you almost nothing.
If your origin's IP might be exposed, lock it down so it only accepts traffic from your CDN. Then nobody can sneak around the front door.
Look for a fingerprint
Attack traffic usually shares something. Common examples:
- The same or a strange User-Agent (the text a browser sends to say what it is)
- Missing headers that real browsers always send
- The same request body pattern, or emails following a pattern like
user1@,user2@
If you find one, block or challenge that pattern at the edge. It works no matter how many IPs the attacker uses.
Add a challenge, not just a wall
A CAPTCHA or an invisible bot check asks the client to prove it's a real browser with a real human before the reset goes through. Real users barely notice. Scripts sending 40,000 requests a second struggle.
The trade-off is accessibility and friction. Some CAPTCHAs are hard for people with disabilities, including people using screen readers, and some users just find them annoying. Many teams only show the challenge when traffic looks suspicious, rather than to everyone, all the time.
Should you block all of India?
It's tempting. Please don't, unless you truly have no users there. You'd be locking out a huge number of real people to stop one attacker who, as we've seen, may not even be in India. Well-funded bots can get around country blocks anyway. Geography-based blocks are a last resort, not a first move.
Root Cause Investigation: Once Things Are Calm
With the system stable, you can finally play detective. Here's where I'd look, and what each place can tell you.
| Where to look | What you're trying to learn |
|---|---|
| Edge / CDN / WAF logs | The real source IPs, countries, and networks. Is it one IP or thousands? |
| App logs | What's in X-Forwarded-For vs. the actual connection IP. Do they match? |
| Request details | Shared User-Agents, headers, or patterns in the emails being targeted |
| Metrics | When it started, how it grew, and whether it changed when you blocked the IP |
| Tracing | Which step is slow or expensive: the DB lookup, token write, or message send |
| Email / SMS provider | How many messages actually went out, to which countries, and what it cost |
| Database health | Connection pool usage, slow queries, lock contention |
| Recent deployments | Did a recent change remove a rate limit or a CAPTCHA? |
| Infrastructure changes | New subdomains, DNS changes, or anything that might expose the origin IP |
A quick note on tracing, since it's a newer term for many people. Tracing follows one single request through every part of your system, like a parcel tracking number. It shows exactly where time and effort were spent.
A few questions I'd want answered by the end:
- Was the "single IP" ever real, or a header value?
- Were the targeted emails real users, random guesses, or premium-rate phone numbers?
- Did the attacker get anything useful, like a list of which emails have accounts?
- How much did the incident cost in messages sent?
The answers decide which long-term fixes matter most for your system.
Long-Term Prevention: Design the Endpoint to Be Boring
The best outcome is a reset endpoint that's simply not worth attacking. Each idea below removes some reason an attacker would bother.
Give the same answer every time
Whether the email exists or not, respond with the same message, like "If an account exists, we've sent a reset link." Try to make the response take roughly the same time too. OWASP suggests sending the email in the background, which also helps with the next point.
This kills account enumeration. The attacker learns nothing from your replies.
Make the request cheap, and the sending asynchronous
Don't send the email during the request. Instead, put a small job on a queue (a waiting list that background workers process at their own pace) and reply right away.
Why it helps: the queue acts like a shock absorber. Even if 40,000 requests arrive, the workers only send as fast as you allow. You can also dedupe ("this account already has a pending reset") before anything goes out.
Layered rate limiting
Keep the layers from earlier as permanent settings: per account, per IP range, and global. Store the counters somewhere fast and shared, like Redis (an in-memory data store that's very quick at counting things), so every server sees the same numbers.
Bot protection at the edge
Most CDNs and cloud providers offer WAF rules and bot detection. They can challenge suspicious traffic before it reaches you. This is usually much cheaper than building your own.
Protect the origin
Make sure your servers only accept traffic from your CDN or load balancer. Don't leave a side door open.
Circuit breaker on your message provider
A circuit breaker is like the one in your house. If something goes badly wrong, it trips and stops the flow automatically.
Here, it could mean: "if we've sent more than X reset messages in 10 minutes, pause sending and alert a human." That caps your worst-case bill even if every other defense fails.
Restrict where SMS can go
If you only serve certain countries, only allow SMS to those countries. Twilio, for example, recommends turning off every country you don't plan to send messages to. It's one of the simplest defenses against SMS pumping.
Feature flags for emergency switches
Have ready-made switches for things like "disable SMS reset" or "require CAPTCHA for everyone." During an incident you want to flip a switch, not write and deploy code at 2 AM.
Monitoring and alerts on the right signals
Alert on the things that actually hurt, not just request count:
- Reset messages sent per minute
- SMS spend per hour
- Reset requests per account
- Share of reset requests for emails that don't exist
That last one is a great early warning. Real users mostly type real emails. Attackers guessing don't.
Load testing and practice
Load testing means sending fake heavy traffic to your system on purpose, in a safe environment, to see where it breaks. Do it for this endpoint specifically. Then run a practice incident ("game day") so the team knows where the switches are before they need them.
What about auto scaling?
Auto scaling means automatically adding more servers when traffic grows. It's great for real popularity. For an attack, it can backfire: you scale up and simply pay more to process junk. Scaling buys you time. It doesn't solve this problem.
Common Mistakes
These are the traps I'd most likely have fallen into myself before working through this.
- Treating the IP as an identity. An IP address is more like a temporary seat on a bus than a name tag. Many people can share one, and one person can switch between many.
-
Trusting
X-Forwarded-Forblindly. It's just text the client can write. Only trust values added by proxies you control. StackHawk wrote about how this exact mistake let spoofed IPs get past their own API's rate limiting. - Playing whack-a-mole. Blocking IPs one by one feels productive, but attackers can rotate faster than you can type. Look for patterns instead.
- Measuring the wrong thing. "Requests per second" doesn't tell you if you're losing money. "Messages sent" and "cost per hour" do.
- Sending messages inside the request. It ties your cost directly to the attacker's speed. A queue breaks that link.
- Blocking a whole country. It punishes real users and often doesn't even hit the attacker.
- Scaling up to absorb the attack. You end up paying more to serve the attacker better.
- Revealing whether an account exists. Different messages for "found" and "not found" turn your reset page into a free lookup tool.
- Forgetting real users in the response. Overly harsh limits or CAPTCHAs can lock out the very people the reset page exists to help.
Key Takeaways
- The IP you see may not be the attacker's IP. Headers can be faked, proxies hide the real source, and attackers rotate addresses cheaply.
- Blocking doesn't stop sending. It only changes your response. Measure harm (messages, cost, DB load), not just traffic.
- Stabilize first, investigate second. Cap the expensive part, add a global limit, then get clever.
- Rate limit by what's being targeted. Per-account limits survive IP rotation in a way per-IP limits can't.
- Push defenses to the edge and make sure nobody can bypass it.
- Design the endpoint to be boring: same response every time, queued sending, hard caps on spend.
- Every defense has a cost to real users. Good incident response keeps them in mind the whole time.
If there's one sentence I'll remember from this, it's: don't fight the address, fight the pattern.
Over to You
I'm still learning this space, so I'd genuinely like to hear from people who've dealt with it for real.
If you've handled abuse on a password reset or OTP endpoint, what was the one defense that actually made the difference, and which "obvious" fix turned out to be useless?
Sources and Further Reading
These are the pages I leaned on while working through this. If you want to go deeper, start with the first two.
- OWASP Forgot Password Cheat Sheet: consistent responses, user enumeration, per-account limits, and flooding
- MDN: X-Forwarded-For header: why the header can be spoofed and how to pick a trustworthy IP
- StackHawk: Do You Trust Your X-Forwarded-For Header?: a real story of spoofed IPs bypassing rate limiting
- Cloudflare: Using machine learning to detect bot attacks that leverage residential proxies: how residential proxies get around IP blocks, country blocks, and rate limits
- Cloudflare docs: Cloudflare IP addresses: how an exposed origin IP lets attackers bypass the CDN, and allowlisting
- RFC 6177: IPv6 Address Assignment to End Sites: how much IPv6 address space home users typically get
- Twilio: What is SMS Pumping Fraud?: how the fraud works and the geo-permission defense
- MDN: 429 Too Many Requests: the status code used for rate limiting
- Martin Fowler: Circuit Breaker: the classic explanation of the pattern
- W3C: Inaccessibility of CAPTCHA: accessibility trade-offs of bot challenges
The incident steps and their order (stabilize first, then target, then investigate) are my own reasoning, not taken from one source.
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.