The playbook that was supposed to block the account ran zero times

Source: CD's readiness notes Sentinel-Readiness/Analytics rule.md (the Automation / SOAR notes: lab 2 "5 failed logons in 20 minutes → incident, reset password, block user, email the manager", lab 3 "how to stop a rule firing for one analytics rule"), rebuilt as a lab story and cross-checked against Microsoft Learn on 2026-10-28. Lab names only; screens are drawings.

In the spring Contoso's SOC built the automation from my notes: five failed sign-ins to the Azure portal in twenty minutes → incident → playbook pb-block-user disables the account, resets the password and emails the manager. It was demoed to the CISO. It worked.

In October a contractor's account takes 1,400 password guesses in an hour, then one success. The incident is created at 23:12. The account is disabled at 00:05, by hand, by the on-call analyst who happened to look. The playbook's run history for the last five months: 0 runs.

Nothing was broken on the night. It had been broken since the person who built it left in June.

It happened for real: Midnight Blizzard and Microsoft, 2024

Password guessing is not a beginner's attack. In January 2024 Microsoft disclosed that a Russian state actor, Midnight Blizzard, got into its corporate email by guessing a password, slowly enough that a "five failures in twenty minutes" rule would never have fired.

What happened

Why it worked

• Door 1, low and slow guessing. Microsoft's guidance says the actor used a small number of attempts per account to avoid detection. A per-account brute-force rule, like the one in this post, does not see a spray: it sees one or two failures per user.
• Door 2, a test account without MFA. Non-production does not mean unimportant when it can reach production.
• Door 3, an old app with too much power. A test OAuth app nobody reviewed could grant mailbox access to the whole company.
• Door 4, no automatic response. The attack was found weeks later. An automated "disable and revoke" on the first successful sign-in after a spray shortens that to minutes.

What it cost

The same doors in a Microsoft tenant

• Door 4, no automatic response: this whole post. A playbook that actually runs, tested every Monday.
• Door 1, the spray: add a second analytics rule that looks across accounts, not per account (query below), and use Entra ID Protection, which has its own Password spray risk detection.
• Door 2, the test account: Conditional Access MFA for all users, in every tenant you own, test tenants included.
• Door 3, the old app: review apps with application permissions to Exchange (full_access_as_app, EWS.AccessAsUser.All) and Graph, and remove what is not needed. Defender for Cloud Apps app governance flags over-privileged and unused apps.

// Password spray: one IP, many accounts, few attempts each (last 24 hours)
SigninLogs
| where TimeGenerated > ago(1d)
| where ResultType in ("50126", "50053")
| summarize Accounts = dcount(UserPrincipalName), Attempts = count(), FirstSeen = min(TimeGenerated) by IPAddress
| where Accounts >= 10
| order by Accounts desc

Point the same pb-block-user automation at this rule's incidents, with one change: block the sign-in for any account that later succeeds from the same IP, and revoke its sessions.

TL;DR

• Problem: between "incident created" and "account disabled" there is a chain of six links: trigger, conditions, order, the permission Sentinel's own identity needs on the playbook, the playbook's connections, and the identity the playbook acts with. Each one fails silently, and the SOC only sees a playbook that "exists".
• Who it hurts: the on-call analyst (manual response at midnight), the user whose account is abused for 53 minutes instead of 90 seconds, the CISO who was shown a demo, and the budget that paid for a SOAR nobody tested after June.
• Fix: incident-trigger automation rules (not playbooks attached to analytics rules, deprecated March 2026), Manage playbook permissions so the Sentinel service account holds Microsoft Sentinel Automation Contributor on the playbook's resource group, managed identity instead of a person's connection, and a weekly test incident that must produce a run.
• Cost: the playbook costs cents: about $0.0005 per run on Logic Apps Consumption, under $1 a month for a busy lab. The 53 minutes cost more.
• Prove it: a test incident produces a green run within two minutes; the test account shows Account enabled: No in Entra; the run history is checked every Monday.

Table of contents

• It happened for real: Midnight Blizzard and Microsoft, 2024
• 1. The problem
• 2. Who gets hurt
• 3. Cost and value
• 4. How to fix it
• 5. Prove it
• Gotchas
• Checklist
• What changed recently
• Sources
• News sources

1. The problem

A fire alarm wired to a sprinkler. The alarm rings; the sprinkler needs the valve open, the pipe connected, the pump powered and the pump's key in it. Any one missing, and the alarm rings over a dry room. Nobody tests the sprinkler, because the alarm is loud and that feels like testing.

In Sentinel the chain is:

2. Who gets hurt

3. Cost and value

Checked 2026-10-28, USD list, pay-as-you-go. Logic Apps per-execution prices are not readable from Microsoft's static pricing page; the figures below are the published Consumption rates quoted across Microsoft partner material in 2026. Confirm in the pricing calculator.

What the playbook costs to run (Consumption):

The first thousands of built-in actions per subscription per month are free. A Standard plan (WS1, 1 vCPU / 3.5 GB) is about $175 a month for unlimited built-in actions, worth it only when you run dozens of playbooks at volume or need VNet integration.

What the failure cost:

Value of the fix: the same playbook, correctly wired, disables the account in about 90 seconds from incident creation, for less than a dollar a month. Response time goes from "whenever someone looks" to a number you can put in a report.

How others do it: Palo Alto XSOAR and Splunk SOAR are quote-based per user or per action; Tines publishes a free Community Edition and per-story tiers. All of them have the same six links; none of them tests your connections for you.

4. How to fix it

Lab: workspace law-contoso-sec, playbooks in rg-soar, analytics rule Brute force: Azure portal (Scheduled, 5 failed sign-ins in 20 minutes).

Step 1: incident trigger, not analytics-rule attachment. Automation → Create → Automation rule. Trigger When incident is created. If the workspace is onboarded to the Defender portal, this is the only incident-creation trigger that exists; alert-triggered automation for Defender XDR alerts is not available there.

Step 2: conditions that survive a rename. Use Analytics rule name contains "Brute force", or better, tag the analytics rule's incidents and condition on Tags contains "auto-block". Mind the tag semantics: any individual tag versus collection of all tags evaluate "does not contain" differently.

Step 3: order and expiry. Give the blocking rule order 1. Give every "auto-close" rule an expiration date; the pentest rule from the story should have died with the pentest.

Step 4: Sentinel's own permission. In the rule, Actions → Run playbook. If pb-block-user is greyed out, select Manage playbook permissions, tick rg-soar, Apply. You need Owner on rg-soar to do this; it grants the Sentinel service account Microsoft Sentinel Automation Contributor there. Any automation rule can then run any playbook in that resource group.

Step 5: no person in the playbook. Logic App → Identity → System-assigned: On. Grant it:

• Microsoft Sentinel Responder on the workspace resource group (to update the incident).
• Graph application permissions User.EnableDisableAccount.All and User.ReadWrite.All (admin consent) for the HTTP action that disables the account:

# grant Graph app roles to the Logic App's managed identity
$mi    = Get-MgServicePrincipal -Filter "displayName eq 'pb-block-user'"
$graph = Get-MgServicePrincipal -Filter "displayName eq 'Microsoft Graph'"
foreach ($perm in 'User.EnableDisableAccount.All','User.ReadWrite.All') {
  $role = $graph.AppRoles | Where-Object { $_.Value -eq $perm -and $_.AllowedMemberTypes -contains 'Application' }
  New-MgServicePrincipalAppRoleAssignment -ServicePrincipalId $mi.Id -PrincipalId $mi.Id -ResourceId $graph.Id -AppRoleId $role.Id
}

The HTTP action: PATCH https://graph.microsoft.com/v1.0/users/{upn} with body {"accountEnabled": false}, authentication Managed identity, audience https://graph.microsoft.com. The Office 365 connection for the manager email: a shared mailbox [email protected], not an analyst's account; the Microsoft Sentinel connector: managed identity.

Step 6: a test that runs every Monday. A scheduled analytics rule, 06:00 Monday, that raises a Low incident tagged auto-block-test for [email protected]. The automation rule disables that account; a second automation rule (order 2) re-enables it and closes the incident. If Monday's run history is not green, someone is paged, on a Monday morning, not at midnight.

5. Prove it

1. Create a test incident (or wait for Monday's): the Runs tab on the incident shows pb-block-user Succeeded within about two minutes.
2. Entra ID → [email protected] → Account enabled: No; AuditLogs | where OperationName == "Disable account" shows the Logic App's identity as the actor.
3. The incident has a comment from the playbook and the manager's mailbox has the email, sent from soc-noreply.
4. Run history for the last 30 days: zero Failed, zero Unauthorized in any Office 365 or Sentinel action.

// playbooks that never ran: incidents the rule matched vs runs on them
SecurityIncident
| where TimeGenerated > ago(30d)
| summarize arg_max(TimeGenerated, *) by IncidentNumber
| where Title startswith "Brute force"
| summarize incidents = count(), with_comment = countif(tostring(Comments) contains "pb-block-user")

Gotchas

• Greyed-out playbook = permission, not a bug. The list greys out playbooks in resource groups where the Sentinel service account has no role. Fix it in the panel; do not move the playbook.
• Playbooks on analytics rules are deprecated (March 2026). Migrate them to automation rules with Microsoft's migration guidance; the old attachments will stop working.
• Alert trigger only for Scheduled / NRT (and Microsoft security) rules. Incidents from Defender XDR need the incident trigger.
• Connections belong to people by default. Any connector authorised with "Sign in" carries that person's token. Use managed identity where the connector supports it, a service mailbox where it does not.
• Defender portal batching. Several updates to one incident within 5 to 10 minutes reach Sentinel as the last one only; update-trigger rules see fewer events than you expect.
• Order collisions. Two rules with the same order run in random sequence. Give every rule a unique number.
• Expiry is a feature. Any rule created for a test or a pentest gets an expiration date on day one.
• Retries are billed. An action with five retries is six executions. A playbook stuck retrying a dead connection is the one that costs money.

Checklist

• [ ] Every playbook is called from an automation rule, none from an analytics rule
• [ ] Conditions use tags or stable names; a rename cannot silence them
• [ ] Unique order numbers; auto-close rules have an expiry
• [ ] Sentinel service account has Automation Contributor on every playbook resource group
• [ ] Playbooks run as managed identity; no personal connections
• [ ] Graph / Sentinel roles granted to the identity and recorded
• [ ] Weekly synthetic incident proves the chain end to end
• [ ] Run history reviewed; any Failed run is a ticket

What changed recently

Checked against Microsoft Learn on 2026-10-28:

• My notes (2024) attach playbooks to analytics rules as one of two options. Learn now states that calling playbooks from analytics rules is deprecated as of March 2026 and should be migrated to automation rules (run-playbooks page updated 2026-07-01).
• The permission model is unchanged but better named: the Sentinel service account needs Microsoft Sentinel Automation Contributor on the playbook's resource group; the UI step is Manage playbook permissions and requires Owner on that resource group (automation rules page updated 2026-05-11).
• Defender-portal specifics are new since my notes: incident-created trigger only there, no alert trigger for Defender XDR alerts, 5 to 10 minute batching of incident updates, and "Simple flows" (preview) with case triggers.
• Update-trigger conditions gained state-change operators (changed from / changed to / added) and the tag any-vs-collection distinction.
• Logic Apps billing (page updated 2026-07-10): triggers are billed per execution even when skipped, retries and loop iterations count, disabled logic apps are free.

Sources

• Automate incident handling with automation rules
• Automate threat response with playbooks
• Run playbooks in Microsoft Sentinel
• Usage metering, billing, and pricing for Azure Logic Apps
• Azure Logic Apps pricing
• Entra ID Protection risk detections (password spray)
• Conditional Access: require MFA for all users
• App governance in Defender for Cloud Apps

News sources

• Microsoft Security Response Center, "Microsoft Actions Following Attack by Nation State Actor Midnight Blizzard" (19 January 2024): https://msrc.microsoft.com/blog/2024/01/microsoft-actions-following-attack-by-nation-state-actor-midnight-blizzard/
• Microsoft Security Blog, "Midnight Blizzard: Guidance for responders on nation-state attack" (25 January 2024): https://www.microsoft.com/en-us/security/blog/2024/01/25/midnight-blizzard-guidance-for-responders-on-nation-state-attack/
• CISA, Emergency Directive 24-02 (April 2024): https://www.cisa.gov/news-events/directives/ed-24-02-mitigating-significant-risk-nation-state-compromise-microsoft-corporate-email-system

Thanks for reading! I'm Cuong, and on CD Security Lab I rebuild real Microsoft security cases in a safe lab and explain them simply. Follow along:

• ▶️ YouTube (daily Shorts + explainers): CD Security Lab
• 💼 LinkedIn: Cuong Do
• 📘 Facebook: CD Security Lab
• 📸 Instagram: @cdsecuritylab
• 📝 More posts like this: Dev.to

Got a question about your own tenant? Drop it in the comments.