A discovery runbook for shadow AI, using logs you already have

Most advice about shadow AI starts by telling you to survey your staff. Try it and you get a partial list, because people answer honestly about the tools they think count and forget the browser extension that summarises their email.

The better starting point is that you are already collecting the evidence. Every company of fifty people has four or five telemetry sources that know exactly which AI services its laptops and servers talk to. Nobody has pointed them at this question yet.

Here is a discovery runbook that uses what you already have, in rough order of how much it returns per hour spent.

1. Decide what you are actually looking for

Write this down before you query anything, because otherwise the list balloons and you abandon it.

You are looking for three distinct things, and they need different responses:

Services that receive your data. A person pastes a customer support thread into a chat assistant, or a tool sends document contents to a provider to summarise them. This is the category that matters for confidentiality and for your customer commitments.

Services that write code or config that ships. An assistant in the IDE, a bot that opens pull requests, a pipeline step that generates something. The risk here is provenance and review, not disclosure.

Agents with credentials. Anything that holds a token and acts on its own: a scheduled job, an integration with repository write access, an assistant connected to your ticketing system. Small in number, largest in blast radius, and almost never on anybody's list.

A tool can be in more than one category. The point is that "we use AI" is not a finding, and "this tool holds a repo-scoped token and sends ticket contents to a third party" is.

2. Start with DNS and egress logs

This is the highest yield hour in the whole exercise.

Your resolver, firewall, or cloud egress logs already record which hostnames your network talks to. Query them for the API hostnames of the major model providers and the common wrapper services, over a ninety day window, grouped by source host and by volume.

Three patterns worth reading carefully:

Steady low-volume traffic from one workstation is usually one person with a browser extension or desktop app.

Traffic from a server or container, not a laptop, means something in your own stack is calling a model. That is a product decision somebody made, and it belongs in your architecture documentation whether or not anyone wrote it down.

A spike that starts on a specific date and never stops is an integration that got switched on. Find the date, then find the change that landed that week.

If you have no egress logging at all, that is a finding in its own right and probably a bigger one than the AI question.

3. Pull the OAuth and connected-app lists

Every place a person can click "allow" is a place an AI tool can attach itself to your data without anybody buying anything.

Check the connected applications list in your identity provider, the installed apps and OAuth grants in your source control organisation, the add-ons and marketplace apps in your document suite, and the integrations in your chat tool and ticketing system.

For each one, record the scopes rather than the name. An assistant with read access to a calendar is not interesting. The same vendor with read and write access to every repository is a different conversation. Scopes are the only part of this that tells you anything about consequence.

4. Look at the endpoints, carefully and with the lightest touch that works

Browser extensions and desktop applications are where most individual usage lives.

If you have device management, an installed-application inventory and a browser extension inventory will answer this in minutes. If you do not, you can ask, and you will get a partial answer.

Be deliberate about proportionality here. There is a real difference between inventorying installed software and reading what people type, and the second one damages the trust you need for the rest of this programme to work. You are building a list of tools, not monitoring individuals. Say that out loud when you ask, and mean it, because the moment this feels like surveillance people route around you and your inventory becomes fiction.

5. Read the expense data last, not first

Card statements and the subscriptions list will surface paid tools, which is useful and incomplete. Almost everything interesting in shadow AI is on a free tier, which is precisely why it never passed through procurement.

Do pull it, because reconciling it against the DNS findings is informative in both directions. A tool you are paying for that generates no traffic is a cancellation. Traffic to a service you are not paying for is someone on a free tier, and free tiers are frequently the ones whose terms allow training on your inputs.

6. Grep your own codebase

This step takes one command and is skipped constantly.

Search the repositories for provider SDK imports, model API base URLs, and environment variable names in the shape of provider keys. Then check which of those live in code that is actually deployed, because a proof of concept in a branch is not the same as a production dependency.

While you are there, check whether those keys are scoped and rotated like your other secrets or whether somebody pasted a personal key into a config file in a hurry. That last one is usually where this exercise pays for itself.

7. Write the findings as rows, not as a report

Resist the narrative write-up. You want a table that stays current, with one row per service and five columns: what it is, what data reaches it, how it authenticates, who owns it here, and whether it is approved.

That last column is the one that turns discovery into governance. It only has three honest values: approved, not yet reviewed, and not allowed. A row sitting at "not yet reviewed" for three months is a decision nobody is making, and that is worth seeing on the page.

Keep the table in version control for the same reasons any inventory belongs there: review, history, and a diff that tells you when something appeared.

8. Decide what happens the next time, before there is a next time

Discovery is a one-off. The thing that keeps it from being a one-off is a route that is easier than going around you.

In practice that means two things. A named way to ask, with a response measured in days rather than committee cycles, because a two week approval queue guarantees people will not use it. And a default answer for the low-risk case, so that most requests resolve without a meeting.

If you want a reference for what the governed end state looks like, IRM's free AI governance playbook sets out a 90 day plan mapped to NIST AI RMF and ISO 42001, which is a reasonable shape to aim at even if you never certify against either.

About IRM's AI Governance services

Disclosure first, so you can weigh the advice accordingly: this post comes from IRM Consulting & Advisory, and the links in it point to our own pages.

Shadow AI discovery is usually the first week of an AI governance engagement, and it is first for a reason. Every later decision, about acceptable use, about vendor review, about what gets logged, depends on knowing what is already running. Teams that skip it write policies for a company they imagine rather than the one they have.

IRM Consulting & Advisory provides AI governance services to SaaS companies, startups and SMBs across Canada and the United States, aligned to the standard frameworks rather than invented ones.

One question, since the answers split sharply: when discovery turns up a tool a team genuinely depends on and nobody approved, do you switch it off while you review it, or leave it running and review it in place? The first is defensible and the second is what mostly happens.