The Northflank AI Key Notice Sent Me Auditing My Own Setup. Here Is the Order That Worked

I read the Northflank status page post twice before touching anything. On October 31, 2026 the cloud development platform said users had reported unauthorized use of AI API keys hosted on it, and that audit logs were now available to all customers, free tier included.

Nothing I run lives on Northflank. What kept me reading was that my own key layout has the same shape as the one being described. Provider credentials spread across environment variables, a couple of local tool configs, and a scheduled job I had not opened in months. So I spent an evening on the inventory and rotation I had been postponing. Below is the order I worked in, including the parts that turned out to be less obvious than they look.

What Northflank Is, and Why So Many Keys Sat There

A Kubernetes PaaS with a BYOC option

Northflank is a cloud-native platform-as-a-service provider built on Kubernetes. It builds and deploys microservices, managed databases, scheduled background jobs, and GPU workloads straight from a Git repository. Alongside the managed cloud it offers a Bring Your Own Cloud mode, where a company connects the control plane to its own AWS, Google Cloud, or Azure account, which removes the operational work of writing Kubernetes configuration files by hand.

Why LLM teams ended up storing credentials there

Teams running AI agents, backend API services, sandboxes, and automated data-processing scripts on Northflank need those containers to call external model APIs such as OpenAI and Anthropic. The usual place for the keys is workload environment variables, Secret Groups, or global secrets, next to database connection strings and third-party service tokens. Keeping credentials in one managed place is convenient, and it also means a platform-side problem puts every hosted model credential into the same exposure path.

What the Notice Said, and When

Three updates, three timestamps

• 13:58 UTC on October 30. Northflank told users to treat all environment variables as if they had been decrypted, and to disable or rotate keys. Later updates clarified that rotation should cover credentials in workloads, Secret Groups, and global secrets, and recommended turning off public access for any add-ons that had it enabled.
• 23:38 UTC on October 30. The platform said it had received user reports of unauthorized AI API key usage. It asked users to revoke existing credentials with the issuer, generate replacements, update the affected configuration and workloads, review usage and billing records with their AI providers, and report suspicious usage to both the platform and the providers involved.
• 01:04 UTC on October 31. Audit logs became available to all customers.

The company later confirmed that some users' AI API keys hosted on the platform had been used without authorization by outside parties.

Two misreadings I had to correct

The first is that asking users to act as if their environment variables had leaked is a baseline precaution in incident response, meant to contain risk quickly. It is not confirmation that every user's variables were stolen. The second is that reports from some users do not mean every customer on the platform saw unauthorized usage.

The Hacker News discussion drew a lot of attention and plenty of practical advice on checking credentials, mixed with speculation about the attack method, the platform's architecture, and related incidents. Community threads are not evidence of how the breach happened. The official notice mentions similar attack activity, but attacker identity and the full intrusion path stay open until the final investigation report lands. For a development team, the urgent question is not how the platform was compromised. It is where credentials are stored, what they connect to, and whether the old ones are actually dead.

Where the Blast Radius Actually Is

The inventory order I used

The real blast radius of a key depends on how many places it is deployed, not on what it was originally created for. A key made for testing models locally gets copied into production. A demo project goes away and its scheduled jobs keep running. Several agents share one key under the same provider account to skip setup work. Updating only the core production services leaves the peripheral environments untouched.

1. Platform configuration, both obvious and hidden. Following the official guidance, start with workload environment variables, Secret Groups, and global secrets. Then walk CI/CD build scripts, automated deployment pipelines, preview environments, and third-party webhook integrations to confirm none of them hardcode or cache the same key.
2. Match configuration to the instances actually running. One environment variable may be inherited by several microservices or hand-copied into different projects. Updating a frontend web service does not refresh background queue workers or cron jobs. Old containers nobody maintains but that are still running belong on the checklist.
3. Keep a redacted tracking sheet. Record each key's name, the prefix or ID the provider shows, the project it belongs to, where it is stored, which services use it, and its current status. Never paste a full key in plain text into the sheet, team chats, or ticketing systems. That creates new leak sources inside the collaboration tools.
4. Check permissions on the provider side. Log in to the console for OpenAI or Anthropic, confirm which organization and project the key belongs to, then look at which model endpoints it can call and whether it can read or write fine-tuning data or upload files. Permission granularity varies widely between providers. An API key is not automatically limited to text generation, and it is not automatically full admin either.
5. Sweep the other infrastructure credentials in the same environment. Database connection strings, object storage keys such as AWS S3 or Google Cloud Storage, and payment API tokens all rotate alongside the model keys if they lived in the affected runtime.

Rotation, and Why Deleting the Variable Does Nothing

Revoke, generate, deploy, verify both directions

Deleting an environment variable removes a string from the platform's database. If the key was already copied to an outside server, whether it can still call the model depends on the provider. The same holds for deleting a local .env file or scrubbing old code from Git history.

• Revoke the old credential in the provider console. Find the key, revoke or delete it, and confirm its status shows it is no longer active. If suspicious calls are coming in right now, cut the credential immediately to limit the charges, while weighing a brief service interruption and preparing a recovery plan.
• Generate the replacement from a trusted device. Reach the console through the official domain, check the team member list and role permissions for unfamiliar changes, then generate the new key under least privilege. If the cloud platform has not finished its isolation, remediation, or migration work, hold off putting new production keys back into the high-risk environment.
• Update every configuration and restart the instances. Backend services and long-lived processes read environment variables once, at container start. Saving a change in the web console does not change the process already in memory, which keeps sending requests with the old key until it is restarted or redeployed.
• Verify both directions. The old key must be fully rejected by the provider, and the services must be making calls successfully with the new one. Never print the new key into application logs, and do not paste a curl test command containing the full key onto a shared server where shell history captures it.

Keep watching for a while afterwards. Occasional 401 Unauthorized errors in the logs mean some old instance or scheduled job has not picked up the new configuration. Unexplained usage on the bill usually means the account has other backup keys nobody dealt with.

Reading Audit Logs Against Provider Bills

Four steps that produce a usable timeline

Audit logs open a view inside the platform. Platform logs and model provider logs verify different things, so the two have to be pieced together. If a key is exfiltrated and used from machines outside the platform, the original application's service logs show nothing at all, and an empty audit log does not rule out abuse. Confirm first how far back the logs go and which fields are retained. From a forensics standpoint, a record that does not appear is not the same as an event that never happened.

Step 1, put everything on one time zone. Convert platform audit logs, application logs, and provider billing timestamps to a single zone such as UTC, and mark the window where the charges started to spike.

Step 2, compare workloads with call patterns. When the bill jumps, check whether the team shipped a release, ran a batch job, or changed a scheduled script in the same period. Then compare the models being charged with the models the code uses. A service that normally calls only lightweight models suddenly hammering expensive flagship models at 3 a.m., or a provider still billing after the containers were shut down, points to someone outside.

Step 3, rule out cost spikes from internal code. An agent stuck in an infinite loop, aggressive retries after API timeouts, misconfigured concurrency, or a batch document-parsing job without rate limiting can burn through quota in minutes. Overlay job execution records on the provider's token consumption chart so an internal bug is not mistaken for an external breach.

Step 4, break the bill down by key or project and preserve evidence. Where the provider allows filtering usage by API key or project, filter down to the affected credentials. Where only an account-level summary exists, narrow it by release times and application logs. Once unauthorized usage is confirmed, export the abnormal time window, Project ID, Request ID, models called, token usage details, and the exact revocation time. A structured timeline moves a ticket forward far more effectively than reporting that the balance was drained. Redact business data from screenshots before submitting, and never attach full old or new keys in plain text.

Keep reviewing bills for the next two or three days. Some providers' usage dashboards lag final billing by several hours or a day or more, and for charges that show up later, check whether the underlying requests happened before or after the revocation. Whether a credit or refund is available for leak-caused charges depends entirely on each provider's risk policies and how they assess the incident, so plan on absorbing losses that already happened.

What I Changed So the Next Notice Is Cheaper

Split credentials by boundary

Development, testing, and production keys stay strictly isolated. Each microservice and each standalone agent gets its own project and key. Where the provider supports scoped keys, enable only the model endpoints the service needs and turn off fine-tuning, file upload, and organization management. Every key gets a named owner. Idle keys are revoked when a temporary project retires, console access is removed when people leave or change roles, and old containers get shut down after a migration so stale credentials stop piling up.

Put virtual keys in front of local projects

Moving debugging back to a laptop does not make credentials safe. Real OpenAI or Anthropic keys written in plain text into several .env files, or into the configs of different CLI tools, get committed when a .gitignore entry is missing, and get picked up by debug logs and cloud backups.

This is where ServBay, an AI development management tool, fits into my setup. Real provider keys stay inside its AI Gateway, and each project or tool, Claude Code, Codex, or an in-house script, gets a separate virtual key pointing at the gateway endpoint. Application configs hold only virtual keys, so the upstream credential never sits in a project directory or a terminal environment variable. A virtual key can be revoked on its own without touching the real key in the provider console, and usage and cost per project are visible on the gateway side.

A local gateway still needs its own boundary check. Run the full workflow with a disposable test credential, then search local log files, build output directories, and test deployment configs for spots where request headers get printed in plain text.

Know where spending controls stop

A budget number in a provider console reads like a guarantee and is not one. Different spending controls trigger in different ways and follow different billing rules, so a team can believe it has a limit it never actually had. Worth checking before the next incident rather than during it.

The Checklist I Keep in a Note

Four steps, in the order I run them

• Cut off access first. On confirmed or strongly suspected exposure, revoke the old key in the provider console. Deleting the cloud environment variable or the local config file is not revocation.
• Inventory every location. Trace the key through environment variables, Secret Groups, CI/CD pipelines, scheduled background jobs, and peripheral test containers. Generate the new key from a trusted device and restart every running instance.
• Cross-check the bills. Put platform audit logs, application job records, and the provider console on one time zone, separate internal loops from external abuse, export request details for the abnormal window, and keep checking for delayed charges.
• Isolate environments and real credentials. Separate keys for development, testing, and production, turn off unnecessary auto-recharge in the provider console, and for local development keep real keys behind virtual ones with per-project assignment, revocation, and separate usage tracking.