Observability without prompt storage: securing LLM API keys in production
When you ship an LLM feature, the default instinct is to log full prompts and completions. It makes debugging easy, but it also builds a store of user content you now have to secure and expire. For most cost and reliabil
When you ship an LLM feature, the default instinct is to log full prompts and completions. It makes debugging easy, but it also builds a store of user content you now have to secure and expire. For most cost and reliability questions you don't need it.
1. Log metadata, not content
Request ID, model, input/output tokens, status and latency answer "why did spend jump?" without retaining user text:
{ "request_id": "req_123", "key": "prod-web", "model": "oroo-chatgpt", "input_tokens": 412, "output_tokens": 188, "status": 200, "ms": 940 }
(Example log shape only; values are illustrative.)
2. One key per app and environment, each with limits
Separate keys make usage attributable and shrink the blast radius. Set allowed models, an output-token cap and requests-per-minute per key so a runaway loop fails fast.
3. Treat key secrets like passwords
Server-side only, never in a client bundle. Show the secret once, store only a hash, rotate on a schedule.
4. IP allowlists for production keys
A leaked key that only works from your own servers is far less useful to an attacker.
5. A one-minute freeze procedure
Document how to freeze, rotate or revoke a key and make sure more than one person can do it.
6. Idempotent retries
Send a new Idempotency-Key for every new request and reuse it only when retrying that request, so a network retry doesn't trigger a second billable model call.
Common mistakes to avoid
- Shipping a key inside a mobile app "temporarily"
- Sharing one key between staging and production
- Retry loops with no ceiling and no idempotency
- Keeping raw prompts in logs forever because nobody set a retention period
- Only one person knowing how to revoke a key
Each of these is cheap to fix before launch and expensive to fix after an incident.
Where a gateway helps
I work on OROO Blue, a secure AI gateway for Indian teams that bakes these patterns in: one API and one OROO key for OpenAI, Google and Anthropic models; per-key usage and activity; secrets shown once and stored as a hash; optional IP allowlists; freeze/rotate/revoke; idempotent replays that aren't charged again; and OROO does not store prompts or completions. Billing is a prepaid INR wallet.
The patterns above apply whatever stack you use.
Full write-up: https://orookids.blogspot.com/2026/11/usage-without-secrets-6-privacy-habits.html
OROO Blue: https://orooblue.com
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.