Dev.to AI πŸ€– Ai πŸ‘ 0 πŸ“– 2 min read

Observability without prompt storage: securing LLM API keys in production

When you ship an LLM feature, the default instinct is to log full prompts and completions. It makes debugging easy, but it also builds a store of user content you now have to secure and expire. For most cost and reliabil

When you ship an LLM feature, the default instinct is to log full prompts and completions. It makes debugging easy, but it also builds a store of user content you now have to secure and expire. For most cost and reliability questions you don't need it.

1. Log metadata, not content

Request ID, model, input/output tokens, status and latency answer "why did spend jump?" without retaining user text:

{ "request_id": "req_123", "key": "prod-web", "model": "oroo-chatgpt", "input_tokens": 412, "output_tokens": 188, "status": 200, "ms": 940 }

(Example log shape only; values are illustrative.)

2. One key per app and environment, each with limits

Separate keys make usage attributable and shrink the blast radius. Set allowed models, an output-token cap and requests-per-minute per key so a runaway loop fails fast.

3. Treat key secrets like passwords

Server-side only, never in a client bundle. Show the secret once, store only a hash, rotate on a schedule.

4. IP allowlists for production keys

A leaked key that only works from your own servers is far less useful to an attacker.

5. A one-minute freeze procedure

Document how to freeze, rotate or revoke a key and make sure more than one person can do it.

6. Idempotent retries

Send a new Idempotency-Key for every new request and reuse it only when retrying that request, so a network retry doesn't trigger a second billable model call.

Common mistakes to avoid

  • Shipping a key inside a mobile app "temporarily"
  • Sharing one key between staging and production
  • Retry loops with no ceiling and no idempotency
  • Keeping raw prompts in logs forever because nobody set a retention period
  • Only one person knowing how to revoke a key

Each of these is cheap to fix before launch and expensive to fix after an incident.

Where a gateway helps

I work on OROO Blue, a secure AI gateway for Indian teams that bakes these patterns in: one API and one OROO key for OpenAI, Google and Anthropic models; per-key usage and activity; secrets shown once and stored as a hash; optional IP allowlists; freeze/rotate/revoke; idempotent replays that aren't charged again; and OROO does not store prompts or completions. Billing is a prepaid INR wallet.

The patterns above apply whatever stack you use.

Full write-up: https://orookids.blogspot.com/2026/11/usage-without-secrets-6-privacy-habits.html
OROO Blue: https://orooblue.com

πŸ“° Read the original article on Dev.to AI

Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.