AI Image App Storage Pattern: Originals, Thumbnails, Private Signed Links, and Cleanup
Short answer: keep generated AI images in private object storage, treat every original and derived file as immutable, authorize access before issuing a brief signed link, and clean up only data classes with an explicit r
Short answer: keep generated AI images in private object storage, treat every original and derived file as immutable, authorize access before issuing a brief signed link, and clean up only data classes with an explicit recovery and retention policy.
The tempting first version is one bucket, one predictable filename, and a public thumbnail URL. It works in a notebook because there is one user, one worker, and no delete queue. Production changes the question. A generated image has bytes, ownership, publication state, transform history, and a retention obligation; putting all of that meaning into a path turns storage into a fragile second database.
This pattern is deliberately boring. It is also easier to evaluate: inject a retry between upload and the database write, then verify that one visible asset, rather than two, remains.
How should an AI image generation app handle originals, thumbnails, variants, private buckets, signed links, and lifecycle cleanup?
Use a database record as the authority for asset state and object keys as immutable locators. A useful record holds an asset identifier, tenant identifier, media type, dimensions, checksum, current derivative recipe, deletion state, and the keys it owns. The object store holds bytes; it does not decide who can view them.
Separate security and retention boundaries into buckets only when their policies truly differ. Within a boundary, purpose-based prefixes make lifecycle rules and inventory easier to reason about: staging/, originals/, and derivatives/ are enough for many systems. Do not put email addresses, prompts, or display names in a key. Paths wind up in logs, inventories, and debugging tools.
An original should remain unchanged after it is accepted. A thumbnail or other variant should also get a new key when its transform recipe changes. Replacing thumb.webp creates an awkward race between caches, readers, and workers. Writing derivatives/{asset_id}/thumb-v4/{digest}.webp lets the application select the active rendition in its database while older bytes have a clear cleanup path.
There is a real trade-off. Immutable keys add reconciliation work and can temporarily retain more bytes. They are not suitable for a workflow that needs POSIX semantics or frequent in-place edits to huge working files; use a filesystem-oriented working area there, then export completed deliverables. For a disposable single-machine prototype, a local directory can be the sensible starting point until multiple workers, durability, or controlled sharing become requirements.
Make writes retryable before making them visible
Generation and image processing will be retried. The storage design should make a retry converge instead of creating a second public-looking file. A worker can write to a unique staging key, validate the result, place bytes at a deterministic final key, and commit the database transition. If the process repeats, the same key and state check provide a decision point.
The important failure window sits between the object write and the state commit. If the worker writes the original, starts a thumbnail transform, and exits before the asset row becomes publishable, the bytes are real but no caller should discover them. Re-running the job must first inspect the durable state, then either reuse a validated final key or resume the missing transition. A database transaction cannot roll back an object write, so the application needs an explicit compensating rule: retain the object as recoverable staging until it is attached, or mark it as an orphan candidate for a delayed reconciler. Do not delete it immediately on a timeout. A delayed cleanup window gives an independent retry time to complete, while the database remains the decision-maker for visibility. The same reasoning applies in reverse for deletion: remove authorization first, record the deletion transition, and let an asynchronous worker remove derived objects and originals according to the policy. This ordering prevents a link-issuance request from racing a cleanup job and makes the remaining mismatch observable rather than silently serving a supposedly deleted image.
The following Python boundary is intentionally small. A cloud object store, a self-hosted implementation, or a test double can satisfy it, which keeps a notebook experiment from dictating the production client.
from dataclasses import dataclass
from hashlib import sha256
from typing import Protocol
class ObjectStore(Protocol):
def put(self, *, key: str, data: bytes, content_type: str) -> None: ...
@dataclass(frozen=True)
class ImageKeys:
original: str
thumbnail: str
def immutable_key(asset_id: str, kind: str, recipe: str, data: bytes, suffix: str) -> str:
digest = sha256(data).hexdigest()
return f"{kind}/{asset_id}/{recipe}/{digest}.{suffix}"
def store_image_pair(store: ObjectStore, asset_id: str, original: bytes, thumbnail: bytes) -> ImageKeys:
original_key = immutable_key(asset_id, "originals", "source-v1", original, "png")
thumbnail_key = immutable_key(asset_id, "derivatives", "thumb-v4", thumbnail, "webp")
store.put(key=original_key, data=original, content_type="image/png")
store.put(key=thumbnail_key, data=thumbnail, content_type="image/webp")
return ImageKeys(original=original_key, thumbnail=thumbnail_key)
The code alone is not the protocol. Put a uniqueness constraint around the durable asset transition, record the checksum, and make publication depend on that transition. A key proves that bytes exist. It does not prove that a particular session may read them.
Measure the failure paths before copying the layout: upload-to-record latency, retry count, duplicate final keys, objects without a record, records without an object, and generation-to-thumbnail delay. Median latency can look calm while queueing makes the long tail painful. For AI applications, track regeneration separately from storage cost, because a cleanup rule that saves bytes can create more model work and a worse user wait.
Signed delivery belongs after authorization
Keep the bucket private. On a view or download request, the application authenticates the caller, checks the asset's tenant and sharing state, selects the exact key and permitted method, then creates a short-lived signed URL. The signing endpoint must not accept an arbitrary object key supplied by the browser.
Small rule, large consequence.
Don't sign first and ask questions later.
Signed URLs are credentials in a query string. Avoid logging the complete URL or signature, and log an asset ID, authorization decision, request correlation ID, and expiry policy instead. Link lifetimes depend on object size, networks, and whether the interaction is a preview or download, so a fixed duration is not universal. An authenticated refresh endpoint is usually clearer than a long-lived link. Highly sensitive media may warrant an authenticated proxy, accepting the added bandwidth and operational cost.
The useful test matrix is compact: a different tenant, an expired link, a changed request method, an asset deleted in the database while its bytes still exist, and a valid link after the user's session is no longer authorized. These checks belong in an integration harness alongside image-format and prompt-cost evaluations. Access behavior is product behavior.
Cleanup is a retention decision, not a cron afterthought
Start lifecycle cleanup with objects whose recovery story is obvious. Abandoned staging uploads and failed intermediate transforms are common candidates. Regenerable thumbnails may be candidates too, but their expiration window should account for the queue, compute, and model demand required to recreate them. Originals need a separate policy tied to user deletion, retention commitments, and any legal obligations.
Object-storage lifecycle features can transition or expire objects, but the exact controls and versioning behavior differ by implementation. Write policy in application terms first: which class may expire, after what event or age, who approves an exception, and how a deletion request propagates. Then test the selected store's rule against versioned objects and the storage classes in use. Your mileage may vary when provider semantics or compliance requirements differ; the policy document and a disposable test bucket resolve that uncertainty.
A bounded reconciler complements lifecycle configuration. It scans one prefix and age range at a time, compares candidates with durable asset state, reports counts and bytes, and quarantines uncertain items for review before deletion. Its dry-run output should include sample keys and each eligibility reason. This is less dramatic than a global age rule, and much safer for a library of generated images whose variants are cheap while originals are not.
Before rollout, practice deletion and recovery in a non-production boundary. Confirm that a user delete removes the application's ability to issue new links, that cleanup cannot reach retained originals through a broad prefix, and that an interrupted worker leaves a trace a reconciler can classify. That evidence is more useful than a pretty bucket diagram.
References
- AWS S3 documentation, Object lifecycle management: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
- Google Cloud Storage documentation: https://cloud.google.com/storage/docs
Further reading
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.