Private Object Storage Thumbnails — Signed Image URLs and Retention in 3 Stages
For a fintech application accepting large image uploads, start with a private bucket, a narrowly scoped signed upload URL, and a thumbnail worker. The bill has several parts: retained bytes over time, storage requests, t
For a fintech application accepting large image uploads, start with a private bucket, a narrowly scoped signed upload URL, and a thumbnail worker. The bill has several parts: retained bytes over time, storage requests, transformation work, and data delivery. Quantify retention first. In an illustrative workload of 1,000 uploads a day, each with a 20 MB original and two 0.2 MB thumbnails, a 30-day window holds 600 GB of originals and 12 GB of thumbnails. These are planning assumptions, using decimal units, not measurements or quoted prices.
Short answer: send the original directly from the client to private object storage, authorize thumbnail generation through application metadata, and issue signed download URLs only after checking access. Give originals, derivatives, and abandoned uploads separate retention rules. Shortening the original's retention window moves the largest retained-byte term in this example; resizing alone barely touches it.
Keep deletion explicit.
What actually changes the retention bill?
With a steady arrival rate and no other retained copies, daily object volume multiplied by retention days gives the steady-state logical storage footprint. The example above deliberately ignores versions, replication, incomplete uploads, and temporary processing files. Account for those separately before treating logical bytes as billed bytes. The lifecycle documentation in References describes expiration, version handling, and incomplete multipart upload cleanup; the applicable behavior depends on the storage configuration.
Here is the same hypothetical workload with one policy change:
| Object class | Daily logical volume | Initial retention | Revised retention | Revised steady-state volume |
|---|---|---|---|---|
| Originals | 20 GB | 30 days | 7 days | 140 GB |
| Two thumbnails per original | 0.4 GB | 30 days | 30 days | 12 GB |
The original volume falls from 600 GB to 140 GB. That is a change in retained data, not a promised bill reduction. Request volume, delivery traffic, temporary files, and the storage system's billing rules still matter. A frequently viewed thumbnail can contribute little stored volume while generating substantial delivery activity; determine its importance from the workload rather than its dimensions.
Seven days is an example, not a fintech compliance requirement. Before adopting it, identify what the original proves, who owns the retention decision, and whether an applicable obligation or hold requires keeping it longer. A thumbnail may be sufficient for a dashboard yet unsuitable for reviewing the underlying evidence. Make that distinction before the upload endpoint exists.
There is a direct engineering trade-off: fewer retained originals mean fewer inputs available for later reprocessing. A decision to retain every original indefinitely buys that option with accumulating storage and a larger deletion obligation.
How do signed URLs serve private image thumbnails from object storage?
Use three stages: authorize an upload, finalize and transform the accepted object, then authorize delivery. Keep the application in charge of permissions and metadata while storage handles the file transfer.
At authorization time, have the application check the authenticated user, tenant, purpose, and permitted upload size. Allocate an opaque object key and record a pending asset with its intended retention policy. Return a short-lived signed upload URL limited to the intended operation and object. Treat every restriction as something to verify against the chosen storage implementation: don't assume a field in application metadata constrains a request made directly to storage.
The client then uploads the bytes to storage. For a browser client, configure the required cross-origin access rules too; successful signing doesn't establish browser permission. The application should receive a completion signal and verify the stored object against its pending record before marking it accepted. A client-supplied filename or completion message should not become proof that the intended bytes arrived.
Private means every delivery needs an access decision in this design. After checking tenant membership and asset state, issue a signed download URL for the specific original or thumbnail. Treat that URL as a temporary credential. Keep it out of application logs, notification text, analytics events, and support screenshots.
For a team building email, SMS, or OTP flows, this deserves the same care as a verification link. An asset identifier in a notification can lead to an authenticated application page; a reusable media credential embedded in the message creates another exposure path. Track notification delivery separately from asset readiness so a retry in one system cannot silently authorize access in the other.
Test expiration directly. Also test the tenant that knows an object identifier but has no permission to view it.
Put the thumbnail worker behind an asset state check
The familiar Node.js and Sharp worker pattern fits this architecture as a transformation implementation: the application coordinates access, and a worker reads the original and writes resized derivatives. The retention contract should remain outside the image library. Changing a resize implementation must not change who can read an asset or how long it survives.
Define a small set of approved thumbnail profiles rather than accepting arbitrary dimensions from a download request. For example, a review screen and a compact activity feed can use separate profiles with fixed output requirements. Include the profile revision and source revision in derivative metadata so a worker can identify what it produced. Avoid putting account names, payment descriptions, or other sensitive context into object keys.
An edge case worth designing first is deletion during processing. A worker can read an accepted original, spend time resizing it, and finish after the user has requested deletion. A check at the start of the job is insufficient. Have the worker check the current asset state before publishing its result, and make the deletion process account for an output that arrives during that transition.
This is a coordination problem.
One proposed contract is to mark the asset as deleting, stop new delivery authorization, and prevent workers from publishing against that asset revision. Then delete the recorded object set and reconcile any late output. Storage operations and application transactions are separate steps in this design, so include reconciliation rather than treating one database update as proof that all bytes disappeared.
Duplicate jobs deserve similar attention. Give repeated processing of the same source revision and profile a deterministic destination or an equivalent deduplication rule. Mark the derivative ready only after verifying its write. A timeout after a successful write should lead to reconciliation, not an unbounded sequence of additional objects.
Bound the worker's resources as well. An upload size limit and a decoded-image resource limit serve different purposes. Specify acceptable formats, decoded dimensions, processing time, and temporary storage use, then test rejection behavior with the selected decoder. The application should expose a useful failed-processing state without returning storage credentials or internal file paths.
Deletion needs a ledger and an observable finish
Give each asset record an owner, purpose, retention deadline, hold status, source revision, and inventory of derivatives. These are proposed application fields, not a storage API schema. They let the team answer a concrete operational question: which objects still need to be removed for this asset?
Use lifecycle rules as a backstop for object classes with predictable retention. The lifecycle reference explains why expiration rules, noncurrent versions, and incomplete multipart uploads need distinct attention. Don't equate a lifecycle deadline with an application-level deletion receipt.
For an explicit deletion request, first make the asset unavailable to new authorized reads, then track removal of the original, derivatives, and relevant temporary data. If versioning or caching is enabled, include those systems in the deletion contract. A hold requires an explicit policy decision; it should not be indistinguishable from a failed delete.
Measure the gap.
Useful operational signals include the age of the oldest pending upload, processing delay, deletion backlog age, and objects found without an active asset record. Alert thresholds should come from the product's stated deadlines. A completed cleanup request is less informative than evidence that the expected object set is gone.
Avoid logging image contents or full signed URLs to diagnose those signals. Use opaque asset identifiers and restricted operational records. An incident responder needs enough evidence to reconstruct authorization and deletion decisions without turning telemetry into an extra media archive.
Before deployment, exercise an interrupted upload, duplicate completion, worker retry, deletion during resizing, expired download authorization, and a retention hold. Roll out a changed deletion rule to a bounded object population, reconcile the results, and expand only after verifying which records and objects it affected. This takes more coordination than adding an expiration rule, but it gives operations a reviewable result.
Decide what the system will stop keeping
The example policy stops keeping eligible originals after seven days and keeps their thumbnails for thirty. It also requires a separate short cleanup window for uploads that never become accepted assets. The actual deadlines belong to the retention policy owner, with legal and operational review where needed.
The sacrifice is specific. Once an original has been deleted, a later decoder fix or new thumbnail profile cannot regenerate it. A thumbnail may help an incident responder understand the screen a reviewer saw, but it cannot restore the source image. If source fidelity is required for an investigation, the retention or hold decision must happen before deletion.
Keep the decision record for an independently justified period, with its own access controls and deletion rule. That record can explain which policy authorized removal without becoming a hidden copy of the image.
Direct uploads solve the application bandwidth problem. A private delivery boundary and an explicit retention contract make the resulting media system operable. The most consequential choice is what evidence the team deliberately gives up, and whether it can defend that choice when something goes wrong.
Further reading
References:
- Object lifecycle management: expiration, version handling, and incomplete upload cleanup. https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html
- Storage pricing reference for checking which usage dimensions enter a bill; no rates are assumed in the example. https://www.backblaze.com/cloud-storage/pricing
Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.