EdTech Report Deletion: A Production Upload Strategy for Large AI-Generated Files in Private Buckets

Short answer: for an edtech product serving generated reports to authenticated customers, use a private object-storage bucket, upload unusually large image files in numbered parts, and make retention and deletion state part of the application record. A complete upload is not permission to keep an object forever.

The least complex option is still a single upload for an image that is small and predictable enough to fit the service's normal request limits. Multipart upload earns its complexity when a report worker handles large outputs, unreliable links, or resumable work. It creates a second problem, though: the system now has an upload in progress, an object that may be complete, and possibly abandoned parts that are not yet an object. Those states need owners and deadlines.

In this system, the important question is not “can the bytes arrive?” It is “who may retrieve them, how long should they exist, and what proves that deletion happened?”

How should production teams handle large AI-generated images in private object storage?

Start with a database record for the generated report. Give it a stable generation ID, an object key that is not a customer-supplied filename, a retention deadline, and a state such as pending, uploading, ready, expired, or deleted. Store the multipart upload ID and the completion data for each numbered part beside that record. A worker can then resume the intended operation or close it deliberately after a retry, instead of guessing from a friendly filename.

The bucket should remain private. The application authorizes the customer, checks that the report is still within its retention window, and only then issues a short-lived signed read. The signed URL is a delivery credential, not a replacement for the authorization check. It must not be cached as a permanent link in a customer-facing record.

Keep generation and storage behind separate responsibilities. The generation worker produces image bytes and metadata; a storage adapter owns key allocation, multipart state, retries, completion, abort, and signed retrieval. This boundary matters during deletion. A customer deletion request should update the application state, revoke future access, enqueue object deletion, and record the result. It should not depend on a browser remembering to remove a URL.

Small rule: publication and retention are different transitions.

Why do multipart uploads create retention and deletion work?

Multipart transfer is a coordination protocol. A typical flow creates an upload, sends numbered parts, records the response associated with each part, and completes the upload with the intended set. If a worker disappears after part 7 of 12, the system has neither a usable report nor a reason to keep those partial bytes indefinitely.

I treat every multipart upload as a leased piece of work. The database stores its owner, creation time, last progress time, and terminal decision. A reconciler periodically finds records whose lease has expired. If the report is still valid, a replacement worker may continue the recorded upload; if the job was cancelled or the retention deadline passed, the reconciler aborts the upload and marks the record accordingly. The exact timeout needs measurement from the workload, and I'm not sure there is a universal value worth copying.

Deletion has two paths. A completed object needs an authenticated delete operation and an application state transition. An incomplete upload needs an abort operation. Normal object expiration rules do not automatically express every incomplete-upload cleanup policy, so the reconciler must account for both. The audit record should distinguish “deletion requested,” “object deletion confirmed,” and “partial upload aborted.” That distinction makes a retention review useful instead of ceremonial.

Retries deserve the same discipline. A 429 should use exponential backoff and honor Retry-After when supplied; a retry of a control operation should carry a stable idempotency key where the storage API supports one. I also record the part number, byte count, attempt number, and response status. A green worker log after an HTTP 204 is not evidence that the application published the right object, and a successful part response is not evidence that completion occurred.

What should the Node.js storage adapter prove before publishing an image?

The adapter should expose a small contract rather than leaking provider-specific calls through report generation code:

1. Create an upload for an application-owned key.
2. Transfer each part and persist its returned completion data.
3. Complete only the recorded upload and expected part set.
4. Mark the report ready only after completion succeeds.
5. Issue a signed read only after authorization and retention checks.
6. Abort or delete on cancellation, expiration, and failed terminal jobs.

The code below shows the transfer boundary in Python so the HTTP behavior is explicit; a Node.js implementation can preserve the same contract with its standard HTTP client. The presigned part URL carries the storage authorization for that transfer, so the application credential should not be added to the request.

import time
from pathlib import Path
from urllib.error import HTTPError
from urllib.request import Request, urlopen


def upload_part(url: str, part_path: str, attempts: int = 5) -> str:
    payload = Path(part_path).read_bytes()
    for attempt in range(attempts):
        request = Request(
            url,
            data=payload,
            method="PUT",
            headers={"Content-Length": str(len(payload))},
        )
        try:
            with urlopen(request) as response:
                if not 200 <= response.status < 300:
                    raise RuntimeError(f"part upload returned HTTP {response.status}")
                completion_value = response.headers.get("ETag")
                if completion_value is None:
                    raise RuntimeError("part upload did not return completion data")
                return completion_value
        except HTTPError as error:
            if error.code != 429 or attempt == attempts - 1:
                raise RuntimeError(f"part upload returned HTTP {error.code}") from error
            retry_after = error.headers.get("Retry-After")
            delay = float(retry_after) if retry_after and retry_after.isdigit() else 2**attempt
            time.sleep(delay)
    raise RuntimeError("part upload attempts exhausted")

The adapter must also make state transitions observable. Emit an event for upload creation, each completed part, final completion, abort, signed-read issuance, and deletion. Include a correlation ID and generation ID, but avoid putting customer content or signed URLs in ordinary logs. Alert on old uploading records, repeated completion attempts, and a growing gap between deletion requests and confirmed deletes.

Three words: prove each transition.

Which storage design fits an authenticated report service?

Compare designs by the control they give the team over retention, recovery, and access, rather than by the maximum advertised object size.

For generated reports, backend-mediated transfer is usually easier to reason about because the generation job already has an authenticated owner and a durable state record. Browser-direct transfer can reduce backend bandwidth, but it moves more lifecycle and authorization behavior to an environment the product does not fully control.

The catch is that a private bucket is not a complete retention policy. It does not tell the application whether an expired report may be restored, whether legal holds exist, or whether a deletion request covers derived thumbnails as well as the original image. A system that needs immutable retention, automatic geographic replication, server-side metadata search, or public URLs should choose an architecture that supplies those capabilities and should not hide the gap behind multipart terminology.

A deletion-first rollout for production

Begin with one report type and a short, explicit retention window. Write tests for cancellation before completion, worker replacement between parts, duplicate completion, an expired signed URL, a deletion request racing with retrieval, and a reconciler finding an abandoned upload. Use a fake storage adapter in unit tests, then run a small persistent-write test against the selected object-storage implementation.

During rollout, keep the old read path available until the new record reaches ready; do not expose an object key merely because generation succeeded. Make deletion idempotent, retain an audit event for the configured compliance period, and measure the age and count of incomplete uploads. If those metrics cannot be explained, the upload protocol is not operationally owned yet.

The decision rule is plain: choose single-request uploads for small, stable images; choose multipart for large or failure-prone transfers; choose neither as a substitute for a retention state machine. In an edtech report service, the customer should receive a readable image only after authorization, and the system should be able to explain why that image still exists.

References

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• https://cloudinary.com/documentation

Further reading

• https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
• https://cloudinary.com/documentation