Logistics Report Retention: Browser-to-Object Storage with Presigned Uploads
Short answer: for a logistics report that must survive an ordinary customer connection, use a presigned PUT for a small object and a presigned multipart workflow for a large one; use POST when a form policy is the requir
Short answer: for a logistics report that must survive an ordinary customer connection, use a presigned PUT for a small object and a presigned multipart workflow for a large one; use POST when a form policy is the requirement. The more important decision is retention: keep an upload private and addressable until the report is authenticated and validated, then delete it on a recorded schedule rather than treating a successful upload as the end of the job.
That answer fits the way I build Python RAG and agent features. The browser should move bytes directly to object storage. The Python service should authorize the upload, write a durable report record, evaluate the completed object, and issue a short-lived download URL. Keeping the report bytes out of the API process makes the notebook-to-prod path smaller, but it also makes ownership explicit: the browser owns transfer, the application owns report state, and storage owns object bytes.
A report is not ready just because a request returned 200.
The awkward case is a dispatcher who uploads a generated route report from a hotel network, closes the laptop when the progress bar reaches 68 percent, and opens the customer portal again the next morning. The portal must not create a second report record just because the browser lost its local state. It should find the existing uploading record, see whether its presigned capability is still valid, and either resume the known parts or issue a new plan for the same application-owned report. If the retention deadline passed overnight, resuming is the wrong action: mark the attempt expired, abort or remove its temporary storage state, and ask the user to generate a fresh report. That decision belongs to the report state machine, not to whichever callback happened to run last in the browser. In an eval harness, I would assert that the old object cannot become customer-visible after the replacement plan completes, that a duplicate completion cannot extend retention silently, and that the cleanup worker has enough identifiers to remove both the abandoned attempt and any rejected final object. This is where a direct upload stops being a file-transfer trick and becomes part of an authenticated customer workflow.
Keep that boundary visible.
What should a browser upload workflow do with large logistics reports?
Start with one application record per report. Give it a non-guessable object key, the customer identity, the expected byte count, a content type, a retention deadline, and a state such as created, uploading, uploaded, validated, or deleted. The backend returns an upload plan only after it has authenticated the customer and checked that the requested report belongs to that customer. The browser uploads directly, then tells the backend which report it attempted. A worker verifies the object before it enters parsing, chunking, embedding, or retrieval evaluation.
This order prevents a common logistics failure: a customer can see a progress bar for a report that the AI pipeline has never accepted. It also gives deletion a subject. A lifecycle rule can remove old bytes, but the application still needs to know whether it is deleting an abandoned upload, an invalid report, or a report whose customer-visible retention period has ended. Those are different audit events.
For a small report, a single presigned PUT is usually the easiest contract. There is one request, one progress stream, and one whole-object retry. For a large report, multipart upload changes the unit of recovery from the whole object to a part. If part 8 fails, the browser can retry part 8 instead of sending parts 1 through 7 again.
Presigned POST is a different shape. It gives the browser a form submission plus policy conditions, which can be useful when the storage integration already standardizes on form fields and constrained keys. It is not a universal upgrade over PUT. Choose it because its policy model fits the application, not because the word βpresignedβ makes it more resilient.
A Python upload plan that keeps report state durable
The code below is intentionally transport-neutral. A trusted Python API has already produced upload-plan.json; its schema is owned by the application and contains a part size plus presigned URLs. The browser would normally execute the same bounded loop in JavaScript, but Python makes the state transitions easy to inspect in an eval harness. The presigned URL carries the storage authorization, so the application credential never goes to the browser.
import json
import os
import random
import time
from pathlib import Path
from urllib.error import HTTPError
from urllib.request import Request, urlopen
def retry_delay(error: HTTPError, attempt: int) -> float:
retry_after = error.headers.get("Retry-After")
if retry_after and retry_after.isdigit():
return float(retry_after)
return min(30.0, (2 ** attempt) + random.random())
def put_part(url: str, data: bytes, attempts: int = 5) -> str:
for attempt in range(attempts):
request = Request(url, data=data, method="PUT")
try:
with urlopen(request, timeout=120) as response:
etag = response.headers.get("ETag")
if not etag:
raise RuntimeError("The completed part has no ETag")
return etag
except HTTPError as error:
if error.code == 429 and attempt + 1 < attempts:
time.sleep(retry_delay(error, attempt))
continue
raise RuntimeError(
f"Part upload failed with HTTP {error.code}"
) from error
raise RuntimeError("The part retry budget was exhausted")
def upload_file(plan_path: str, file_path: str) -> list[dict[str, int | str]]:
plan = json.loads(Path(plan_path).read_text())
completed = []
source_path = Path(file_path)
with source_path.open("rb") as source:
for part in plan["parts"]:
payload = source.read(plan["part_size"])
if not payload:
raise RuntimeError("The upload plan has an empty part")
completed.append({
"part_number": part["part_number"],
"etag": put_part(part["url"], payload),
})
if source.read(1):
raise RuntimeError("The upload plan does not cover the file")
return completed
if __name__ == "__main__":
parts = upload_file("upload-plan.json", os.environ["REPORT_FILE"])
Path("completed-parts.json").write_text(
json.dumps(parts, indent=2)
)
The helper does not guess the create or complete request body. Those fields belong in the plan contract or in a separately tested backend client. After the browser has uploaded every part, the backend should compare the returned part numbers and ETags with its durable report record, request completion, and mark the report uploaded only after completion succeeds. The worker then performs content validation before changing the state to validated.
I treat a 429 as a scheduling signal, not as permission to retry forever. The retry budget, timeout, and concurrency limit are part of the product behavior. A browser with eight large requests in flight can make a weak connection feel broken even when every individual request is valid. Start conservatively, measure completion rate and total attempts by size bucket, and tune with a real customer-like network. I'm not sure there is a useful universal multipart threshold; the answer depends on file size, connection quality, browser limits, and the cost of replaying a failed object.
How do presigned POST, PUT, and multipart change retention and deletion?
| Workflow | Good fit | Retention and deletion obligation |
| Workflow | Good fit | Retention and deletion obligation |
| --- | --- | --- |
| Presigned PUT | Small or easily replayed reports | Delete the incomplete or rejected object; a retry replaces the whole transfer |
| Presigned POST | Existing form-policy integration | Bind policy conditions to one report key and remove rejected objects |
| Presigned multipart | Large reports or costly whole-file retries | Persist the upload identifier and parts; abort abandoned uploads and delete rejected objects |
The table is the trade-off, not a ranking. Multipart is unsuitable when the client cannot persist enough progress state, when the backend cannot own abort and cleanup, or when the object is so small that its additional transitions add more code than recovery value. Stick with a single PUT when a whole-object retry fits the failure budget. Use POST when policy-constrained form fields matter more than a simpler request.
Deletion should be a state transition with an actor and reason. A customer cancellation can delete a pending object immediately. A failed content check should prevent publication and schedule removal. A validated report can remain available until its customer contract expires, after which the application marks it expired and a cleanup worker removes the bytes and records the deletion result. If a delete is retried, the operation should be idempotent from the applicationβs perspective.
Do not make the download URL your source of truth. It is a capability with a lifetime. Authorize the customer against the report record each time a URL is issued, and use response metadata such as Content-Disposition to give the browser a safe filename; the HTTP header defines how a response is presented as inline content or an attachment. The object key should not encode a customerβs email, shipment address, or other sensitive business data.
The failure modes worth putting in the eval harness
A storage demo tests the happy path. An AI application needs a failure matrix. I would run it against a test bucket and a disposable customer record: cancel before the first byte, close the tab after several parts, retry a part after a 429, refresh after completion, submit the same completion twice, change the customer identity before download, upload a wrong content type, and let the retention deadline pass. For every case, assert both the customer-visible state and the object state.
One particularly useful assertion is that an object cannot enter the RAG pipeline merely because the browser reported success. The validator should check the expected size, content type, and any format-specific constraints before enqueueing extraction. The eval harness can then ask a sharper question than βdid upload work?β: did the right customer receive the right generated report, and did the system remove it when its retention rule said to?
Keep metrics for time to first byte, completion time, retry count, abandoned multipart count, validation rejection count, and deletion age. Log report IDs rather than raw object keys or customer data. Alert on old uploading records and on objects without a matching application record. A cleanup job should be able to explain what it removed, what it could not remove, and what it will try next.
There is a cost trade-off here. Multipart reduces retransmission work for large files, but it increases coordination, metadata, and cleanup work. A single PUT is operationally quiet, but a failed transfer can consume the same network budget again. Optimize the failure you can actually observe, not a threshold copied from another team.
A practical decision rule for the Python service
Keep the first version boring: authenticated report record, private object key, short-lived presign, direct browser transfer, explicit validation, and a cleanup worker. Add multipart only when tests show that whole-object replay is hurting completion or customer wait time. Add resumability only when the browser and backend can agree on durable part state.
Before shipping, check that the storage CORS rules permit the exact browser origin, method, and exposed response headers; verify the download filename behavior; test the real customer authentication boundary; and run deletion tests against both completed objects and abandoned multipart state. Read the storage providerβs current multipart and lifecycle documentation as part of that check because limits and policy details can vary. Your mileage may vary.
The final choice is therefore retention-first: PUT for a small report, POST for a form-policy need, multipart for a large report whose retry cost is material. None of those choices replaces an application record, a validation gate, or an explicit deletion policy. That is the architecture that lets a Python AI builder move from notebook ingestion to a customer-facing logistics workflow without losing track of who may read a report or when its bytes should disappear.
References
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.