Your n8n Workflow Only Processed the First Page: Pagination Bugs That Look Like Success

This is part of my build-in-public series on running LLM workflows in n8n in production. Everything here is from real templates I sell and support — the mistakes are mine, the fixes are tested.

The sync finished green. Twelve percent of the rows were missing

A client sync workflow of mine ran nightly, pulled new orders from an API, and wrote them into a sheet. It ran green for three weeks. Then the client asked why a dozen orders never appeared in their reports. Nothing had errored. The workflow had simply never asked for them.

The API returned 100 orders per page, and the workflow fetched exactly one page. The client's order volume had grown past 100 a night, and the extra orders sat on page two, untouched, while the run reported success. This is the pagination failure mode that survives every other check: no error, no timeout, no partial-failure flag — just a quiet, confident subset presented as the whole.

It is easy to blame the API. The real bug was in my workflow: I treated "got a response" as "got everything." A paginated API never promises that. It promises one page, and the burden of asking for the rest is entirely yours.

Three checks that catch it before your client does

First, never trust a page without checking whether more exist. Most APIs tell you, in one of three places: a next cursor or URL in the response, a total count you can compare against what you received, or a page that comes back shorter than the page size. My rule now: a fetch loop ends only on an explicit signal — an empty next cursor, or a page smaller than the limit — never on "we got one good response." If the API gives you a total, assert received == total at the end of the loop. If it does not, log the page count on every run so a jump from one page to three is visible instead of silent.

Second, handle the two pagination styles differently, because they fail differently. Offset pagination (?page=2, ?offset=100) breaks when rows are inserted or deleted mid-sync: the same row can appear twice, or slip between pages and vanish. Cursor pagination is stable against that, but cursors expire, and a stale cursor fails in ways that look like "no more data." Whichever style the API uses, my loop stores the cursor or offset in the workflow state after each page, not just at the end — so a crash on page four resumes at page four instead of re-fetching from page one and double-processing everything before it.

Third, dedupe on the business key, not the row position. When a sync overlaps — a retry re-fetches page one, or an offset shift duplicates a row — the only safe dedupe key is the record's own id, checked against what is already stored before any write. Position-based dedupe ("skip the first N rows") is exactly what breaks when the data moves underneath you. This is the same identity lesson as deduping webhook replays: the id that the source system assigns is the truth; anything you invent from ordering is a guess.

What a healthy sync reports

A sync that only reports "done" is hiding the one number that matters. Mine now reports four: pages fetched, rows received, rows written, and rows skipped as duplicates. When pages fetched is one and rows received equals the page size exactly, that is a smell, not a success — it usually means there is a page two nobody asked for. When rows skipped as duplicates jumps, something retried or shifted, and I want to know before the client does.

The rule I now build by: a paginated fetch is a loop with an end condition, a resume point, and a count reconciliation — not a single HTTP node with a green checkmark. One HTTP node that returns 100 rows is not a sync. It is a sample, and samples presented as totals are how clients lose orders for three weeks without anyone noticing.

I'm building AI Automation Lab — production-ready n8n workflow templates. The free AI Hook Generator shows how I structure real workflows; my open-source templates live at github.com/zhp910318/n8n-ai-automation-lab.