Why n8n Drops Marketplace Records When a Run Exceeds the 300 Second Cap

The architecture of a stateful Marketplace ingestion pipeline

Scraping public marketplaces is only the first step in a data ingestion pipeline. If you configure a scheduled crawler to search for products, real estate, or vehicles, the target source will continue to return identical records on subsequent executions. The facebook-marketplace-scraper extracts live records based on your parameters, but it operates statelessly across runs. If you schedule it to run every hour, you will capture the same active listings repeatedly.

Without an intermediate state layer, every downstream action is duplicated. Your database will waste writes, your messaging system will broadcast duplicate alerts, and any connected language model or categorization API will consume credits processing the exact same records. An industrial-grade ingestion pipeline must separate data extraction from state tracking.

The optimal architecture consists of three distinct layers:

1. An execution layer that initiates the crawl asynchronously to avoid system timeouts.
2. An ingestion and buffering layer that handles the webhook payload or trigger from the run.
3. A stateful deduplication layer that cross-references incoming listing IDs against a persistent store before writing to a data warehouse or CRM.

By isolating these layers, you guarantee that network failures, schema changes, or database downtime will not lose data. You also minimize downstream operations, ensuring that you only write, process, and alert on genuinely new listings.

How do I stop duplicate listings from processing on subsequent scraper runs?

You can prevent duplicate listings by tracking already-processed listing IDs in a database and filtering them out before they reach your downstream applications. When new listings are ingested, the system compares the incoming IDs against the database, processes only the new ones, and appends them to the store. This stateful deduplication ensures that re-running the pipeline does not waste resources on redundant operations.

The scraper itself has a built-in input parameter called deduplicateAcrossInputs, which is enabled by default. However, this parameter only removes duplicate listing IDs within the same run. If the same item appears in the results of two different runs on consecutive days, the scraper has no memory of the prior execution and will output it again.

To establish persistent, cross-run state tracking, you must capture the unique id field from each listing object in the dataset. This ID is a stable, unique identifier assigned by Facebook. When your downstream worker processes an incoming payload, it must run a lookup query against your state database. If the ID exists, the record is discarded. If it does not exist, the record is processed and its ID is written to the database with a timestamp.

import sqlite3
import requests

def init_db(db_path="listings_state.db"):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()
    cursor.execute("""
        CREATE TABLE IF NOT EXISTS processed_listings (
            id TEXT PRIMARY KEY,
            processed_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)
    conn.commit()
    return conn

def get_new_listings(dataset_url, conn):
    response = requests.get(dataset_url)
    response.raise_for_status()
    items = response.json()

    cursor = conn.cursor()
    new_items = []

    for item in items:
        listing_id = item.get("id")
        if not listing_id:
            continue

        # Check state store
        cursor.execute("SELECT 1 FROM processed_listings WHERE id = ?", (listing_id,))
        if cursor.fetchone() is None:
            new_items.append(item)

    return new_items

def mark_as_processed(listing_ids, conn):
    cursor = conn.cursor()
    cursor.executemany(
        "INSERT INTO processed_listings (id) VALUES (?)",
        [(lid,) for lid in listing_ids]
    )
    conn.commit()

Why does the synchronous Apify run endpoint fail with a 408 error?

The synchronous run endpoint fails with a 408 Request Timeout because the Apify platform enforces a hard cap of 300 seconds on synchronous execution. If the scraper runs longer than five minutes, the connection drops and returns an HTTP 408. To avoid this, you must initiate the run asynchronously, poll the run status, or configure a webhook to receive a POST notification when the job finishes.

When setting up a pipeline via tools like n8n or Make, developers often attempt to make a blocking HTTP POST request to the run endpoint, waiting for the results to return in the HTTP response. If your run configuration requires visiting detail pages using scrapeMode: "deep", or if you set maxItems to a high number, the execution will easily exceed five minutes. When the 300-second mark is reached, the platform abruptly terminates the connection, returning HTTP status 408.

To prevent these timeouts, you must invoke the Actor using the asynchronous run endpoint. This returns a running state object immediately with HTTP 201. Your workflow must then rely on a webhook pointing to your endpoint, or use an integration platform's native trigger node that listens for the execution success event.

curl --request POST \
  --url "https://api.apify.com/v2/acts/crawlerbros~facebook-marketplace-scraper/runs?token=YOUR_APIFY_API_TOKEN" \
  --header "Content-Type: application/json" \
  --data '{
    "searchQuery": "furniture",
    "location": "nyc",
    "maxItems": 20,
    "scrapeMode": "fast"
  }'

Building the state store with a lightweight database

A production-grade pipeline must scale. When designing state storage, you have to account for volume. While key-value caches like Redis are excellent for quick lookups, they require persistent volume configurations to avoid data loss on container restarts. A relational engine such as PostgreSQL or SQLite provides ACID compliance and simple schema definitions.

If your pipeline runs within a serverless function, SQLite is highly efficient provided its database file resides on persistent storage. For containerized environments like Docker, an external PostgreSQL instance is more resilient.

Your state store should track more than just the listing ID. Tracking the retrieval source, the listing date, and insertion timestamps lets you run cleanups. Facebook listing IDs eventually expire or are taken down when items sell. To keep your database from growing indefinitely, implement a sliding window retention policy. If a listing is not returned for 30 days, it is likely inactive, and its ID can be safely pruned from the state database.

Apify storage limits must also be considered. While your custom database can scale, Apify default datasets are optimized for high concurrency. However, trying to write directly to a shared request queue across multiple simultaneous actor runs will fail. A request queue can only be processed by one Actor or task run at a time. If you require shared state during extraction, you must handle it downstream in your database, not inside the crawler storage.

Writing the n8n orchestration workflow to handle webhook payloads

Using n8n to orchestrate your pipeline removes the need for polling scripts. The native n8n Apify trigger node listens directly for run completions, which eliminates the need to poll the run status continuously.

If you are hosting n8n yourself, you can use the API key credentials. If you are on n8n Cloud, you can use OAuth2. When n8n receives the execution success event, it receives metadata including the defaultDatasetId. The workflow should then call the Apify API to fetch items from that dataset in batches.

This batching mechanism protects your database. The default dataset item pushes can handle up to 400 requests per second, but your database might bottleneck on massive writes. By using an n8n batch node, you can divide a large listing payload into smaller chunks.

Below is an extraction of the JSON output object shape produced by the scraper. Your downstream n8n nodes must parse these fields to extract the listing data and trace it back to the original search query.

{
  "id": "772594759027016",
  "title": "ThinkPad T14 Gen 2 Laptop",
  "price": {
    "amount": 350,
    "currency": "USD"
  },
  "city": "New York",
  "state": "NY",
  "locationText": "New York, NY",
  "primaryPhotoUrl": "https://scontent.fnyc1-1.fna.fbcdn.net/v/example",
  "sourceUrl": "https://www.facebook.com/marketplace/nyc/search/?query=laptop",
  "sourceType": "search",
  "inputIndex": 0,
  "searchQuery": "laptop",
  "locationQueried": "New York, NY",
  "extractionSource": "search_page",
  "qualityStatus": "complete",
  "facebookUrl": "https://www.facebook.com/marketplace/item/772594759027016/"
}

How do I pass custom search criteria to the scraper via the API?

You pass custom search criteria by making an authenticated HTTP POST request to the run endpoint with the structured configuration in the JSON payload body. You must specify the parameters explicitly in the JSON request because the Apify Console prefill values are ignored by direct API calls. Key parameters include startUrls or a combination of searchQuery and location.

Many developers make the mistake of configuring input fields inside the Apify Console UI and expecting those settings to apply automatically when they call the API without a payload. The schema prefill parameters are designed strictly for the Console UI. When invoking the Actor via API, only the standard default values in the actor schema are applied. Any custom queries, locations, filters, or proxy groupings must be passed explicitly in your HTTP request body.

If you omit the startUrls array, you must supply both searchQuery and location. The location value must match a supported city name or slug like "nyc" or "Austin, TX". You should also pass your proxy configuration to ensure the actor uses residential IPs, as datacenter proxies are quickly blocked by Facebook's traffic controls.

{
  "searchQuery": "used bicycle",
  "location": "seattle",
  "radiusKm": 50,
  "sortBy": "newest",
  "condition": "used_good",
  "minPrice": 100,
  "maxPrice": 1000,
  "maxItems": 50,
  "scrapeMode": "fast",
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"],
    "apifyProxyCountry": "US"
  }
}

Writing the Python pipeline to load and deduplicate dataset items

When building a high-volume ETL pipeline, using raw SQLite connections can lead to race conditions if multiple workers run concurrently. A Python ETL worker should implement a transaction-safe database write using an ORM or a context manager to lock the table during lookups and inserts.

The following script implements a complete ETL worker. It initializes the client, launches the run asynchronously, polls for completion, pages through the resulting dataset to stay within storage rate limits (60 requests/sec per storage object), filters out duplicates using SQLite, and saves only new records.

import time
import sqlite3
from apify_client import ApifyClient

def process_marketplace_pipeline(api_token, actor_id, run_input, db_path="state.db"):
    client = ApifyClient(api_token)

    # 1. Start the Actor asynchronously
    print("Starting crawler run...")
    run = client.actor(actor_id).start(run_input=run_input)
    run_id = run["id"]

    # 2. Poll the run status safely
    while True:
        status_detail = client.run(run_id).get()
        status = status_detail.get("status")
        print(f"Current run status: {status}")

        if status == "SUCCEEDED":
            break
        elif status in ["FAILED", "ABORTED", "TIMED-OUT"]:
            raise RuntimeError(f"Actor run failed with status: {status}")

        time.sleep(15)

    dataset_id = status_detail["defaultDatasetId"]
    dataset_client = client.dataset(dataset_id)

    # 3. Connect to local state database
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()
    cursor.execute("""
        CREATE TABLE IF NOT EXISTS tracked_items (
            id TEXT PRIMARY KEY,
            added_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        )
    """)
    conn.commit()

    # 4. Stream dataset items page by page to avoid memory exhaustion
    offset = 0
    limit = 100
    new_records_processed = 0

    while True:
        page = dataset_client.list_items(offset=offset, limit=limit)
        items = page.items
        if not items:
            break

        # Filter items inside a single database transaction
        cursor.execute("BEGIN TRANSACTION")
        try:
            for item in items:
                listing_id = item.get("id")
                if not listing_id:
                    continue

                cursor.execute("SELECT 1 FROM tracked_items WHERE id = ?", (listing_id,))
                if cursor.fetchone() is None:
                    # Record is unique: write to database and forward to pipeline
                    cursor.execute("INSERT INTO tracked_items (id) VALUES (?)", (listing_id,))
                    print(f"New listing found: {item.get('title')} ({listing_id})")
                    new_records_processed += 1
            conn.commit()
        except Exception as e:
            conn.rollback()
            raise e

        offset += limit

    conn.close()
    print(f"Pipeline finished. Processed {new_records_processed} new listings.")

Real limitations and failure modes of Facebook Marketplace scraping

No scraper can guarantee search results because public web data extraction is subject to target platform controls. Facebook Marketplace is particularly dynamic. It does not provide a public data API, meaning the facebook-marketplace-scraper must read data exposed by public marketplace web pages. If Facebook alters its DOM structure or class names, selectors can break, resulting in empty outputs until the scraper is updated.

The scraper distinguishes between valid empty searches (where no matches exist for your filters) and access failures like login walls, consent pages, or CAPTCHA checkpoints. If the actor hits a challenge page, it reports the block instead of returning a false empty dataset. Your pipeline should always inspect the run status and the output structure. If a run succeeded but yielded zero listings, verify that the scraper was not served a login checkpoint.

Another critical limitation is the lifespan of media URLs. The primaryPhotoUrl field contains CDN links pointing directly to Facebook's media servers. These links are subject to strict expiring signature parameters. After a few days, these image URLs will return HTTP 403 Forbidden. Your pipeline should never store these URLs with the expectation that they will remain accessible indefinitely. If your application needs long-term image storage, you must download the images during execution and host them on your own storage buckets.

Proxy management is also a common point of failure. Datacenter IP addresses are heavily throttled or outright blocked by Facebook. The actor requires residential proxies to maintain consistency. When Apify Proxy is selected, the scraper automatically forces the use of US residential proxies.

Because residential proxy sessions expire after approximately 30 minutes, running the scraper in deep mode with thousands of inputs can result in aborted sessions if the total execution time stretches too long. Ensure your runs are tightly focused by scoping locations and listing counts.

True cost calculation of payload events on the Apify platform

Understanding costs is essential for operating pipelines at scale. The scraper operates under a PAY_PER_EVENT billing model. Every operation that incurs a charge is a distinct event, meaning you do not pay a platform usage or compute subscription fee. The total run cost scales strictly with the volume of events your crawl processes.

The billing schema is composed of exactly two charged events:

1. "Actor Start" (apify-actor-start): This event is charged exactly once per execution. The cost is $0.05 per GB of memory allocated to the run. For example, if you allocate memory to the scraper run, the event is charged based on this exact flat per-event rate.

2. "result" (apify-default-dataset-item): This event is charged for every single item generated and written to the default dataset. The standard cost is $0.002 per event. The unit price decreases if your account qualifies for Apify volume tiers. The exact tier pricing is:

   • FREE: $0.002 per event
   • BRONZE: $0.00167 per event
   • SILVER: $0.00133 per event
   • GOLD: $0.001 per event
   • PLATINUM: $0.001 per event
   • DIAMOND: $0.001 per event

Your primary cost driver is the number of results returned by the run, modified by your account's volume tier. Running the scraper with high item limits will naturally scale your cost.

If you switch the scrapeMode to "deep", the run will take longer because it has to visit each listing page individually. However, because billing is strictly event-based rather than time-based, the extra container execution time does not increase your cost. You will still only pay for the "Actor Start" event and the number of "result" items written to the dataset.

To prevent runaway billing charges, you should configure the maxTotalChargeUsd parameter on your API calls. This parameter is exposed inside the Actor runtime as ACTOR_MAX_TOTAL_CHARGE_USD. If a run reaches this spending limit, the system gracefully shuts down execution. This cap ensures that a misconfigured search query does not emit hundreds of thousands of items and deplete your balance unexpectedly.

Operational validation and verifying pipeline integrity

Before deploying your pipeline to a production cron schedule, you must perform validation. The daily prefill parameters supplied in the Actor's input schema represent the production-safe baseline configuration for testing. This baseline configuration conducts a small search for laptops in New York, limiting results to 5 items in fast mode with deduplication active.

Verify that your pipeline can execute this test and correctly handle the resulting dataset. Your ingestion logic must programmatically confirm the presence of core fields:

• id: Must be a non-empty string.
• sourceUrl: Must match the input pattern.
• qualityStatus: Must report complete or partial.

To test your state store, trigger the test configuration twice in succession. On the first run, your SQLite or PostgreSQL database should record 5 inserts. On the second run, the database should log 0 inserts and discard all 5 items as duplicates. If your database records duplicates on the second run, your primary key constraint or lookup logic is failing.

Checked against the Actor's input schema and Apify docs on 2026-09-16. Ensure that any scheduled tasks you configure on the platform are fully tested before activation, as new schedules are created disabled by default to prevent unintended runtime execution costs. Once verified, enable the schedule to run at your desired interval to feed your cleaned, deduplicated, and state-managed Marketplace pipeline.

The Actor's README is the source of truth for its inputs, outputs and limits. Need a hand wiring this into your stack? Email [email protected]