Scraping Bayut Dubai into a Fixed 91-Column Schema (Zero Proxies, 90-Day DLD Medians & MCP)
Pulling 3,771 active Dubai apartment listings every morning exposes how messy raw UAE portal feeds actually are: the same unit routinely appears 3 to 5 times under different brokerages at asking prices 15% apart, usually
Pulling 3,771 active Dubai apartment listings every morning exposes how messy raw UAE portal feeds actually are: the same unit routinely appears 3 to 5 times under different brokerages at asking prices 15% apart, usually paired with "8%+ gross ROI" headlines that ignore 22 AED/sqft service charges and 4% DLD transfer friction.
For the Bayut pipeline, running headless browsers and burning residential proxies on individual HTML detail pages turned out to be wasted compute. Bayut's search index returns the full listing recordβdirect agent mobile numbers, TruCheck verification timestamps, off-plan payment milestones, and GPS coordinatesβin a single sub-second HTTPS pass.
I published that extractor on Apify Store as kotaro001/bayut-com-scraper: zero residential proxies, a locked 91-column flat schema, uncapped free-tier runs, and 90-day Dubai Land Department (DLD) registered sold medians joined directly onto each row.
Flat Schema & Measured Fill Rates (50-Row Dubai Marina Benchmark)
Raw portal JSON tends to drift across buy, rent, commercial-buy, and commercial-rent verticals. Every run here outputs the exact same 91 flat columns so downstream SQL tables, pandas pipelines, and LLM tool calls never break on nested objects.
| Field Group | Key Columns | Measured Fill Rate & Pipeline Behavior |
|---|---|---|
| Core Pricing & Specs |
priceNumeric, pricePerSqftAed, bedrooms, bathrooms, areaSqft, furnishing
|
50/50 on active residential runs; commercial desks under 180 sqft are isolated via isBusinessCentre: true
|
| 90-Day DLD Sold Benchmark |
dldMedianSoldPriceAed, dldMedianSoldPricePerSqftAed, vsDldSoldMedianPercent, dldSampleTransactionsCount, dldBenchmarkBasis
|
Joined from Dubai Pulse open ready-sale records (90d window, trimmed below 150 or above 8,000 AED/sqft); requires dldSampleTransactionsCount >= 3 at building level before falling back to community
|
| Velocity & Price Cuts |
daysOnMarket, priceChanged, previousPrice, priceDiffPercent
|
daysOnMarket computed on first run directly from posted_at; tracks portal-recorded asking cuts |
| Visa & Portal Trust |
goldenVisaEligible, isVerified, verifiedAt, isTruBroker
|
50/50 on verifiedAt; goldenVisaEligible flags units at 2,000,000 AED+
|
| Off-Plan Schedules |
developerName, handoverDate, paymentPlanDownPercent, paymentPlanPostHandoverPercent
|
4/50 to 7/50 in mixed Marina searches (populated strictly when primary off-plan listings publish payment milestones) |
| Broker Contacts |
agent1Name, agent1Phone, agent1MobileDigits, agent1Whatsapp, agencyName
|
50/50 on agent1Phone; agent1Whatsapp is scrubbed to null when an agency repeats one switchboard number across 80%+ of listings |
| DLD / Madhmoun Permit | permitNumber |
0/50 β Bayut's search index does not expose permit numbers, so the column stays null by design |
Python Quickstart: Screen Ready Units Below 90-Day DLD Medians
pip install apify-client pandas
import os
import pandas as pd
from apify_client import ApifyClient
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("kotaro001/bayut-com-scraper").call(
run_input={
"listingType": "buy",
"communities": ["Dubai Marina", "Business Bay", "JVC"],
"completionStatus": "ready",
"belowCommunityMedianPercent": 10,
"goldenVisaOnly": True,
"onlyVerified": True,
"maxItems": 100,
}
)
df = pd.DataFrame(client.dataset(run["defaultDatasetId"]).iterate_items())
cols = [
"community",
"towerOrBuilding",
"bedrooms",
"priceNumeric",
"pricePerSqftAed",
"dldMedianSoldPricePerSqftAed",
"vsDldSoldMedianPercent",
"dldBenchmarkBasis",
"daysOnMarket",
"agent1Phone",
]
# Negative vsDldSoldMedianPercent = asking PSF sits below 90-day DLD sold median
under_dld = df[df["vsDldSoldMedianPercent"] < -5.0].sort_values("vsDldSoldMedianPercent")
print(under_dld[cols].head(10).to_string(index=False))
In vsDldSoldMedianPercent, ((pricePerSqftAed / dldMedianSoldPricePerSqftAed) - 1) * 100 measures the exact spread against registered DLD sales rather than portal-wide averagesβfor instance, -14.5 means the unit asks 14.5% per sqft below the 90-day DLD median for its building or community cohort.
Wiring Into Claude Code or Cursor via Apify MCP
Because all 91 columns are flat scalars and clean arrays (up to 12 photo URLs in images, total count in imagesCount), an LLM agent can filter 100 rows without context bloat. Drop the Streamable HTTP endpoint into .mcp.json:
{
"mcpServers": {
"bayut-dubai": {
"type": "http",
"url": "https://mcp.apify.com/?tools=kotaro001/bayut-com-scraper"
}
}
}
Authenticate via /mcp OAuth in Claude Code (or Authorization: Bearer <APIFY_TOKEN>), then query concrete parameters directly:
Run kotaro001/bayut-com-scraper for ready 2-bed apartments in Dubai Marina and Downtown Dubai (maxItems 60, onlyVerified true). Return units where vsDldSoldMedianPercent is below -10 and dldBenchmarkBasis is "building", sorted by daysOnMarket descending.
Cost Math & Live Endpoints
Billing runs strictly on Pay-Per-Event with compute and retries baked in ($0 residential proxy fees, $0 detail-page fees, and free-tier row caps unlocked):
-
Free & Starter ($19/mo plan):
$0.005per actor start +$0.75 / 1,000 listings($0.00075per row). Apify's default$5.00monthly free credit covers roughly 6,660 listings/month at zero out-of-pocket cost. -
Scale ($199/mo+) & Business:
$0.004start +$0.55 / 1,000 listings.
Links to test or inspect the schema directly:
-
Apify Actor:
kotaro001/bayut-com-scraperand 10 pre-configured Dubai & Abu Dhabi task presets - Live UAE Deals Screener (with RERA/Mollak fees deducted): propdealradar.com/ae/deals
-
Machine-Readable
/llms.txt& Full Scraper Suite: propdealradar.com/developers
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.