How I scrape hackathon platforms every day with GitHub Actions and Supabase (and the bug that froze my site for six weeks)

Hackathons are published all over the place. Devpost, Devfolio, Unstop, MLH, CTFtime, DoraHacks, Google Developer Groups, plus a lot of local events that only exist as an Instagram post. I'm a student in Pakistan and I kept finding out about events after the deadline, so I built DevSafar, a free site that collects them into one list.

This post is about how it works and, more usefully, about what went wrong.

The pipeline

• Scrapers run daily on GitHub Actions. Some platforms have JSON endpoints; the JavaScript-heavy ones need Puppeteer. Each scraper writes rows into Supabase.
• Two tables: one for community-submitted events, one for scraped ones. Both are public-readable for approved rows.
• Cleanup scripts deduplicate (the same event often appears on two platforms), normalise locations and categories, and mark ended events.
• The app is Expo and React Native for web and Android, from one codebase, hosted on Vercel.

The bug: a frozen site and 15 indexed pages

I assumed the site updated itself. It didn't.

The scraper wrote to the database every night, and the app read the database, so everything looked fine when I used it. But the pages that search engines see are generated when Vercel builds, and nothing triggered a build after the scrape. A secret for the deploy hook was missing and the workflow step skipped silently when it wasn't set.

For six weeks, the live sitemap stayed frozen at 3,588 URLs while the database had grown to 5,191 listings. Every new hackathon was served as a blank page with an empty title.

Worse, even the pages that were built were nearly identical. The details loaded with JavaScript after the page opened, so the HTML that Googlebot and Bingbot read had the same 219 characters of body text on all ~5,000 hackathon pages. Google's verdict in Search Console: 15 pages indexed, over 3,600 "crawled, currently not indexed".

The fix

1. Rebuild after every scrape. The workflow now calls a Vercel deploy hook, and fails loudly (a red run and an email) if the secret is missing, instead of skipping.
2. Real content in the HTML. Expo Router has build-time data loaders. Each page's loader runs once during expo export and the data is written into the page, so a crawler sees the title, dates, prize and links without running any JavaScript.
3. Don't pre-render everything. Pre-rendering all 5,191 pages ran the build out of memory, and events that ended two months ago don't deserve a place in the index anyway. They're now noindex and left out of the sitemap.
4. Pages that match what people search for, generated from the data: live now, online, free, by country, by city, by month. A page only exists if it has enough listings, and none is generated if it would be nearly identical to its parent.
5. A smaller JavaScript bundle. three.js (used only for a login-screen background) was in the bundle every page downloaded. Moving it behind a dynamic import() cut the shared bundle from 3.1 MB to 1.6 MB.

What I'd tell myself earlier

• A scheduled job that writes data and a static site that reads it need an explicit link between them. Check the deployed output, not the app.
• View source on a few pages. If the content isn't in the HTML, search engines mostly won't see it.
• Make "this step is not configured" a failure, not a skip.

The site is free and I'd like feedback: www.devsafar.app. Which platforms are missing?