neweb.ai: Architecture of an Autonomous Website Manager

Overview

neweb.ai is a full-stack SaaS platform that automates WordPress website creation and blog content generation using multiple AI models. Users configure their business details, and the platform provisions a live WordPress site, generates SEO-optimized blog content on autopilot, tracks keyword rankings, and audits content performance — all through a real-time dashboard.

Stack: Node.js/Express, MongoDB, PostgreSQL, Redis, BullMQ, React 19, Redux Toolkit, TanStack Query v5, Socket.IO, Stripe, Clerk Auth, Docker, Prometheus + Grafana

Live: dashboard.neweb.ai | landing.neweb.ai

Architecture

┌─────────────────┐     ┌──────────────────┐     ┌───────────────────┐
│  User Dashboard  │     │  Admin Dashboard  │     │   Landing Site    │
│  (React + Vite)  │     │  (React + Vite)   │     │   (Next.js)       │
└────────┬────────┘     └────────┬──────────┘     └───────────────────┘
         │                       │
         └───────────┬───────────┘
                     │ REST + Socket.IO
              ┌──────┴──────┐
              │   Backend   │──── Clerk (Auth)
              │  (Express)  │──── Stripe (Payments)
              └──────┬──────┘──── DataForSEO (Rankings)
                     │         ── Azure OpenAI (Content Gen)
          ┌──────────┼──────────┐── Claude Haiku (Audits)
          │          │          │── Runware (Image Gen)
     ┌────┴───┐ ┌───┴───┐ ┌───┴────┐
     │MongoDB │ │ Redis  │ │ PgSQL  │
     └────────┘ └───┬───┘ └────────┘
                    │
              ┌─────┴─────┐
              │  BullMQ   │
              │  Workers   │
              └───────────┘

• Monorepo with 5 packages: backend, user dashboard, admin dashboard, landing site, WordPress themes
• Dual databases: MongoDB (primary data), PostgreSQL (secondary/analytics)
• Redis powers BullMQ job queues, Socket.IO adapter, and SEO data caching

Core Features

1. WordPress Provisioning Pipeline (12-Step Orchestration)

The standout engineering feat — a 12-step sequential pipeline that provisions a full WordPress site from scratch:

1. Create Subdomain (mybiz-f4d2-a7x9.newebai.com)
2. Install WordPress (admin creds held in-memory)
3. Install Theme (dynamically resolved by business type)
4. Install Plugins
5. Insert Business Details (via WP REST API)
6. Shortcode Modifications
7. Health Check (graceful if DNS hasn't propagated)
8. Add Page Templates
9. Import Global Styles
10. Load Environment Config
11. Core Setup
12. Editor Configuration

Key design decisions:

• Full-sequence retry, not per-step — if step 8 fails, the entire pipeline retries from step 1 after cleaning up the failed subdomain. This guarantees atomic consistency: you never get a half-built site.
• Max 3 attempts with full cleanup between retries
• Real-time progress via Socket.IO (wp:build:progress events streamed per step)
• Credentials emailed as PDF after successful provisioning — avoids storing plaintext admin passwords in logs

2. AI Blog Generation & Autopilot

Multi-model LLM strategy:

• Azure OpenAI (gpt-4o-mini-2) — bulk content generation (fast, cost-effective)
• Claude Haiku 4.5 — content auditing and ranking-drop diagnosis (better at analysis)
• Runware SDK — AI image generation

The generation pipeline enforces structured JSON output with strict constraints:

• SEO-optimized titles (<60 chars with focus keyword)
• Meta descriptions (exactly 150-160 chars)
• Target keyword density of 1-2%
• Word count bands: short (500-600), medium (900-1100), long (1400-1600)
• Suggested internal links and image placements

Autopilot mode: Users configure a schedule (daily/weekly/monthly) with a pre-populated topic pool. The system generates and optionally auto-publishes content, pausing automatically after 3+ consecutive failures.

3. SEO Intelligence

• DataForSEO integration for keyword rank tracking (SERP depth 100)
• Backlink analysis — total backlinks, referring domains, domain authority
• Lighthouse audits — performance, accessibility, SEO, best practices scores
• On-page SEO scoring — real-time analysis of title, meta, keyword density, headings, word count
• Weekly content audits (Wednesday 4AM UTC) — Claude Haiku analyzes blogs with ranking drops, diagnoses issues, suggests fixes with priority levels

All SEO APIs use Promise.allSettled() for graceful degradation — missing ranking data never blocks site creation or publishing.

4. Subscription & Marketplace

• Stripe integration with webhook signature validation
• Accumulated credits model for yearly plans — monthly credits added to a pool rather than hard per-action limits
• Add-on system — purchased add-ons expand effective limits via getEffectiveLimits() utility
• Stripe Connect marketplace — agencies enable Connect to receive a percentage of client billing
• Usage tracking with canPerformAction() checks before every billable operation

5. Agency & RBAC System

Five user roles: user, agency, admin, moderator, manager

The agency model is elegant:

• Agency users have agencyId: null (they own the agency)
• Managed users have agencyId: <agencyClerkId>
• Resources track both clerkId (direct owner) and agencyClerkId
• Ownership middleware checks agency relationship before falling back to direct ownership
• Admins/moderators/managers bypass all ownership checks

Infrastructure & Operations

Job Queue Architecture (BullMQ)

Jobs use explicit jobId for idempotency: rank:${siteId}:${timestamp} prevents duplicates if cron fires twice.

Monitoring Stack

• Prometheus — HTTP request duration histograms, total request counters
• Grafana — dashboards accessible at /grafana/
• Loki — structured log aggregation via Winston
• Custom error monitor — classifies severity, throttles identical errors for 5 minutes, only alerts on non-operational errors, sanitizes sensitive fields

Real-Time Architecture (Socket.IO)

• JWT verification via @clerk/backend verifyToken() on handshake
• Users auto-joined to user:${clerkId} rooms (enables multi-tab sync)
• Redis adapter for horizontal scaling across server instances
• Events: wp:build:progress, wp:build:done, wp:build:failed, notification:*, chat:message

What I learned building it

Key Professional Takeaways

1. Full-Sequence Retry > Per-Step Retry for Multi-Step Pipelines

When you have a tightly coupled pipeline (like WordPress provisioning), retrying individual steps creates inconsistent state. Retrying the full sequence with cleanup between attempts guarantees atomicity. The tradeoff is speed — but correctness matters more when you're provisioning infrastructure.

When to apply: Any multi-step process where partial completion leaves the system in an invalid or hard-to-recover state (database migrations, infrastructure provisioning, payment flows).

2. Multi-Model LLM Strategy Optimizes Cost and Quality

Using different AI models for different tasks isn't just about vendor diversification — it's about matching model strengths to use cases. Fast/cheap models (gpt-4o-mini) handle bulk generation; analytical models (Claude) handle auditing where reasoning quality matters more than speed.

When to apply: Any AI-heavy product. Profile your LLM calls by volume, latency requirements, and reasoning complexity, then assign models accordingly.

3. Accumulated Credits > Hard Limits for Subscription Flexibility

Instead of "5 blogs per month" as a hard cap, pooling credits allows users to front-load or spread usage. This reduces churn from users who hit limits early in their billing cycle while still enforcing overall quotas.

When to apply: Any usage-based SaaS where customers have variable usage patterns within billing periods.

4. Graceful Degradation with Promise.allSettled()

Third-party APIs will fail. The pattern of wrapping non-critical API calls in Promise.allSettled() and continuing with partial data keeps your core flow running. DataForSEO being down shouldn't prevent blog publishing.

When to apply: Anywhere you aggregate data from multiple external sources. Identify which data is critical vs. enrichment, and only block on the critical path.

5. Error Throttling Prevents Alert Fatigue

Deduplicating identical errors within a 5-minute window means your team gets one alert for "MongoDB connection timeout" instead of 500. Combined with severity classification (only alert on non-operational errors), this keeps monitoring actionable.

When to apply: Any production system with error alerting. Without throttling, your team starts ignoring alerts — which is worse than no monitoring at all.

6. Job Idempotency via Explicit jobId

BullMQ (and most queue systems) accept a jobId that prevents duplicate enqueuing. Using deterministic IDs like rank:${siteId}:${timestamp} ensures cron-triggered jobs don't double-execute if the scheduler fires twice.

When to apply: Any queue-based system with scheduled or event-triggered jobs. Always ask: "What happens if this job gets enqueued twice?"

7. Agency/Ownership Middleware Pattern

The three-tier ownership check (admin bypass → agency relationship → direct ownership) is a clean, composable pattern for multi-tenant SaaS. It's implemented as middleware, keeping route handlers free of authorization logic.

When to apply: Any platform with B2B features where one organization manages resources on behalf of others (agencies, resellers, white-label products).

8. In-Memory Credential Handling for Provisioning

WordPress admin credentials are generated, used during setup, saved to the database after successful provisioning, then emailed as a PDF — never logged or stored in plaintext during the process. This minimizes the window where credentials exist in an insecure state.

When to apply: Any automated provisioning that generates credentials. Think about the lifecycle: generation → usage → persistence → delivery, and minimize exposure at each stage.

9. Topic Pools Decouple Generation from Scheduling

Rather than generating topics on-the-fly during scheduled runs (which can fail and waste a generation credit), pre-populating a topic pool lets users curate content direction. The scheduler just pulls from the pool. Empty pool? Generate one topic, but the user controls the editorial pipeline.

When to apply: Any automated content or task system. Separating "what to do" from "when to do it" gives users control without sacrificing automation.

10. Metadata Maps for Schema Flexibility

Using metadata: Map on MongoDB documents allows storing arbitrary data without schema migrations. WP setup logs, publish errors, custom fields — all go in metadata. The core schema stays clean while edge cases are handled flexibly.

When to apply: Any document-based system where different entities of the same type need different auxiliary data. Prefer typed fields for common data, metadata maps for the long tail.

Source: https://github.com/uditrajmr3/neweb-user-dashboard

Originally published on uditraj.site — from the Workshop.