Dev.to WebDev πŸ›  Dev πŸ‘ 0 πŸ“– 11 min read

Hybrid on-device + cloud AI in a Chrome extension with Firebase AI Logic

AI disclosure: This article was written with AI assistance (Claude). The project it describes was also built with AI assistance. All code below is copied from the public repository, and the behaviour described matches it

AI disclosure: This article was written with AI assistance (Claude). The project it describes was also built with AI assistance. All code below is copied from the public repository, and the behaviour described matches its unit tests.

Chrome now ships a small language model, Gemini Nano, that a page or extension can call through the Prompt API. That is great for privacy and cost: the text never leaves the machine and you pay nothing per call. It is less great for anything long. Nano's context window is small, it isn't available on every machine, and the first call may have to download the model.

Firebase AI Logic has a "hybrid" mode that wraps both worlds in one getGenerativeModel() call: you say whether you prefer on-device or cloud, and the SDK falls back to the other one. That covers the basic case. In a real app you still have to decide which mode to ask for, what to do with documents that don't fit on the device, and how to trust what a small model tells you.

This post walks through how I handled those three problems in Plain Terms, a Chrome extension and web app that rewrites terms of service and cookie banners into plain language and flags risky clauses:

1. Make routing a pure function

The most useful decision I made was to keep the routing policy out of the Firebase code entirely. src/core/router.js takes facts it can observe and returns a plan, plus a one-sentence reason that the UI shows the user. It imports nothing, so it runs in Node tests without a browser.

The inputs are:

  • document length in characters
  • the user's privacy setting (private, balanced, cloud)
  • navigator.onLine
  • the result of LanguageModel.availability()
  • whether Firebase is configured at all

The mode strings are deliberately equal to Firebase AI Logic's InferenceMode values, so the plan can be passed straight to the SDK:

export const MODE = {
  PREFER_ON_DEVICE: 'prefer_on_device',
  ONLY_ON_DEVICE: 'only_on_device',
  PREFER_IN_CLOUD: 'prefer_in_cloud',
  ONLY_IN_CLOUD: 'only_in_cloud',
};

export const DEFAULT_LIMITS = {
  // Gemini Nano in Chrome has a ~9k token window shared by prompt + output.
  // ~12k characters of document leaves room for instructions and JSON output.
  onDeviceMaxChars: 12000,
  chunkChars: 9000,
  maxChunks: 8,
  // Above this, even the cloud call is truncated (keeps free-tier usage small).
  cloudMaxChars: 120000,
};

The policy itself reads top to bottom like a decision table. Here is the core of it, trimmed slightly:

const long = p.chars > limits.onDeviceMaxChars;
const cloudAllowed = privacy !== PRIVACY.PRIVATE && p.online !== false;

if (!cloudAllowed) {
  const why = privacy === PRIVACY.PRIVATE ? 'On-device only is switched on' : 'You are offline';
  if (!nanoReady && !nanoSoon) {
    return { engine: 'rules', strategy: 'none',
      reason: `${why} and Gemini Nano isn't available on this device, so only the built-in rules ran.` };
  }
  if (long) {
    return { ...deviceChunked, reason: `${why}: long document split into parts and read by Gemini Nano on this device.` };
  }
  return { engine: 'firebase', mode: MODE.ONLY_ON_DEVICE, strategy: 'single',
    reason: `${why}: read by Gemini Nano on this device. Nothing was sent to the cloud.` };
}

if (long) {
  return {
    engine: 'firebase',
    mode: MODE.ONLY_IN_CLOUD,
    strategy: 'single',
    reason: `Long document (${Math.round(p.chars / 1000)}k characters) exceeds the on-device window, so it was sent to Gemini in the cloud.`,
    // If the cloud call fails (quota, network), read it on-device in parts.
    fallback: nanoReady || nanoSoon
      ? { ...deviceChunked, reason: 'Cloud was unreachable, so Gemini Nano read it on this device in parts.' }
      : null,
  };
}

A few things worth pointing out:

Privacy wins over quality. If the user picked "On-device only", nothing in the rest of the function can send text to the cloud. The check happens first, before length is even considered.

Long documents go to the cloud with a planned fallback. The SDK's own fallback (PREFER_IN_CLOUD falling back to Nano) would try to fit the whole document into Nano's window, which fails for anything long. So for long text I ask for ONLY_IN_CLOUD and attach my own fallback plan: read it on the device in chunks.

Short documents prefer the device. For the default "Balanced" setting with a short text, PREFER_ON_DEVICE lets the SDK do what it is good at: use Nano if it is ready, otherwise go to the cloud.

There is always a plan that works. No Firebase config, no network, no Nano: you still get the rules-only result. The extension never shows an empty error screen.

Because the function is pure, testing every row of the table is cheap. The repo has 30 node:test tests; the router tests just call planAnalysis({...}) with different facts and check the mode and strategy.

Tuning without shipping a new build

The four limits come from Firebase Remote Config, with the defaults above as a fallback. Chrome Web Store reviews take time, and guessing Nano's real-world limits right on the first try is unlikely, so being able to change on_device_max_chars from the console is worth the small setup cost. The fetch has a short timeout so a slow network never blocks the first analysis:

try {
  await Promise.race([fetchAndActivate(rc), new Promise((r) => setTimeout(r, timeoutMs))]);
} catch {
  /* offline or not set up: defaults apply */
}

2. One model factory, one schema

The Firebase client (src/firebase/client.js) is the only module that imports Firebase. It exposes a single runModel({ mode, prompt, schema }) function that the engine receives as a dependency. In tests, a fake runModel is injected instead.

The model factory passes the same JSON schema to both sides: as responseJsonSchema for the cloud model, and as responseConstraint for the Prompt API on the device:

getGenerativeModel(ai, {
  mode,
  inCloudParams: {
    model: cloudModel,
    generationConfig: schema
      ? { responseMimeType: 'application/json', responseJsonSchema: schema, temperature: 0.2 }
      : { temperature: 0.2 },
  },
  onDeviceParams: {
    createOptions: { ...DEVICE_CREATE_OPTIONS },
    promptOptions: schema ? { responseConstraint: schema } : undefined,
  },
});

This means the rest of the pipeline never needs to know which model answered. Both return { tldr, points, risks[] }, where each risk has a category, severity, quote and why.

Two practical details from runModel:

// Cloud calls get a hard timeout so a bad network or App Check problem
// degrades to the next plan instead of spinning forever. On-device calls
// get longer: Gemini Nano on a slow GPU can take a while per chunk.
const limit = mode === InferenceMode.ONLY_ON_DEVICE ? 180000 : 30000;

and, after the call:

const source =
  response.inferenceSource ??
  (mode === InferenceMode.ONLY_ON_DEVICE ? InferenceSource.ON_DEVICE : InferenceSource.IN_CLOUD);
return { text: response.text(), source };

inferenceSource is important. With a PREFER_* mode you don't know in advance where the call will run. The engine collects the sources of every call, and at the end it rewrites the reason shown to the user so it describes what actually happened:

// Tell the truth about where inference actually happened.
if (plan.mode === MODE.PREFER_ON_DEVICE && sources.has('in_cloud') && !sources.has('on_device')) {
  reason = 'Gemini Nano isn’t ready on this device, so Firebase AI Logic fell back to Gemini in the cloud.';
}

If you tell users "this stayed on your device", it had better be true.

One more gotcha: LanguageModel.availability() can hang (I saw it in headless Chrome and while the component updater was busy). Never await it bare in a UI path:

const timeout = new Promise((r) => setTimeout(() => r('unavailable'), 3000));
return await Promise.race([LM.availability(DEVICE_CREATE_OPTIONS), timeout]);

3. Map-reduce on Gemini Nano

When a long document has to stay on the device, either because the user chose "On-device only" or because the cloud call failed, the engine splits it into chunks of about 9,000 characters and runs one Nano call per chunk.

Chunking cuts on paragraph boundaries first, then sentences, and only hard-cuts a single giant sentence as a last resort. That keeps clauses intact, which matters for the quote check later:

for (const para of clean.split(/\n\n+/)) {
  if (para.length <= maxChars) add(para, '\n\n');
  else for (const s of splitSentences(para)) add(s, ' ');
}

The map step tells the model which part it is reading, so it doesn't write "this document is about..." as if the chunk were everything:

export function buildPrompt(text, { kind = 'legal text', part, parts } = {}) {
  const scope = parts > 1 ? ` This is part ${part} of ${parts} of the ${kind}.` : '';
  return `${STYLE}\n\nDocument type: ${kind}.${scope}\n\n<document>\n${text}\n</document>`;
}

Then the engine runs the chunks one after another. Sequential is intentional: Nano runs on the local GPU, and parallel sessions just compete for it. Each part updates the progress text, because a 6-part document on a laptop GPU can take a while:

for (let i = 0; i < chunks.length; i++) {
  progress(`Gemini Nano is reading part ${i + 1} of ${chunks.length} on this device…`);
  parts.push(await callModel(p.mode,
    buildPrompt(chunks[i], { kind, part: i + 1, parts: chunks.length }),
    RESULT_SCHEMA, chunks[i]));
}
const combined = combineChunkResults(parts);

combineChunkResults does the cheap part of the reduce without a model: it de-duplicates bullet points and merges risks by category. Only the TL;DR needs language understanding, so the reduce prompt is tiny. It sends the per-part summaries, not the document:

export function buildReducePrompt(tldrs, kind = 'legal text') {
  const list = tldrs.map((t, i) => `${i + 1}. ${t}`).join('\n');
  return (
    `These are plain-language summaries of consecutive parts of one ${kind}.\n${list}\n\n` +
    'Write one plain summary of the whole document in at most two sentences. ' +
    'Answer only with JSON: {"tldr": "..."}'
  );
}

If the reduce call fails, the joined part summaries are kept. Not elegant, but correct.

Two caps keep this bounded: maxChunks (default 8) limits how many parts the AI reads, and the UI adds a note when that happens. The rule engine, which is plain regex and scoring, still scans the full text.

4. Don't trust quotes you didn't check

Small models are good at sounding confident. When Plain Terms says "this clause lets them sell your data", it shows the quote it is based on. If that quote isn't in the document, the finding is worse than useless.

So every risk the model returns goes through isGrounded() before the user sees it:

/**
 * Is `quote` really in `source`? Exact match after squashing, or at least 80%
 * of its 4-word shingles present (tolerates small punctuation/ellipsis edits).
 */
export function isGrounded(quote, source) {
  const q = squash(quote);
  if (q.length < 12) return false;
  const src = typeof source === 'string' ? squash(source) : source.squashed;
  if (src.includes(q)) return true;
  const words = q.split(' ');
  if (words.length < 6) return false;
  let hit = 0;
  let total = 0;
  for (let i = 0; i + 4 <= words.length; i++) {
    total++;
    if (src.includes(words.slice(i, i + 4).join(' '))) hit++;
  }
  return total > 0 && hit / total >= 0.8;
}

squash() lowercases, normalises curly quotes and strips punctuation, so β€œWe may share…” and we may share compare equal. The exact-substring check handles the common case. The shingle check handles a model that dropped a comma or replaced a middle phrase with "…". Very short quotes are rejected outright, because a 3-word "quote" like "we may share" matches almost any privacy policy and proves nothing.

The check is done against the text the model actually saw. For a chunked run that is the chunk, not the whole document, so a model can't "quote" from a part it never read.

Findings that fail are dropped, but not silently. The count goes into the result notes:

if (ai.dropped) notes.push(`${ai.dropped} AI finding${ai.dropped === 1 ? ' was' : 's were'} dropped because the quote wasn’t in the document.`);

Finally the model's grounded risks are merged with the rule engine's findings. When both found the same clause, the model's wording wins and the card is marked ai+rules. When only the rules found something, it is still shown. This gives a floor of known patterns (auto-renewal, forced arbitration, data sale) that no model failure can remove.

5. Degrade, don't fail

Put together, analyzeDocument() never throws because of a model:

  1. Run the rules (always).
  2. Execute the plan.
  3. If it fails and the plan has a fallback, run the fallback.
  4. If that fails too, return the rules-only result with a reason that says so.

There is also one repair attempt for bad JSON, since small models sometimes wrap JSON in prose or code fences:

let obj = parseModelJson(res.text);
if (!obj) {
  // One repair attempt: small models occasionally emit prose.
  const retry = await ctx.runModel({ mode, prompt: `${prompt}\n\nReturn ONLY valid JSON.`, schema });
  sources.add(retry.source);
  obj = parseModelJson(retry.text);
}

parseModelJson strips fences and takes the text between the first { and the last }, which handles nearly every malformed reply I saw.

Extension-specific notes

  • App Check in MV3. Firebase AI Logic requires App Check. reCAPTCHA can't load in a Manifest V3 extension because remote scripts are forbidden, so the extension uses the debug provider with a per-install token shown in its settings panel; the hosted web app uses reCAPTCHA. No token is committed to the repo.
  • User activation. initializeDeviceModel() (which may start the Nano download) has to run from a user gesture, so it's called from the button click, not on load.
  • Permissions. activeTab, scripting, sidePanel, storage, contextMenus. No blanket host permissions.

What I'd keep for any hybrid app

  • Make the routing decision a pure function with a human-readable reason, and test it like a table.
  • Use one output schema for both models, so nothing downstream cares where the answer came from.
  • Report inferenceSource, not your intention.
  • Plan for documents bigger than the device window yourself; the SDK's fallback can't chunk for you.
  • Verify anything a model claims to quote, and say how many findings you dropped.

The full code, including the rule engine and the tests, is at https://github.com/agebreakco-prog/plain-terms, and you can try the web app at https://plain-terms.web.app. On-device inference needs desktop Chrome with Gemini Nano enabled; without it the app falls back to the cloud, then to rules, as described above. It explains text; it isn't legal advice.

πŸ“° Read the original article on Dev.to WebDev

Originally published by Dev.to WebDev. Aggregated on AIWithGhost for educational purposes β€” full credit and traffic to the original publisher.