I put a 740M vision model in a browser tab and finally searched my meme folder

I have about 1,600 memes on this laptop. Filenames like IMG_4821.PNG, download (14).jpg, screenshot-2024-11-02.png. I knew exactly which one I wanted last week and spent four minutes scrolling a grid. The thing I remembered was the picture: a dog sitting in a burning room saying this is fine. There is no filename to search for that.

So I built Meme Vault. You drop a folder in, the browser embeds every image locally, and then you type the memory instead of the name. It runs in one tab with no server, no API key and no upload, and after the model downloads it keeps working with the wifi off.

This post is the build. What the model actually is, how the index is shaped, the four things that cost me the most time, and the numbers from the test run.

Why this needed a multimodal embedding model

Keyword search is useless here. The property that makes a meme findable is what it looks like, and nobody has ever tagged theirs. The tool you need is one that maps a sentence and an image into the same vector space, so "a dog in a burning room" lands near the picture of the dog in the burning room.

That is an embedding model, and until recently the good ones needed a GPU in a datacentre. Google DeepMind's EmbeddingGemma 2 changed the arithmetic:

• 740M parameters total, split into a 270M text model, a 170M vision encoder and a 300M audio encoder
• one shared 768-dimensional space for text, images, video frames and audio
• quantised to q4, the text and vision graphs come to 284 MB combined
• it runs on WebGPU through Transformers.js in a normal page, no extensions

The last point is the one that matters. A 740M model is not going to fit in a naive page, but with q4 weights on the GPU it fits comfortably, and it embeds an image in about a second on an M4. Suddenly the index can live where the files live.

The shape of the app

One page, three processes, one model.

┌─────────────────────────────── browser tab ───────────────────────────────┐
│                                                                           │
│  ┌──────────────┐   blobs    ┌──────────────────────────────────────────┐  │
│  │  Main thread │ ─────────▶ │           Embedding worker               │  │
│  │              │            │                                          │  │
│  │ React UI     │            │  EmbeddingGemma 2 (ONNX, WebGPU)         │  │
│  │ canvas       │ ◀───────── │  ├─ text backbone  270M  (q4, 174 MB)    │  │
│  │ thumbnails   │  Float32   │  ├─ vision encoder 170M  (q4, 109 MB)    │  │
│  │              │  [768]     │  └─ audio encoder 300M  disabled         │  │
│  └──────┬───────┘            └──────────────────────────────────────────┘  │
│         │                                                                 │
│         ▼                                                                 │
│  ┌─────────────────┐        ┌──────────────────────────────────────────┐  │
│  │ IndexedDB       │        │ Cache Storage                            │  │
│  │ ├ meta  (thumb, │        │   model graphs, cached after first load  │  │
│  │ │        vec)   │        └──────────────────────────────────────────┘  │
│  │ └ files (bytes) │                                                      │
│  └─────────────────┘                                                      │
└───────────────────────────────────────────────────────────────────────────┘

Two decisions are worth explaining.

The model lives in a worker. A forward pass on a 740M model blocks for 200 to 300 ms. On the main thread that is a visibly janky page. In a worker, the UI can animate a progress bar and keep the search box responsive while a folder is being indexed.

The database has two stores. meta holds the thumbnail and the 768-d vector. files holds the original bytes. Startup reads only meta, so a vault of 2,000 memes opens without decoding 2,000 full-size images. The originals are touched only when you open one.

Step 1: normalise everything before the model sees it

The model takes images through RawImage.fromBlob, and browsers are inconsistent about what they can decode. HEIC from an iPhone, animated GIFs, SVG, AVIF. So I do not hand files to the model. I decode with createImageBitmap, draw to a canvas at 640 px, and re-encode as JPEG:

const source = await decode(file);          // createImageBitmap, with an <img> fallback for SVG
const full  = draw(source, 640);            // what the model sees
const thumb = draw(source, 320);            // what the grid shows

const embed = await toBlob(full.canvas, 'image/jpeg', 0.9);
const thumbBlob = await toBlob(thumb.canvas, 'image/webp', 0.82);

That one step buys three things: any format the browser can render becomes a valid model input, a 4 MB photo becomes a 90 KB transfer into the worker, and the grid gets a thumbnail that is cheap to keep in IndexedDB.

I paint a white background first, because memes are frequently transparent PNGs and the embed should not depend on whatever the page background happens to be.

Step 2: the quantisation trap

This is the thing I would put at the top of the docs if I were writing them.

WebGPU can run the q4 graphs. WebAssembly cannot. The q4, q8 and q4f16 text and vision graphs use an operator called GatherBlockQuantized, and the ONNX Runtime WASM build has no kernel for it. So a fallback path that reuses the same dtype fails with an error that tells you nothing about the
real cause.

The fix is to pick the dtype by device:

export const DTYPE = {
  webgpu: { model: 'q4',   vision_encoder: 'q4'   },   // 284 MB total
  wasm:   { model: 'fp16', vision_encoder: 'fp16' },   // 879 MB total
};

879 MB is not a rounding error. It is the difference between a demo people finish and a demo
people abandon. So the app detects the fallback, says so in plain language, and tells you which
browser to use instead. If you build on this, gate the WASM path behind an explicit confirmation.

The second half of the same step is dropping what you do not need:

const config: any = await AutoConfig.from_pretrained(MODEL_ID);
config.audio_config = null;          // the 300M audio encoder never downloads

Audio is 189 MB on its own. If the app does not embed sound, do not ship the encoder.

Step 3: the silent batch merge

Here is a bug that produces no error and no wrong shape. It just quietly halves your data.

The processor signature is processor(text, images). For a batch of images, each image has to be its own list:

const images = await Promise.all(batch.map(async (blob) => [await RawImage.fromBlob(blob)]));
const out = await run(null, images);     // processor(null, [[imgA], [imgB]])

Pass a flat list, processor(null, [imgA, imgB]), and the processor treats the pair as one multimodal input. You get one vector back for two images. If you then zip vectors against filenames in order, every meme after the first gets someone else's vector, and search looks "vaguely bad" rather than broken. I lost an hour to this.

The batch also has a size budget, because a batch pads to its longest item and an image costs 282 tokens:

export const IMAGE_TOKENS = 282;                              // 280 soft tokens + markers
export const BATCH_BUDGET = { webgpu: 2600, wasm: 300 };      // 9 images per batch on WebGPU

Above roughly 2,700 tokens on WebGPU the dispatch limit is exceeded. On WASM I run one image per batch, because the fp16 graph plus a padded batch will exhaust the 4 GB heap.

Step 4: the index is just vectors in IndexedDB

Nothing exotic. The 768-d vector goes next to the thumbnail:

meta:  { id, name, addedAt, bytes, width, height, thumb, thumbType, vector }
files: { id → original Blob }

Ingest is chunked on purpose. Sixteen images per group on WebGPU, four on WASM, and after each group the vectors are written and the React state updated. If the tab dies at image 900 of 2,000, the next load resumes from 900 instead of zero. On a folder of that size this is the difference between a tool and a toy.

The one lifetime detail that matters: the tensor you read is backed by GPU memory, so copy it before disposing:

const flat = new Float32Array(output.sentence_embedding.data);   // copy first
for (const tensor of Object.values(output)) tensor?.dispose?.();  // then release

Dispose first and you read released memory. It will mostly look like it works, which is worse than crashing.

Step 5: search is a dot product, and that is enough

The model L2-normalises every vector, so cosine similarity collapses to a dot product:

let dot = 0;
for (let i = 0; i < vector.length; i++) dot += vector[i] * queryVector[i];

For 2,000 memes that is 1.5 million multiply-adds, about a millisecond. There is no vector database, no ANN index, no WASM SIMD gymnastics. You do not need any of that until roughly 100k items, and if I ever hit that number I will reach for a quantised int8 comparison before I reach
for HNSW.

Because it is that cheap, search runs on every keystroke with a 400 ms debounce. No search button, no spinner. It feels like filtering a list, which is the correct feeling.

Proof, not vibes

I do not trust "it works" from a screenshot, so the repo has an end-to-end harness that drives real headless Chrome with WebGPU, indexes real images, and asserts ranking:

$ python3 scripts/e2e.py
model ready in 57.1s on webgpu
indexed 8 images in 16.7s
'a cat'                  -> cat.jpg (0.71), 8 hits
'pizza'                  -> pizza.jpg (0.73), 8 hits
'the eiffel tower'       -> eiffel.jpg (0.75), 8 hits
'a banana'               -> meme-banana.png (0.77), 8 hits
'a painting of a woman'  -> monalisa.jpg (0.73), 8 hits
'a golden retriever'     -> meme-dog.png (0.75), 8 hits
OFFLINE search 'a cat'   -> cat.jpg (0.71)
CHECKS {"rank1": true, "offline_top": true, "persistence": true,
        "lightbox": true, "no_console_errors": true}

The offline line is a real context.set_offline(True) followed by a search, not a flag I set in the UI. Persistence is a real reload with a query after it.

Two details from that harness are worth stealing for any browser model project.

First, headless Chrome does run WebGPU on macOS, with the right flags:

p.chromium.launch(channel="chrome", headless=True, args=[
    "--enable-unsafe-webgpu", "--enable-features=WebGPU", "--use-angle=metal"
])

Second, the assertions need a seam. My first version waited for a score badge to appear, which passes instantly because the previous query's scores are still in the DOM. Every query after the first was asserting against stale results, and all six "passed" with identical rankings. The app now exposes data-search-seq, and the test waits for that counter to increase:

before = int(page.evaluate("() => document.querySelector('.app')?.dataset.searchSeq ?? '0'"))
page.fill(".searchbar input", text)
page.wait_for_function(f"() => Number(document.querySelector('.app')?.dataset.searchSeq ?? 0) > {before}")

A test that cannot distinguish new state from old state is not a test.

The one result I did not expect

The dog meme I composited out of a dog photo beat the dog photo itself for "a golden retriever", 0.75 to 0.71. The banana meme tied the banana photo at 0.77.

That is the model reading the picture and treating the caption as more picture. For this product it is exactly right, because a meme's identity is the composition, not the base image. It also taught me not to write an assertion that expects the "source" image to win. Both are correct answers.

What I would do differently

• No fixed similarity threshold. Correct hits land around 0.70 to 0.77 and unrelated images around 0.50 to 0.60. That band is too narrow for a constant, so the app ranks and shows scores instead of filtering on a magic number. The "close matches only" toggle is relative to the top hit for the same reason.
• The score is not a probability. It is a cosine similarity. Showing it as a percentage would be lying with typography.
• Text in memes is not OCRed. A query that only matches the caption can miss. Adding a small OCR pass on the thumbnail and blending a text score is the obvious next step, and also the point where "one model in a tab" becomes "three models in a tab".

Try it

npm install
npm run dev

Chrome or Edge 113 or newer. The first load downloads 284 MB once, then it is cached forever and the tab works offline. test-images/ has six photos and two memes to start with, or point it at your own folder.

The source is a static Vite app, about 900 lines of TypeScript across nine files, plus the Playwright harness. If you build something on it, the two things worth more than the UI are the ingest budget and the honest limits.

Code & more: https://www.dailybuild.xyz/project/291-meme-vault