Dev.to Security ๐Ÿ” Cybersecurity ๐Ÿ‘ 0 ๐Ÿ“– 11 min read

Building a Small Link Unfurler in Node.js: Tag Fallbacks, Relative Image URLs, Charsets and Not Fetching Your Own Network

Most writing about link previews is from the publisher's side: which tags to put in your <head> so your page looks good when shared. This post is from the other side. Say you're building a chat feature, a comment box or

Most writing about link previews is from the publisher's side: which tags to put in your <head> so your page looks good when shared. This post is from the other side. Say you're building a chat feature, a comment box or a bookmarks app, and you want pasted links to turn into cards. Now you're the one fetching strangers' pages, and you have to decide what to do when the tags are missing, duplicated, relative or in the wrong encoding.

I build OG Scope, a free link preview checker, so I've spent a lot of time looking at how real pages publish (and fail to publish) their share tags. Below is a small unfurler in one Node file with no dependencies. It's not a drop-in replacement for a battle-tested library, but writing it makes every decision visible, and the decisions are the interesting part.

It does five things:

  1. Fetches the page with redirects handled by hand, a timeout and a size limit
  2. Refuses to fetch private or local addresses, on every redirect hop
  3. Decodes the bytes using the right character set
  4. Reads <meta> tags, <title> and the icon from the <head> only
  5. Builds a card with a clear fallback order and absolute URLs

Everything runs on Node 18 or newer (I tested on Node 24), using the built-in fetch and test runner.

The fallback order

Before any code, decide what wins when a page publishes more than one candidate. This is the order I use, which is close to what you'll see most apps do, though each product has its own rules and they change:

Card field First choice Then Last resort
Title og:title twitter:title, then <title> the hostname
Description og:description twitter:description, then meta name="description" nothing
Image og:image:secure_url / og:image twitter:image no image
Site name og:site_name the hostname without www.

Two rules matter as much as the order:

  • Empty values don't count. <meta property="og:title" content=""> happens more than you'd think (a CMS field left blank). Treat it as missing and fall through.
  • The first duplicate wins. Pages often print og:title twice, once from the theme and once from an SEO plugin. Pick one rule and stick to it. Taking the first is the common choice, and it's predictable.

Fetching safely

An unfurler is a server that fetches any URL a user types. That's the textbook setup for server-side request forgery (SSRF): someone pastes http://169.254.169.254/ or http://localhost:6379/ and your server politely requests it from inside your network. So the fetch step checks the address before every request, including after each redirect, because a public URL can redirect to a private one.

// unfurl.mjs: fetch a page and build a link preview the way a chat app would
import { lookup } from 'node:dns/promises';
import { isIP } from 'node:net';

const MAX_BYTES = 512 * 1024;   // stop reading after 512 KB
const MAX_REDIRECTS = 5;
const TIMEOUT_MS = 5000;

function isPrivateAddress(ip) {
  if (isIP(ip) === 6) {
    const v = ip.toLowerCase();
    if (v.startsWith('::ffff:')) return isPrivateAddress(v.slice(7));
    return v === '::1' || v === '::' || v.startsWith('fc') || v.startsWith('fd') || v.startsWith('fe80');
  }
  const [a, b] = ip.split('.').map(Number);
  return a === 10 || a === 127 || a === 0 ||
    (a === 169 && b === 254) ||
    (a === 172 && b >= 16 && b <= 31) ||
    (a === 192 && b === 168) ||
    (a === 100 && b >= 64 && b <= 127);
}

async function assertPublicUrl(url) {
  if (url.protocol !== 'http:' && url.protocol !== 'https:') throw new Error('Only http and https links are previewed');
  const host = url.hostname.replace(/^\[|\]$/g, '');
  const addresses = isIP(host) ? [{ address: host }] : await lookup(host, { all: true });
  if (addresses.some(({ address }) => isPrivateAddress(address))) throw new Error('Refusing to fetch a private address');
}

async function fetchHead(startUrl) {
  let url = new URL(startUrl);
  for (let hop = 0; hop <= MAX_REDIRECTS; hop++) {
    await assertPublicUrl(url);   // check every hop, not just the first
    const res = await fetch(url, {
      redirect: 'manual',
      signal: AbortSignal.timeout(TIMEOUT_MS),
      headers: { 'user-agent': 'ExampleUnfurler/1.0 (+https://example.com/bot)', accept: 'text/html' },
    });
    if (res.status >= 300 && res.status < 400 && res.headers.get('location')) {
      url = new URL(res.headers.get('location'), url);
      continue;
    }
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
    const type = res.headers.get('content-type') || '';
    if (!type.includes('html')) throw new Error(`Not an HTML page (${type || 'no content-type'})`);
    return { finalUrl: url, contentType: type, bytes: await readLimited(res) };
  }
  throw new Error('Too many redirects');
}

A few choices worth explaining:

  • redirect: 'manual' so we see each Location and check it. With automatic redirects, fetch would follow a hop to 127.0.0.1 without asking.
  • A timeout with AbortSignal.timeout(). A slow page shouldn't hold a chat message hostage. Five seconds is generous.
  • An honest user agent with a URL that explains the bot. Some sites block unknown crawlers; a clear name at least lets site owners decide.
  • Only HTML. If someone pastes a link to a 2 GB video, we stop at the content type.

One honest limitation: lookup() resolves the name, then fetch resolves it again. A hostile DNS server could answer differently the second time (DNS rebinding). For a production service, fetch through an egress proxy that enforces the allow rules, or pin the resolved IP with a custom agent. The check above still stops the casual cases, which are the common ones.

Reading only what you need

Share tags live in <head>, so there's no reason to download a 3 MB page. The reader stops at 512 KB or as soon as a chunk contains </head>, whichever comes first:

async function readLimited(res) {
  const chunks = [];
  let total = 0;
  for await (const chunk of res.body) {
    chunks.push(chunk);
    total += chunk.length;
    if (total >= MAX_BYTES) break;                                    // big page: stop early
    if (Buffer.from(chunk).toString('latin1').includes('</head>')) break; // tags live in <head>
  }
  return Buffer.concat(chunks).subarray(0, MAX_BYTES);
}

Getting the characters right

If you decode everything as UTF-8, a Shift_JIS or Windows-1252 page turns its title into mojibake. The charset can come from the Content-Type header or from a <meta charset> near the top of the document. The header wins when both exist:

function decodeHtml(bytes, contentType) {
  const fromHeader = /charset=["']?([\w-]+)/i.exec(contentType)?.[1];
  const sniff = bytes.subarray(0, 2048).toString('latin1');
  const fromMeta = /<meta[^>]+charset=["']?([\w-]+)/i.exec(sniff)?.[1];
  try { return new TextDecoder(fromHeader || fromMeta || 'utf-8').decode(bytes); }
  catch { return new TextDecoder('utf-8').decode(bytes); }   // unknown label
}

TextDecoder in Node supports the common legacy encodings, and the try catches labels it doesn't know.

Parsing the head without a parser

For a production service I'd reach for a real HTML parser. For share tags, a careful pattern match over <meta> and <link> tags covers what pages actually publish: double quotes, single quotes, unquoted values, attributes in either order, and entities like &amp; inside content.

const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ' };
function decodeEntities(s) {
  return s.replace(/&(#x[\da-f]+|#\d+|\w+);/gi, (m, e) => {
    if (e[0] === '#') {
      const n = e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);
      return Number.isFinite(n) && n <= 0x10ffff ? String.fromCodePoint(n) : m;
    }
    return ENTITIES[e.toLowerCase()] ?? m;
  });
}

function parseAttributes(tag) {
  const attrs = {};
  for (const [, name, , v1, v2, v3] of tag.matchAll(/([\w:-]+)\s*(=\s*(?:"([^"]*)"|'([^']*)'|([^\s>]+)))?/g)) {
    attrs[name.toLowerCase()] = decodeEntities(v1 ?? v2 ?? v3 ?? '');
  }
  return attrs;
}

export function readTags(html) {
  const head = html.split(/<\/head>/i)[0];
  const meta = {};
  for (const [tag] of head.matchAll(/<meta\b[^>]*>/gi)) {
    const a = parseAttributes(tag.slice(5));
    const key = (a.property || a.name || '').toLowerCase();
    if (key && a.content !== undefined && !(key in meta)) meta[key] = a.content.trim(); // first one wins
  }
  const title = /<title[^>]*>([\s\S]*?)<\/title>/i.exec(head)?.[1];
  const icon = [...head.matchAll(/<link\b[^>]*>/gi)].map(([t]) => parseAttributes(t.slice(5)))
    .find((a) => /(^|\s)icon(\s|$)/i.test(a.rel || ''))?.href;
  return { meta, title: title ? decodeEntities(title).replace(/\s+/g, ' ').trim() : '', icon };
}

Note the split(/<\/head>/i)[0]: tags after </head> are ignored. Some pages print share tags in the body by mistake, and many consumers won't see them, so neither should we.

Building the card

This is where the fallback table becomes code. URLs are resolved against the final URL after redirects, not the one the user pasted, because a relative og:image like /img/card.png belongs to the page that served it. Anything that isn't http or https after resolving is dropped, which quietly removes javascript: and data: values.

const first = (...values) => values.find((v) => typeof v === 'string' && v.trim() !== '')?.trim();
const clip = (s, n) => (s && [...s].length > n ? [...s].slice(0, n - 1).join('').trimEnd() + 'โ€ฆ' : s);

export function buildPreview({ meta, title, icon }, pageUrl) {
  const abs = (u) => { try { const r = new URL(u, pageUrl); return /^https?:$/.test(r.protocol) ? r.href : undefined; } catch { return undefined; } };
  return {
    url: pageUrl.href,
    siteName: first(meta['og:site_name'], pageUrl.hostname.replace(/^www\./, '')),
    title: clip(first(meta['og:title'], meta['twitter:title'], title, pageUrl.hostname), 120),
    description: clip(first(meta['og:description'], meta['twitter:description'], meta.description), 300),
    image: abs(first(meta['og:image:secure_url'], meta['og:image'], meta['og:image:url'], meta['twitter:image'], meta['twitter:image:src']) ?? ''),
    imageAlt: first(meta['og:image:alt'], meta['twitter:image:alt']),
    largeImage: meta['twitter:card'] === 'summary_large_image' || Boolean(meta['og:image:width'] && Number(meta['og:image:width']) >= 600),
    icon: abs(icon ?? '/favicon.ico'),
  };
}

export async function unfurl(link) {
  const { finalUrl, contentType, bytes } = await fetchHead(link);
  return buildPreview(readTags(decodeHtml(bytes, contentType)), finalUrl);
}

The title is clipped to 120 characters and the description to 300, counting code points ([...s]) so an emoji or an Urdu character isn't cut in half. largeImage is a hint for your UI: show a wide image when the page asks for summary_large_image or declares a reasonably wide image, and a small thumbnail otherwise.

Tests

Node's built-in runner is enough. These cover the fallback order, duplicates, empty values, tags in the body and a javascript: image:

import { test } from 'node:test';
import assert from 'node:assert/strict';
import { readTags, buildPreview } from './unfurl.mjs';

const page = new URL('https://example.com/blog/post?ref=chat');

test('og tags win over twitter tags and <title>', () => {
  const html = `<head><title>Post | Example</title>
    <meta property="og:title" content="Fish &amp; Chips">
    <meta name="twitter:title" content="Twitter title">
    <meta property="og:image" content="/img/card.png"></head>`;
  const p = buildPreview(readTags(html), page);
  assert.equal(p.title, 'Fish & Chips');
  assert.equal(p.image, 'https://example.com/img/card.png');
});

test('falls back to twitter tags, then <title> and meta description', () => {
  const p = buildPreview(readTags(`<head><title> Just a\n title </title><meta name="description" content="Plain description"><meta name="twitter:image" content="card.png"></head>`), page);
  assert.equal(p.title, 'Just a title');
  assert.equal(p.description, 'Plain description');
  assert.equal(p.image, 'https://example.com/blog/card.png');
});

test('first duplicate tag wins, empty values are skipped', () => {
  const p = buildPreview(readTags(`<head><meta property="og:title" content=""><meta property="og:description" content="One"><meta property="og:description" content="Two"><title>T</title></head>`), page);
  assert.equal(p.title, 'T');
  assert.equal(p.description, 'One');
});

test('ignores tags after </head> and javascript: images', () => {
  const p = buildPreview(readTags(`<head><meta property="og:image" content="javascript:alert(1)"></head><body><meta property="og:title" content="Body tag"></body>`), page);
  assert.equal(p.image, undefined);
  assert.equal(p.title, 'example.com');
});

test('single quotes, unquoted values and long titles', () => {
  const long = 'A'.repeat(200);
  const p = buildPreview(readTags(`<head><meta content='${long}' property='og:title'><meta name=twitter:card content=summary_large_image></head>`), page);
  assert.equal([...p.title].length, 120);
  assert.ok(p.title.endsWith('โ€ฆ'));
  assert.equal(p.largeImage, true);
});

$ node --test unfurl.test.mjs
โ„น tests 5
โ„น pass 5
โ„น fail 0

And a real page:

$ node unfurl.mjs https://www.ogscope.pk/og-image-checker
{
  "url": "https://www.ogscope.pk/og-image-checker",
  "title": "OG Image Checker: What It Tests ยท OG Scope",
  "image": "https://www.ogscope.pk/og-image-checker-what-it-tests.webp",
  "largeImage": true,
  ...
}
$ node unfurl.mjs http://127.0.0.1/
Refusing to fetch a private address

What I'd add before shipping it

  • Cache the result per URL for a while (an hour or a day). Pasting the same link twenty times in a channel shouldn't mean twenty fetches. This is also why publishers see old cards after fixing a page: the consumer cached it.
  • Proxy the image through your own server instead of hot-linking it into users' browsers, so the image host doesn't learn who viewed the chat, and so you can enforce a size limit on the image too.
  • Rate-limit per user and per domain, so nobody uses your unfurler to hammer a site.
  • Respect opt-outs your product decides to honour, and show the plain link when anything fails. A card is a nice-to-have; the link is the message.

Checking your output against a reference

When your unfurler and another app disagree about a page, it helps to see the raw tags the page publishes. I use the Open Graph preview in OG Scope for that: it reads the public HTML on the server, lists the Open Graph and Twitter tags, the title, description and canonical, and draws estimated cards for several apps. If both tools see the same tags but show different cards, the difference is in the fallback rules, which, after this post, you now get to choose on purpose.

๐Ÿ“ฐ Read the original article on Dev.to Security

Originally published by Dev.to Security. Aggregated on AIWithGhost for educational purposes โ€” full credit and traffic to the original publisher.

Share ๐• Twitter โœˆ Telegram โ†— Reddit ๐ŸŸง HN
โ† News ๐Ÿ” Cybersecurity ๐Ÿ  Home