AI Embeddings Break Columnar Assumptions in Parquet and Arrow

TL;DR — Columnar engines like DuckDB and Arrow assume you scan a few columns and filter on values. An embedding column violates every one of those assumptions: it is wide, high-entropy, never filtered by value, and always read whole. Treat it as a separate physical tier, use fixed-size layouts, and late-materialize vectors after metadata filters.

Columnar storage is a bet on a specific access pattern: read a few columns out of many, filter on their values, and aggregate. Parquet, Arrow, DuckDB, and GPU dataframe engines are all optimized around that bet. It's a good bet for analytics.

Then we add an embedding column and quietly lose it.

My thesis: an embedding column is the one column type where the columnar assumptions invert, and most "slow AI data prep" is the result of storing it like any other column. The fix isn't a faster engine. It's treating vectors as their own physical tier.

The access pattern flips

Look at what a normal analytical column gets for free:

• Value predicates. Min/max statistics, dictionary pages, and bloom filters let the engine skip data based on what's inside.

• Compression. Low-cardinality strings, sorted integers, and timestamps compress dramatically with dictionary, run-length, and delta encodings.

• Narrow reads. The query touches three columns out of fifty.

Now look at an embedding column:

• No value predicates. Nobody writes a filter on the seventh dimension of a vector. Min/max stats on a float inside a vector are noise. You pay for the metadata and never use it.

• Poor compression. Learned floats are high-entropy. Dictionary and run-length encodings have nothing to grab. Byte-oriented transforms for floats help a bit, and general-purpose codecs buy you little for the CPU they burn.

• All-or-nothing reads. A consumer wants the whole vector or none of it. There is no useful "read half of this column."

So the column that is usually the largest in your table, by bytes, gets none of the features that justify a columnar format. You're paying the format's costs (row-group overhead, encoding and decoding, nested-type bookkeeping) without collecting its benefits.

Nested lists are the quiet tax

The default way people land embeddings in a table is a variable-length list of floats. In Parquet, nested and repeated data is represented with repetition and definition levels, extra bookkeeping the reader must decode to reconstruct list boundaries. For a variable-length list, that work is legitimate. For a vector where every row has exactly the same dimension, it's pure waste: you're encoding "where does each list end" for data whose answer is a constant.

Arrow has the right abstraction: a fixed-size list. Every row is the same width, so the whole column is one flat contiguous buffer of floats plus a known stride. That layout is what NumPy, a GPU kernel, or an ANN index wants. Row i lives at offset i * dim. No offsets array, no per-row decode.

The practical rule: if the dimension is a schema invariant, encode it as one. Declare fixed-size at the Arrow layer, and verify what your Parquet writer and reader actually do with it end to end. Whether you get a zero-copy view or a silent conversion back through variable-length lists depends on the toolchain, and the failure mode is a slow pipeline, not an error. Check by measuring a round trip, not by trusting the type name.

Split the vector from the metadata

The bigger win is physical, not logical. In a typical embeddings table you have a handful of small, filterable columns (document id, source, timestamp, tenant, language) and one huge opaque blob of floats. Storing them in the same files means every metadata scan drags row groups sized for vectors.

Row-group sizing is where this bites. A row group is the unit of skipping and the unit of reading. If the vector column dominates the bytes, a row group sized by byte target holds very few rows. Your metadata statistics then cover tiny spans, footers balloon, and a filter that should prune aggressively instead opens thousands of small groups.

The pattern that works is vertical partitioning:

• A metadata table with the filterable columns, laid out normally, sorted or clustered on your common predicates, with healthy row-group sizes.

• A vector table keyed by a stable row id, stored fixed-size, uncompressed or lightly compressed, with row groups sized for sequential throughput rather than pruning.

• A late join that filters on metadata first, collects the surviving ids, and only then fetches vectors.

This is late materialization, the oldest trick in column stores, applied to the one column where it matters most. DuckDB-style engines do a version of this inside a single query. But they can only do it well if the physical layout lets them avoid touching the vector bytes until the end.

The GPU makes the layout mistake louder

GPU-accelerated analytics changes the arithmetic, but not in the direction people expect. The kernels are fast. The cost that dominates is moving bytes: from storage to host, from host to device, and across the interconnect. A vector column is the heaviest thing you'll move.

That has three consequences:

• Filter before you transfer. Running a metadata predicate on the CPU and shipping only surviving vectors usually beats shipping everything to the device and filtering there. A GPU that sits idle waiting on a transfer isn't accelerating anything.

• Contiguity is the feature. A fixed-size, flat buffer can be handed to the device in one bulk copy. A list column with offsets forces a gather or a repack first, and that repack happens on the slow side of the bus.

• Don't decompress on the critical path. Since vectors barely compress, a heavy codec costs CPU cycles to save few bytes. Storing them with light or no compression often lets the read path run at storage speed.

This is why "just put the dataframe on the GPU" disappoints on embedding-heavy workloads. The compute was never the bottleneck. The layout and the transfer were.

What I'd actually do

If I were setting up an embedding-bearing dataset today, this is the checklist:

• Declare vectors as fixed-size lists at the schema level, and write a test that a round trip preserves that type and avoids a copy.

• Keep vectors out of the table you filter and aggregate. Join by stable id.

• Size row groups differently for each tier: pruning-friendly for metadata, throughput-friendly for vectors.

• Pin the embedding model version next to the vector tier, so a re-embed replaces the whole tier atomically instead of mixing generations in place.

• Choose precision deliberately. Storing vectors at lower precision is a bigger lever than any codec, but it changes retrieval quality, so evaluate it rather than assuming.

None of this is exotic. It's the same discipline column-store designers have always applied: store things the way they're read. The mistake is assuming a vector is "just another column" because the table format let you declare it as one.

Columnar engines are excellent at what they were built for. An embedding column isn't that. Give it its own tier, and the fast engines you already have get fast again.