Before attributing multilingual quantization loss, validate the likelihood path

My long-term goal is to make capable multilingual models practical on personal devices while preserving their usefulness across languages. I want to measure what compression changes, identify which parts of a model need more precision, and test whether allocating that precision differently can preserve quality within the same deployment size budget. Bangla is the language I want to investigate most deeply, with comparisons across other languages to keep the conclusions honest. The eventual test is both linguistic and practical: what can people do with the model, and what does it cost to run on a real device?

The starting point is personal. As a Canadian, I feel a particular connection to Cohere and want to explore what can be built with its multilingual models. This is independent work, not a claim of Cohere affiliation or endorsement. As a Bangla speaker, I also want this work to address a language I use and care about. Choosing Cohere's Tiny Aya and making Bangla the focus brings those interests together. That motivation determines where I start; it does not determine what the results should say.

The path through this article is: the model architecture, three runtime failures, probability validation, position sensitivity, fresh paired baselines, historical comparisons, and the limits of this checkpoint.

1. Establish what the evaluation measures

This article documents an early step toward that larger goal. Before testing a precision-allocation strategy, I needed to establish what the evaluation was actually measuring. The contribution here is a validated measurement procedure and fresh baselines. A sensitivity map, controlled precision-allocation experiments, broader language evaluation and real-device measurements remain later stages, each requiring its own evidence.

A multilingual quantization experiment can produce plausible accuracy numbers while measuring more than compression. The tokenizer, detokenizer, prompt format, request history and answer selection rule all sit between the model weights and the final table. A difference at any of those boundaries can look like a model capability difference.

This project began with a deployment question: how does compression affect Tiny Aya across languages, and could a later precision allocation recover quality at a fixed size budget? Before making that intervention claim, I needed a credible measurement path. That work encountered three distinct failures in the pinned local stack, then led to a fresh paired baseline. The result is an engineering checkpoint: a narrower, auditable comparison, with uncertainty and historical differences retained.

The model is CohereLabs/tiny-aya-global, a 3.35B Cohere2 model at revision af89d219b53ed9b13b8a4645f9c8028973510324. The comparison here uses a BF16 GGUF control and local Q4_K_M built from that pinned BF16 artifact without an importance matrix. It is not a survey of four-bit methods. The original BF16 vLLM reference remains a separately identified experiment.

The task is original-order multiple-choice reading comprehension from the Belebele benchmark (facebook/belebele, revision 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2, CC-BY-SA 4.0) in English, Bangla and Hindi. The model is released under CC-BY-NC 4.0 for non-commercial use. The final fresh comparison uses the same 300 development questions per language in both arms. No held-out evaluation, human semantic annotation, generation experiment or phone-performance result is included.

2. The architecture defines candidate precision tests

The pinned model inspection records 36 decoder layers, hidden width 2,048, MLP width 11,008, 16 query heads and 4 key/value heads with head dimension 128, and a vocabulary of 262,144. Three sliding-attention layers with a 4,096-token window followed by one full-attention layer repeat nine times; the configured maximum is 8,192 positions. These are architecture facts, not measured sensitivity results. Source: pinned model inspection.

Each block normalizes the residual input once, sends that normalized input through attention and a gated SiLU MLP in parallel, then adds both outputs to the residual. After the decoder stack, a final LayerNorm feeds the output head, which shares weights with the token embedding. The Cohere2 implementation makes this parallel computation explicit.

The full model path is shown above a close-up of one decoder block. Teal attention and violet gated MLP branches share normalized input and join the original residual. Amber marks future precision tests; sensitivity and recovery remain unmeasured.

These are hypotheses for recovering quality toward the BF16 comparator within a fixed size budget; they do not promise improvement above BF16. Allocation tests need matched shipped bytes, uniform and random controls, and actual quantizer rounding rather than nominal bit arithmetic alone. Accuracy and changed decisions must be assessed in English alongside Bangla and Hindi. Latency and memory effects require device profiling. KV-cache changes are a distinct runtime optimization and are not validated by these weight-precision tests.

3. Three failures, with different exposure

Native tokenization: changed input IDs

The first failure was in native tokenization. A minimal mismatch involved two spaces followed by 46: the pinned native Tiny Aya path did not preserve the reference digit splitting. Prefix-isolation and word-boundary/join controls distinguished the tokenizer behavior from an accidental interaction with surrounding text. In the selected parity probe, 5 of 150 contexts differed and only 580 of 600 complete candidate inputs matched; the patched path matched 600 of 600. These are scoped probe counts, not population prevalence. Expanded checking then found a join-control boundary case; the final correction matched 900 contexts and 3,600 candidate sequences, preserving the earlier patch as a distinct identity.

This is a token-identity issue: it changes what reaches the model before likelihood scoring starts. It should not be collapsed into the later Python decoding failure or used to label every historical result invalid. The pinned tokenizer investigation retains the controls and affected artifacts.

Python detokenization: a truncated round trip

The second failure was in the Python wrapper's detokenizer. A fixed 32-byte buffer was too small for some Bangla token pieces. The native API signaled the required larger buffer with a negative return value, but the wrapper did not correctly resize and retry. A reproduced prompt shrank from 899 to 851 characters on the broken round trip. The failed pilot stopped after completing 24 of 40 candidates; that is a progress count, not 24 affected cases. These are diagnostic and failed-pilot observations, not an estimate of population prevalence or evidence that every completed baseline contained that corruption.

The repair respects the native API contract: allocate the requested capacity, retry, and fail explicitly if decoding still fails. This was a local reproduction, not a claim of novel discovery: an earlier upstream report and proposed fix already existed (issue 2362 and PR 2363). My later PR 2388 remains a separate, overlapping proposal; coordination is pending. The recorded buffer investigation preserves the local reproducer and repair.

Byte-exact round-trip checks matter here. A visually plausible prompt, a successful HTTP response and a finite likelihood are insufficient evidence that the text survived the path unchanged.

Prefix reuse: request history changed likelihoods

The third failure emerged later, after strings and token counts were already matching. Echoed likelihoods depended on reused-prefix execution history. In the failed A3 validation, independent cold forwards disagreed with supplied-prompt log probabilities by as much as 0.18238 nats; tested chat continuation differences reached 0.02854 nats. A fresh-server request permutation also changed likelihoods. That made request order part of the measurement.

This dependence was observed in the llama-cpp-python server path used for scoring; it was not reproduced against native llama-server, so it is routed to the wrapper integration first rather than claimed as a core llama.cpp defect.

Instead of relaxing the numerical tolerance, I preserved the failed output and changed the diagnostic. Seven same-order wrapper completions were checked against an independent full-vocabulary float64 log-softmax of the raw logits. They matched the preserved HTTP arrays within 9.3e-7 nats. This bounded result helped separate execution-history effects from a parser or echo-alignment error. It was not a complete correctness gate.

The patched llama-cpp-python scoring wrapper subsequently forces cold prefill for echoed likelihood requests and bypasses cache restoration for that branch. It does not establish that caching is generally unsafe or that all nonscoring generation paths have been validated. A separate exact top-one ranking optimization removed a CPU bottleneck while preserving the tested ranking and tie behavior; its microbenchmark is not model throughput.

These failures have different exposure histories. A tokenizer discrepancy changes input IDs; a failed detokenization round trip corrupts the text interface; prefix reuse changes numerical execution for otherwise matching requests. The preserved evidence distinguishes diagnostic reproducers, failed pilots, historical completed runs and the fresh validated identities. That separation is necessary before claiming that a particular defect caused an accuracy change.

4. Validate probabilities before collecting another accuracy table

The final A3 runtime gate covered 192 candidates in 48 contexts per precision arm, followed by 36 candidates in a separate reversed-order check. The contexts covered all three languages, scoring variants and rotations, with longest-input and continuation cases. The independent reference used raw logits, rather than trusting the same HTTP response that the client parsed.

Validation required exact token IDs, one automatic BOS, matching echo strings and counts, and the expected single generated suffix. Numerical tolerance never excused token misalignment. Both the 0.0002-nat per-token limit and the summed continuation limit, max(0.001, continuation_tokens × 0.0002) nats, applied. The parser had its own 1e-8 agreement check against the supplied echo slice. Both arms passed, with no correctness failures or winner inversions in that coverage.

The later T1 baseline had its own, smaller T1-specific gate: 12 fixed contexts and all four candidates per arm (48 candidates), covering indices 0, 100, 299 and the longest input in each language, plus the same 48 candidates in reversed request order. All candidate continuations were independently checked to be single model tokens. These are bounded correctness checks, not a proof about arbitrary future prompts or implementations.

The evaluation path stayed on the supported upstream configurable task and likelihood scorer. I did not introduce a custom scoring adapter to make the probabilities agree. Runtime sources, native library, tokenizer, model bytes, commands and gate artifacts were hash-pinned. Resumption revalidated saved shards against the same identity rather than silently mixing environments.

5. Position sensitivity survives the runtime control

A2 had already shown substantial option-position sensitivity. Its historical runtime identity remains attached to those observations; the cache finding is an additional possible confound, not a retrospective explanation of every A2 flip.

A3 repeated the format question under the cold runtime. It used 100 questions per language, four cyclic option orders, and three variants: plain prompt with letter completion, tokenizer chat template with letter completion, and plain prompt with full option-text completion. Rotations and variants are repeated measurements of the same questions, not extra independent samples.

Under plain-letter scoring, all-four content consistency was 67%/65% in English, 43%/44% in Bangla and 52%/46% in Hindi for BF16/Q4. Consistency means selecting the same original answer content in all four orders, whether correct or wrong. Position sensitivity therefore remained in both precisions after the runtime control. It cannot be described as a quantization-only failure.

The same cold-runtime A3 plain-letter cells show the displayed-letter imbalance:

Each cell contains 400 predictions across four rotations, with exactly 100 gold answers in each displayed position. These are A3 rotation histograms; T1's original-order gold distribution is different. The counts describe position sensitivity, not an internal defect or a validated correction.

The formats also changed measured performance. A3's complete results show exploratory chat-letter Q4−BF16 rotation-mean accuracy differences of −3.50 percentage points in English, with a 95% interval of [−5.95, −1.25], and −3.00 points in Bangla [−5.93, −0.25]. Those intervals exclude zero. All raw English-relative compression interaction intervals included zero, however. The evidence does not establish a Bangla-specific mechanism, and the many exploratory endpoints were not multiplicity adjusted.

Full option-text completion performed substantially worse here. Dividing its summed likelihood by displayed-choice Unicode character count changed results again. That normalization excludes the added ASCII-space delimiter; it is neither token nor byte normalization. For letter choices, normalization changes nothing because every displayed choice has one character. None of these diagnostics justified choosing an endpoint after seeing which one made compression look best.

The accepted downstream contract remained plain-letter summed continuation likelihood: one ASCII-space delimiter, no appended EOS, one automatic BOS, and the first displayed maximum on an exact tie. The next requirement was fresh full development comparators under that contract, not relabeling the 100-question rotation diagnostic as a 300-question baseline.

6. Fresh T1: small net differences, appreciable churn

Both fresh arms completed 180 five-question shards: 900 original-order questions and 3,600 candidate scores per arm. Every shard passed complete-membership, request, target, finite-score and provenance validation. The conditional overlap with A3 covered 300 original-order requests per arm; all 2,400 candidate comparisons across both arms matched without failures or winner inversions. Those rows were freshly collected, not copied from A3.

The intervals use 10,000 jointly resampled source-cluster draws, with 250 source clusters and seed 42. The same source draw is used across languages and paired arms. Questions sharing a source are not treated as independent, and the bootstrap denominator follows the sampled question count. The English-relative endpoint tests a difference in compression deltas, rather than confusing lower absolute Bangla or Hindi accuracy with extra compression damage.

All three fresh accuracy-difference intervals and both nontrivial English-relative intervals include zero. That is uncertainty, not equivalence or proof of no harm. The BF16 GGUF control already scores lower in Bangla and Hindi than English. This is a pre-quantization accuracy gap in this evaluation, not a comparison against a separate “base” model or evidence identifying its cause.

Net accuracy also hides changed decisions:

Bangla churn is about 1.57 times English churn here, not doubled.

A small net delta can coexist with both regressions and improvements, and two wrong answers can still differ.

These counts motivate inspecting decisions; they do not by themselves identify a language-specific internal defect.

Original-order T1, 300 questions per language; BF16 GGUF control, not the separate vLLM reference. Intervals jointly resample 250 source clusters.

7. Historical results are preserved, not silently repaired

The fresh run was joined to historical A1 by exact source/question identity, targets and request/candidate bytes, with matching model artifacts. Within-arm historical-to-fresh changed predictions were 0/1/0 for BF16 and 7/12/10 for Q4 in English/Bangla/Hindi. Fresh Q4 correct counts rose from 226/189/176 to 229/190/180; BF16 moved from 230/188/184 to 230/187/184.

The score changes were larger than the earlier selected bridge might suggest. Across 1,200 paired candidate scores per language, maximum absolute Q4 changes were 0.89890 nats in English, 1.33790 in Bangla and 0.79041 in Hindi. The earlier 15-context bridge maximum was 0.42425 nats. Its smaller coverage never supplied a uniform bound on future score changes or accuracy.

These are combined observed runtime/settings transitions. They do not isolate a patch's causal effect. Keeping old results under their original identities, and publishing the score and margin distributions beside the fresh runs, is more informative than replacing the historical table with a corrected-looking number.

Historical and fresh identities remain distinct. These observed changes combine recorded runtime/settings differences; they are not isolated patch effects or universal error bounds.

8. Margins suggest a next analysis, not an explanation

The fresh BF16 top-two summed-score margin provides an exploratory view of the flips. English had 38 questions below one nat, with 15 flips; Bangla had 82, with 28 flips; Hindi had 83, with 29 flips. Above or equal to one nat, flips still occurred: 6/262, 5/218 and 5/217 respectively.

Lower-margin cases had more flips descriptively. The binning existed before this collection, but these counts have no source-cluster intervals or language-adjusted causal analysis. They do not show that margin explains away language, localize a damaged tensor, or select a precision repair. They establish a question for later controlled work.

9. What this checkpoint supports

The useful deliverable is a reproducible measurement contract and a fresh paired comparator, not a claim that multilingual compression is solved. The T1 evidence package preserves the complete report, raw shards, identities and validation evidence. Its report SHA-256 is c1ffee6b6030b1d5beb6c6afe3d855c1e1703df3e2fd6329270b06d55a1941a1. Independent internal checks reconstructed all 1,800 rows and recomputed the fresh source-bootstrap endpoints. That is an audit of this evidence, not external peer review.

Future work still needs human Bangla labels before the gated sweep, real-build and matched-byte controls before allocation claims, and a separate generation protocol. The recorded EOS 6 versus older EOS 3 discrepancy remains unresolved for generation; no-EOS likelihood candidates do not ratify a generation setting. Device latency, memory and energy require device measurements. The protected holdout was not used for selection or scoring.

For engineers building the same kind of evaluation, the transferable practice is concrete: verify bytes and token boundaries, compare parsed probabilities to independent logits, perturb request order, retain failures, and create new identities when the measurement path changes. Only then does the compression delta have a sufficiently clear meaning to guide the next experiment.

Sources and reproducibility: runtime validation and preserved failed probes, A2 historical rotations, A3 cold-runtime format results, accepted scoring contract, and fresh T1 report and reproducible archives. Dataset: the Belebele dataset card, revision 7899cdfa4e1e0d733fd77c848e2c273cb1d32be2, CC-BY-SA 4.0.