Same quantized model, two different fates: one chip refused it, the other ran it in 1.39 ms

I took one quantized NanoDet model and offered it to two Qualcomm devices. One refused to load it at all. The other ran all 324 layers on the NPU in 1.39 ms. The model was identical. The quantization was identical. The difference was a float tail the quantizer couldn't touch — and whether the device was willing to look past it.

The model

NanoDet, quantized to w8a8 with 23 real images from COCO val2017 as calibration data. (24 were planned; one download failed, and I'd rather say so than pretend the set was clean.) Same graph, two devices: a Dragonwing RB3 Gen 2 (QCS6490) and a Galaxy S25.

What each device did with it

(The layer counts differ between precisions because a quantized graph carries extra quantize ops. Compare latencies, not layer counts, across precisions.)

Why the same graph has two fates

YOLOv11-N quantized cleanly on the RB3 in the earlier experiment — 341 layers, all NPU, 5.03 ms — because its export is raw convolution output. NanoDet's export carries its decode arithmetic inside the graph: the GFL head's distribution integral and the box-coordinate decode run in float, after the convolutions. The quantizer can't convert that tail, and the RB3's QNN path won't accept a graph whose I/O isn't fully quantized. Same quantized graph, same device family — one model's tail is clean, the other's isn't.

The S25 takes the same graph and runs it in 1.39 ms, because its QNN path accepts it. And note the S25's float path is 1.47 ms — on the flagship, quantization buys almost nothing, because the float path was already all-NPU. On the RB3, quantization is the difference between 90 ms and 6.47 ms, because the float path was mostly CPU.

Two ways to fix the RB3 rejection are on the table: compile the quantized model through TFLite instead (convolutions on the NPU, tail on the CPU), or split the export — backbone and head as pure convolutions through QNN, decode on the host. Both concede the same point: the decode tail belongs on the host, not in the graph.

The rule to take home

Quantization is a property of the graph and the device's I/O contract, not of the model alone. A model can be "quantized" and still be un-runnable on a given device, if its tail carries float arithmetic the device won't accept. When you report a quantized benchmark, name the path and the device — "w8a8" alone says nothing about whether it ran, let alone how fast.

Same story as the 27× gap in the earlier experiment, from the other side: there, the runtime path decided the speed. Here, the runtime path decides whether the model runs at all. Both times the evidence that mattered was the per-layer profile, not the headline number.

The model itself is now published as an AI Hub external recipe — https://huggingface.co/anguoyang/nanodet — with w8a8 in its supported precisions and the RB3 QNN rejection recorded in the recipe's own test report, so the next person hits the signpost instead of the wall.