The official RKLLM vision demo doesn't run on the transformers it targets — a five-version bisection
I'm converting Qwen2-VL-2B-Instruct to run on a Rockchip board with Rockchip's official rknn-llm toolkit. Step one is exporting the vision tower to ONNX with the repo's own multimodal demo script. It did not run. Not on
I'm converting Qwen2-VL-2B-Instruct to run on a Rockchip board with Rockchip's official rknn-llm toolkit. Step one is exporting the vision tower to ONNX with the repo's own multimodal demo script. It did not run. Not on my code — on the repo's own script, with the transformers generation the repo's newer scripts are written for.
Here is the failure trail, version by version:
| transformers | result |
|---|---|
| 5.8.0 |
AttributeError — the export script reads model.visual; in 5.x the attribute moved to model.model.visual
|
| 4.57.0 |
TypeError — VisionAttention unconditionally unpacks position_embeddings, which comes back as None under the script's calling convention |
| 4.54.x | still broken — the rotary_pos_emb fallback (deriving position_embeddings when it is None) was removed in this generation |
| 4.51.3 |
works — source check confirms it has both self.visual and the rotary_pos_emb fallback |
So the export script is written against a pre-4.54 contract, while sitting in the same repo as a quantization script that already uses the 5.x module layout (model.model.language_model). One repo, half new, half old — and nothing in the repo tells you which half wants which transformers. I ended up running two environments side by side: 4.51.3 (+ torch 2.6.0) for the vision export, 5.8.0 for the quantization embeds.
Two ground rules I kept while doing this:
-
No patching the upstream script, no monkeypatching the model. I did try a
.visualalias shim early on: it cleared the attribute error and then died on the attention contract anyway. Even if it had "worked", a shim that changes how embeddings flow risks silently wrong embeddings — which would poison every benchmark that follows. Version pinning only. - Verify against source, not vibes. The 4.51.3 answer came from reading the transformers source at each generation and checking both conditions, then confirming with the actual export.
Result: the vision tower exported cleanly — a 53 MB ONNX graph plus 2,661,085,184 bytes of weights, onnx.checker passing.
Next steps are the quantization, the RKLLM export, and then the part I actually care about: on-board numbers (vision encode per call, prefill/decode tokens/s) and a 24-question quality check including Chinese signage/OCR-style prompts. If the version archaeology above saves you an afternoon, the board numbers in part two might save you a purchase decision.
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.