Same A40, same Spark file: inside cortiq’s new CUDA path

A longer prompt arrives. The model still fits, but the wait grows—and every new token has more history to look through.

Community developer infosave (Oleg Kirichenko) has been working on that problem with cortiq, a Rust inference engine that runs Spark-X2.6-4B from a single CMF file. The latest step is a native CUDA backend in cortiq 0.8.67. Spark is the first model it runs.

On the author's NVIDIA A40, the same 4-bit file after a 4,000-token prompt generated 111.5 tokens/s on CUDA versus 80.2 on Vulkan. First-token time was 0.66 versus 1.06 seconds. These are the author's tests, not a new SparkLLM benchmark run.

A longer answer changes the comparison

At a 64,000-token prompt, CUDA with the Turbo far field reported 100.7 tokens/s over a 500-token decode, compared with 85.9 over Vulkan. First-token time was 42.3 versus 60.3 seconds.

The memory number needs its full context: 3.9GB while decoding, 8.2GB peak VRAM during the CUDA run. A 3.9GB card is not established as sufficient. CUDA exact attention was quicker to the first token in this test (29.8 seconds), but its 500-token decode was slower (53.4 tokens/s). Which path helps depends on how much you ask the model to generate.

The A40 tests used Ubuntu 24.04, driver 595.91 and CUDA 12.8 libraries. The short benchmark used 60 output tokens and medians of three runs; long decode and decode-memory figures came from one 500-token run per path.

What changed inside the engine

The file is unchanged across backends. CUDA uses cuBLAS for prompt processing and dedicated kernels for decoding. Above 8,192 tokens, the Turbo path retains selected exact rows and summarizes more distant context into a fixed-size far field. That distant context is approximate.

CUDA reads the whole prompt with exact attention before sealing the far field. This explains why peak memory still grows even when decode memory stays flat. A new turn after sealing rereads the conversation; this backend does not support speculative decoding.

The earlier experiment that led here

The following charts come from the linked earlier Turbo model card, on different hardware. On an RTX 2000 Ada 16GB using Vulkan, the author reported 56.2 versus 18.5 tokens/s at 64k input, Turbo versus standard, with 4.2 versus 7.9GB peak VRAM. Shorter prompts did not always get a faster first token.

This earlier diagram describes the Vulkan/Metal path. Its prompt-processing strategy differs from CUDA's full-prompt exact prefill; it should not be read as an A40 implementation diagram.

Keep the quality evidence next to the speed

The original Turbo discussion disclosed missed planted passwords: 6/8 at the reported text lengths. The newer CUDA card reports 8/8 at each of 60k, 120k and 240k characters with its stated setup. These are small, version-specific probes, not a general accuracy score or a controlled before/after quality comparison.

The earlier card also compares short-window WikiText-2 perplexity. The 4-bit cortiq file scored 32.95 versus 31.91 for the similarly sized MLX file; lower is better. A 512-token test uses exact attention and does not measure long-context far-field quality. Different file sizes and quantization schemes matter.

The wider picture: Mac, other engines and CPU

The author's broader comparisons show where the result changes. Prefill and decode can favor different engines, and the model files are not the same quantization or size. Read the hardware labels before comparing bars.

Mac memory accounting also needs care. Activity Monitor's footprint and the resident set count mapped file pages differently. These are process-memory measurements, not CUDA VRAM requirements.

On the tested CPU server, llama.cpp was faster in some conditions. Turbo runs as the standard file on CPU. The new CUDA result is a reason to try the appropriate GPU path, not a promise that cortiq wins on every device.

Try the author's CUDA build

Use cortiq 0.8.67 or later built with the CUDA feature, an NVIDIA driver and CUDA 12.x runtime libraries. The prebuilt releases do not include this backend; check cortiq gpu because missing CUDA libraries can cause a CPU fallback.

cargo install cortiq-cli --features cuda
hf download infosave/Spark-X2.6-4B-turbo-CUDA-cmf spark-x2.6-4b-q4tp-turbo.cmf --local-dir .
CMF_GPU=cuda CMF_MAX_SEQ=131072 cortiq run spark-x2.6-4b-q4tp-turbo.cmf --prompt-file report.txt --greedy --no-think

Try a document you know well. Record first-token time, peak memory, decode speed and whether the answer preserves the facts you care about.

Author's discussion · Pinned CUDA model card · Earlier Turbo card · CUDA guide · Spark-X2.6

All nine original charts are credited to infosave / cortiq and preserved within annotated layouts. Test dates are not stated in the source, so no collection date is substituted. The new discussion was published November 11, 2026 at 04:00:25 Beijing time (November 10, 20:00:25 UTC).