Ripple-65B Preview Benchmark: Gauntlet, Forge 2.0, LFM3-1.7B

Ricursive's Ripple-65B Preview scored 0.028 out of 1 on our Gauntlet sb-7.2 agentic coding benchmark, where DeepSeek V4.1 Flash 1017 scored 0.6957 in October. One wiring line made every /api request in its app fail with a 500, and after seeing that on three HTTP checks it moved its tests in-process, around the server our scorer talks to. On a Mac it runs in llama.cpp but not yet in mlx-lm, the standard library for MLX (the machine-learning framework Apple built for its chips): our Q4_K_M quant wrote 13.5 tokens a second after a 32k prompt on an M3 Ultra that was also re-grading a run, and its measured 49.8 GB peak at 128k fits under the 55.66 GB GPU limit we measured on a 64 GB Mac.

Ricursive Intelligence released the 65B dense model on 4 November under Apache-2.0, post-trained with "a focus on agentic coding". Every benchmark number I found for it on 9 November came from Ricursive or a company that hosts it, so we ran it ourselves. On Forge 2.0, where an agent upgrades a Jira app, its two builds scored 0.0858 and 0.0770. The unchanged starter app scores 0.0617. Liquid AI's LFM3-1.7B scored 0.0055 on the Gauntlet and stopped itself, sure it had "~53 tokens left". Its Forge 2.0 builds changed no file.

I don't read 0.028 as "Ripple can't code". Its backend loaded all 12,289 payments when its own tests called it directly. It never went back to the HTTP entry point our scorer uses. In one run that looks like a habit, and habits are what an agentic benchmark is built to catch. Getting that run took four attempts: a provider's errors ended two, and a setup mistake of ours one.

Ripple-65B Preview benchmark: Gauntlet sb-7.2 and Forge 2.0 results

Both benchmarks run in our agent harness, the desktop coding app we build. Gauntlet sb-7.2, the scorer id of Gauntlet 7.2, gives the agent 150 calls to build a payments console: two Python services and a 3D view. The services sync from a vendor, a stand-in API that is down for the first seconds of a graded boot. Forge 2.0 gives it 300 calls to upgrade a Jira app on Forge, Atlassian's app platform, to version 2. That adds, among other things, a live migration of the app's stored ledger, a UI Kit admin page, signed CI (continuous integration) events and a JQL (Jira Query Language) function. Scores run from 0 to 1.

A call is one model request, which can ask for several tool calls or none.

Every scored run used OpenRouter with one pinned host: PrimeIntellect (bf16, the 16-bit weights) for Ripple and Liquid (fp8) for LFM3. Only attempt 2 below, unscored, used a second host. We asked for medium reasoning effort. Each model got one scored Gauntlet run and two Forge 2.0 builds on 9 and 10 November.

Ricursive's card compares Ripple's base model with DeepSeek V3.1 and V4 Flash. So I compare it with DeepSeek V4.1 Flash 1017, the nearest model on our Gauntlet board, and with Particle-1, Particle AI's coding model post-trained on DeepSeek V4.1 Flash. The Forge 2.0 board says to compare models only within one comparison group, builds from "the same verified task, engine and captured comparison cohort". Ripple and LFM3 share one. Particle-1 and DeepSeek sit in others.

Ripple's Forge build 2 took 129 minutes against 67 for build 1. Two of its requests hung, one for 20 minutes from 01:43 and one for 5, until PrimeIntellect answered each with a 502 error. Five 429 rate-limit replies followed by 02:10. The two requests that never completed cost nothing, and they're why the table shows 302 calls.

The Gauntlet runs weren't identical. Ripple, LFM3 and Particle-1 got the wrap-up nudge, a message from our harness when 30 calls remain. DeepSeek's run on 18 October didn't. Particle AI's docs map our medium effort to High. DeepSeek lists no medium level and Ripple's chat template no effort levels, so I can't tell what "medium" meant for either. Ripple used 360 reasoning tokens per call, DeepSeek 730.

Ripple's checks earned 0.1295. The Gauntlet scorer multiplies that by 0.6 for each of three critical failures, 0.216 in all, which gives 0.028. None of the 12,289 expected payments arrived through the API, the approval flow couldn't complete in the browser, and the hard-kill durability check had no draft to follow, because creating one returned a 500 too.

What we found

1. Ripple's wiring made every /api request a 500, and its tests went around it.
2. Its vendor-error branch calls a method that doesn't exist.
3. With 30 calls left it spent 25 on SQL inserts, where an unscored attempt wrote an 814-line 3D view.
4. Build 1's Forge admin page never rendered, and build 2's rendered but refused our admin.
5. LFM3-1.7B read its call budget as tokens and wrote nothing on Forge 2.0.
6. llama.cpp runs both models on a Mac, and mlx-lm 0.32.2 runs neither.

Where Ripple-65B Preview went wrong on the Gauntlet

Its server answered every /api request with a 500, and its tests went around the server

At its call 47 (22:41:55 on 9 November), Ripple wired up its HTTP server like this (app/ledgerd.py:1557, and the same in its notifier service):

Handler.application = application.handle

Its transport code then calls self.application.handle(...) on that value (app/transport.py:106). A bound method has no .handle, so every /api/* request raised an AttributeError and came back as a 500.

Its own HTTP checks at calls 63, 77 and 83 all got this:

{"error": {"code": "internal_error", "message": "Unhandled error: AttributeError"}}

At 22:47:54 it changed method: "I need a better debugging loop: make a small test harness that imports the app and calls handle() directly ... it's much faster than booting servers." From then on its tests ran in-process. At 22:51:32 one printed payments 12289 / events 12289, on a path that skips the broken line.

Our scorer only talks to the app over HTTP. Its API checks got the same AttributeError. The sync check reads "Eventual evidenced total 0 ... initial expected total 12289". I think the speed move was sensible. It just never went back. Its next HTTP check was its last call, and it got the same 500.

Particle-1 passed the application object itself, serve(args.port, app, Path(__file__).resolve().parent.parent / 'web'). It scored 1 on the sync check (12,291 payments), as did DeepSeek V4.1 Flash 1017. Ricursive's card says the Preview "has gone through relatively light posttraining". I'd guess heavier agentic training teaches the habit of re-running the entry point after a wiring change.

An error branch that calls a method nobody wrote

At call 30, two and a half minutes after quoting the vendor-down rule in its reasoning at 22:34:28, Ripple wrote the branch for it:

            except VendorError as error:
                failures += 1
                self.store.set_sync_state('last_vendor_error', json.dumps({'code': error.code}))
                time.sleep(min(5, 2 + failures))

Its Store class has set_last_sync and sync_value, but no set_sync_state. Its test boots all had the vendor up, so the branch first ran in our scorer. Three scorer boots logged VendorError('unreachable', 'collection walk failed') and then AttributeError: 'Store' object has no attribute 'set_sync_state'. That killed the sync thread. The vendor-down check scored 0.5, because the app bound its port but the sync never came back.

I think it made the name up from the family it had just written (set_last_sync, sync_value, a sync_state table). Particle-1 ran the test that catches this: at its call 109 it booted against a dead vendor, --vendor http://127.0.0.1:9999, and curled its own port. It read ledger:200 back and scored 1 on the vendor-down row. Ripple's own first attempt had the safe version, except Exception as error: sys.stderr.write(...).

Thirty calls left: two plans from the same model

In the scored run (attempt 4) the wrap-up nudge came at call 121, and Ripple replied: "3. Write the frontend (this is scored ~half the points!). I must not leave the frontend unimplemented."

It never got to the front end. It spent 15 of its last 30 calls on the placeholders of one INSERT INTO drafts statement, and 10 more on inserts into a log table. At call 149 it gave up: "The frontend doesn't exist (starter only, draws nothing) — that's a huge loss of points, but I have no budget to build it." The four web files are byte-identical to the starter.

In unscored attempt 1, it reached the same point at 20:34:54: "I'm at 30 model calls left ... the frontend ... is essentially unimplemented! The 3D field is half the score." That time it switched. It wrote a 526-line web/app.js, then ten versions of web/viz.js, ending at 814 lines of raw WebGL2. It ends with window.vs7dbg = null;, which blanks the hook our scorer reads the 3D scene from. The host failed on the next call, so we'll never know its score.

I think the difference was what was half-done when the nudge came: a broken insert in attempt 4, passing backend tests in attempt 1. Particle-1 wrote its four web files at calls 32 to 36.

How its 150 calls went

Of Ripple's 150 tool calls, 39 read files, 30 ran the app or its tests, 39 patched code by script and 17 edited it directly. Seven called a tool named bash, which we don't have (ours is shell), so an alias would have saved 4.7% of its budget. At call 73 it wrote "I keep making the same mistake typing "bash". Let me use shell", then typed bash three more times.

A typo from call 27 survived two patch scripts. Its page cache stores the key 'etrag' and its fetch code reads 'etag', so no re-sync sent the stored ETag back. DeepSeek V4.1 Flash 1017 sent it on 1,353 of 1,353, and Ripple on 0 of 4,933.

Its final summary said "Ledger APIs intermittently raise internal_error (AttributeError)", but I saw it on every HTTP check, and the cause it named was a different line.

Ripple-65B Preview on Forge 2.0: the admin page that never rendered

Ripple's Forge 2.0 builds beat the unchanged starter, but not by much. Both rewrote the manifest and all nine starter files, added 16 and 8 new ones and built clean. Their checks earned 0.3432 and 0.308. The starter, which LFM3 handed back, earns 0.2468. The scorer then multiplies by a reliability factor. Each of its 19 groups of checks takes off 10% when its worst check scores zero, and less for a partial score. Together they "never take the score below 0.25 of what the tests earned". Ripple's 14 and 15 zeros, plus one and two partial scores, gave factors of 0.2271 and 0.1976, so the scorer used the 0.25 floor. The same floor turns the starter's 0.2468 into 0.0617. Neither build moved any of the 59 rows the old app had stored into its new ledger.

The admin page uses UI Kit, Forge's component library. At call 190 of build 1 (00:38:18 on 10 November), Ripple printed the type index of @forge/react, with the answer on screen:

export { ForgeReconciler as default } from './reconciler';

The page it wrote never calls ForgeReconciler.render, ends with export default App; and imports Divider, Flex and TextField, three components @forge/react 12.3.0 doesn't export. It first rendered the page a few calls from the end, and our Forge emulator answered (nothing rendered). At 01:04:05 it guessed at the cause itself, "a missing component import like Divider/TextField". It then spent its next calls reading our emulator's loader and patching the export instead. The second render, at call 296, came back empty too. Its handoff note blamed "the emulator's ... UI Kit loader".

Our scorer's admin found none of the page's 8 controls. It couldn't rotate the CI secret, so all three validly signed CI events got a 401. Particle-1's page, in another comparison group, ends with ForgeReconciler.render(<AdminPage />); and booted on the same emulator version. Ripple's build 2, starting from the same starter, wrote its page with ForgeReconciler.render, and that page rendered. Its save and rotate buttons still changed nothing, because the app told our admin "You need the administrator permission". Its final message had flagged that check as needing "one more pass".

Build 1 recorded Jira's events well, with 170 of 172 ledger rows exact. Build 2 got 0 of 170, and I haven't traced why.

LFM3-1.7B benchmarks: a 1.7B model as a coding agent

Liquid AI announced the 1.73-billion-parameter LFM3-1.7B on 27 October, and it's been on OpenRouter since 6 November. Its card says: "It is not recommended for agentic coding or complex multi-step reasoning due to its small size." It was the most popular new small model I found on 9 November, with 1,569 likes and 283,113 downloads of Liquid's GGUF (the file format llama.cpp loads).

It counted calls as tokens, and stopped with 30 left

Between calls 13 and 41 it searched our harness's hidden runtime folder for its ports: "Let me check the telemetry.jsonl and sessions.db for config (ledger port, notifier port)." The starter already parses the ports from its arguments. The detour cost 11 calls and about 53 seconds. That folder being readable is our mistake. Our harness sends the model there for spilled output, so closing it means moving those files first.

I think it took the vendor on port 8850 for its own half-built server: "So the server on 8850 is a stub ... We need to implement our own services". DeepSeek V4.1 Flash 1017 treated that port as the vendor and logged "192/192 pages served".

Then it changed three files. The app never booted, because app/__init__.py imports a NotifierService that app/notifierd.py never defines. Its ledgerd.py never calls the vendor. It generates "12,288 synthetic rows" with amt = {random.randint(10, 1000000)}, but random is never imported, and the braces make each amount a Python set. It tried to start, import or compile that service 22 times, and twelve tries failed on our workspace path before Python even ran.

At call 120, its last, it wrote its final message and stopped:

**Budget used:** ~107–110 of 150 model-call tokens (≈73% consumed).
...
~53 tokens left (well under the 150 limit).

The same message claims app/transport.py and the four web files as its work, though it never edited them. Under "Still missing" it listed nine items, from the notifier service and the web API to the 3D view and the approval workflow, which is the whole product. Our scorer's boot crashed on the import.

Part of that confusion is ours: our harness shows the call count only past half the budget, one off against our logs. The prompt LFM3 had just read said 119/150 used. Its reasoning said "99/150 used, 51 left".

Two Forge 2.0 builds that wrote nothing, then claimed they'd pass

On Forge 2.0, LFM3 read the contract, the manifest and all nine source files, ran the linter and the build, and wrote nothing. Build 1 used 38 of 300 calls in 65 seconds. Build 2 used 34 in 73 seconds. In 49 and 44 tool calls, neither called write or edit.

Build 1 saw the gap at 21:43:41: "We need to add scope-ledger entity with attributes and indexes. ... Let me grep quickly". The grep failed on our workspace path. Eleven seconds later it decided "Probably fine", and its final message said "the codebase appears already complete for v2".

Build 2 knew what was missing, which I think makes it the more useful failure. It listed the gaps in a table, then handed the table back as a plan with 266 calls unused: "Once those are patched, the app should satisfy the definition of done and score well above the 25% floor." Neither got far enough to see our call count, and build 2 guessed "≈40 calls used" at 34.

Both scored 0.0617, the unchanged starter's score. Our scorer's main check says "None of the v2 surfaces exists": no JQL function, no scope-ledger entity, no admin page, no web trigger.

What the cards claim, and what we saw

Ripple's card sets it against four mixture-of-experts models that switch on 23B to 55B weights per token: Nvidia's Nemotron 3 Ultra 550B, Inkling 975B, Meta Muse Spark Open 816B and Reflection Beam 501B. Ripple scores itself 80.5 on Terminal-Bench 2.1 and 76.5 on SWE-bench Verified (SWE stands for software engineering; the tests are real GitHub issues), against 80.1 and 81.6 for Reflection Beam, and many rival numbers are Ricursive's own runs, in italics. A card edit on 6 November (commit 30cb3b0e7d) raised Inkling from 2.0 to 39.6 and Muse Spark Open from 52.3 to 72.4 on SWE-bench Pro v2.0 Hard. I found no note on the card about why.

On our Gauntlet, Reflection Beam 1.0 scored 0.045 in a run that ended after 27 calls and 1 min 11 s on 19 October, without the wrap-up nudge. Muse Spark Open has only Forge 2.0 scores from us.

LFM3's card says the model "can power agentic applications that used to need 9B parameters". Its table backs that on tool-calling and web-agent rows, for example 71.39 against 62.83 for Qwen3.5-9B on the Berkeley Function Calling Leaderboard (BFCL) V4. It doesn't on coding-agent rows: 43.40 against 51.20 on SWE-bench Verified, and 12.92 against 17.98 on Terminal-Bench 2.1.

Where we went wrong: four attempts for one Ripple score

All of these are our mistakes. Items 6 to 8 happened inside scored runs, so they may have cost LFM3 or Ripple points.

1. We allowed a fallback host. In attempt 2 GMICloud served 33 of its 50 billed calls with no reasoning setting. Its local score of 0.0081 stays unpublished, because one review shouldn't mix two serving stacks.
2. Our harness ends a run on a tool call printed as text, with no nudge to retry. That cost attempt 2 its last 98 calls, and the whole October run of Liquid's previous 24B model.
3. Our retries give up too fast. We retry 3 times, about 0.9, 2.2 and 3.6 seconds apart, so attempt 1 ended 7.6 seconds after its first error. Attempt 3 died at 21:41, our probe got clean answers only from 21:49, and I think a backoff in minutes would have saved it.
4. We scored next to a CI runner. Its browser tests shared the Mac, and scoring took 1,389 seconds against 532 for Particle-1. A re-score still gave 0.028, and the wiring line fails whatever the load.
5. We launched the Forge pair on a failing probe. At 23:57:59 our probe had just seen 3 of 4 PrimeIntellect requests fail. It worked out, but the launch should have waited.
6. Our workspace path has spaces in it. 18 of LFM3's 142 Gauntlet tool calls failed on it unquoted, and 12 and 6 in its Forge builds.
7. We sent no sampling settings. LFM3's card recommends temperature=0.1, top_k=50, repetition_penalty=1.1 for agents. OpenRouter lists those as its defaults, but we can't see if they were applied. Ripple publishes none.
8. Our tool says lines, and the limit is bytes. Its description says output is cut at "2000 lines", but the cap that fired was "Output exceeded 50000 byte limit". LFM3 spent 11 calls cutting a 19-line, 572 KB file by lines.

Licences and prices: Apache-2.0 against a revenue limit

LFM3's licence sets the "Threshold" at "annual revenue of 10 million United States dollars ($10,000,000) or more", and Section 5(b) doesn't license commercial use by a Legal Entity above it. A "Legal Entity" takes in everything under common control, so your group's revenue counts.

I count at least $5.12 of cloud spend, since one of Ripple's bills is incomplete. Ripple's six runs cost $4.93 and LFM3's three $0.19.

Ripple-65B Preview and LFM3-1.7B on a Mac: why mlx-lm can't load them yet, and what can

Our desktop app loads models through mlx-lm, and on 9 November its latest release, 0.32.2, had no model file for either architecture. LFM3 support has sat in an open pull request, mlx-lm #2016, since 27 October. Liquid's MLX card points to mlx-vlm 0.7.11 or later instead. Ripple has no mlx-lm pull request at all, and a community 4-bit MLX build needs --trust-remote-code.

llama.cpp runs both, so I'd start there. It merged LFM3 on 27 October and Ripple on 5 November, first released in b12167 a day later. It was also ahead on multi-token prediction in our llama.cpp vs MLX test, though mlx-lm still ran faster there.

Liquid publishes a Q4_K_M GGUF of LFM3, and Ricursive none of Ripple. We made our own Q4_K_M (llama.cpp's 4-bit format) of both from the official weights with llama.cpp b12230, plus a Q8_0 (8-bit) of LFM3. We haven't published them. These are speed tests only. Every Ripple and LFM3 agent run in this article used OpenRouter. "Reading" is prompt speed and "writing" generation speed. llama-bench reads 512 tokens and writes 128. The prompt rows fill the context with real text through llama-server, then write 32 tokens. llama-bench's reading speed is for 512 new tokens at that depth, and a prompt row's is the average over the whole fill, so the two differ at the same 128k.

Ripple's Q4_K_M, a 38.8 GB file, ran on a Mac Studio (M3 Ultra, 96 GB). Most rows ran while our scorer re-graded a run on it, and the 128k prompt also overlapped a Forge build. The 131k llama-bench row ran later, with no benchmark on the Mac. The GPU sat at 0% as each step started.

LFM3's Q4_K_M (1.1 GB) and Q8_0 (1.8 GB) ran on a MacBook Pro (M3 Max, 64 GB) whose GPU was already 21 to 22% busy with other work.

On a 128 GB M4 Max that was 49 to 56% busy, I only trust peak memory: 2.58 and 3.88 GB at 32k and 128k for Q4_K_M, and 3.46 and 4.76 GB for Q8_0.

Ripple has to fit in the GPU's share of memory. We measured macOS's default cap at 55.66 GB on a 64 GB Mac and 115.45 GB on a 128 GB one, and sudo sysctl iogpu.wired_limit_mb=<MB> raises it. Our GB are billions of bytes, so a Mac sold as 64 GB has about 68.7 of them.

On a 64 GB Mac I'd keep Ripple at 128k or less. LFM3-1.7B fits any of these Macs: its full 262,144 tokens come to about 5.6 GB (Q4_K_M), from its measured growth of about 13,200 bytes per token. That's about a third of a 16 GB Mac's memory, though we haven't measured that Mac's GPU limit.

A 65B dense model is plausible on a Mac because only 16 of Ripple's 80 layers keep the whole context. The other 64 look back 1,024 tokens. By our arithmetic from its config, that's 64 KiB of KV (key-value) cache per token, the attention memory that grows with context, plus a fixed 256 MiB. Our two prompts agree: memory grew by 68,086 bytes per token between them, against the arithmetic's 65,536. The 256k column adds that growth to the measured 128k footprint. Nanbeige4.3-33B, the dense model in our previous weekly review, needed 212,992 bytes per token (our Nanbeige4.3-33B Mac test).

What I'd check before trusting a model like this as an agent

1. Test through the entry point your users hit.
2. Take the dependency down once.
3. Pin one host and probe it before a long run. If 3 of 4 probe requests fail, wait.
4. On OpenRouter, check a fallback host's endpoint entry for reasoning and tool_choice. GMICloud supports neither a reasoning setting nor a required tool_choice.
5. Give your agent a retry policy measured in minutes.
6. Read the card's scope line and send its sampling settings.

If you've run Ripple-65B Preview as an agent, did it test over HTTP? And where did a small model like LFM3-1.7B lose track first for you? The comments are open, and so is our Discord.

Originally published on leanzero.net. More Atlassian, Forge and local-AI write-ups at leanzero.net/blog, and if you're planning a migration or a Forge app, that's what we do: leanzero.net/services.