r/LocalLLaMA

People are making single-slot, half height pcie v100 with nvlink in China

https://preview.redd.it/yu3jolmnm96h1.png?width=899&format=png&auto=webp&s=4a938ad5675015cc6b5e4bb9d…

Rick & Morty

nobody expected HF there submitted by /u/jacek2023 [link] [comments]…

PSA: Throttle GPU power limits, with minor performance deficits

I just feel i need to post this here again so more people see: Test around with throttling the power…

Apple announced new on device inference engine for Apple Silicon

This news seem to have flown under the radar. Apple announced CoreAI on WWDC which is basically a fu…

I put together a Rust-native, CPU-only implementation of LFM2.5-8B-A1B

This is still a work in progress, but since recording the video, I added callbacks for tool use, mor…

Semantic distance as routing layer: an on-device, serverless alternative to the central-index model

Premise: For ~30 years, discovery (of information or of people) has been mediated by a central index…

Jetson Orin NX Build for Hermes Agent + Benchmarking

I had a huge LLM server, and now I have a tiny one! I had a Jetson Orin NX gathering dust from a lon…

Still a VERY lightweight open web-search tool for smaller local LLMs - now with SearXNG support

Hey everyone, TinySearch v0.2.0 (first stable beta) is out. The first version used DuckDuckGo direct…

Gemma 4 31B's competence surprised me

I'm just getting started using local LLMs for code. I'm not interested vibe coding, but I am hoping …

Cheapest setup for >10 tok/sec for 120B dense LLM

Hi all, I'm trying to wrap my head around hardware variables when it comes to LLM, and I have anothe…

Have we reached the point where open-source LLMs are “just good enough”?

The question I’m asking myself is whether open-source LLMs are now “just good enough” to meet 95% of…

benchmark idea: political compass for finetuned/abliterated models

There are political compass benchmarks for cloud models, like this one:https://trackingai.org/politi…