r/LocalLLaMA

I just bought Asus Ascent : Nvidia GB10 (DGX) and It is slower than my Ryzen Ai Max

It is suppose to be 2-4x faster but i am only getting 6TK/s on Gemma4-31B . What am i doing wrong? …

I built a self-hosted open-source MCP server that gives any local LLM real financial data — SEC filings, 13F, insider & congressional trades, short data, FRED

One thing missing when running local models as agents: real, current data. So I built Equibles — a s…

Adding E4B audio encoder to larger models

I am curious if anyone here has tried doing this, I did a bit of digging and it seems like it would …

Qwen 3.6 27B: IQ3XXS KV Q8 vs Q4XL KV Q4 (262K context)

hey yall. So I have a 24GB gpu. What do you think is better? I am using unsloth quants. Both are UD …

Qwen3.6 9B will release around Google I/O?

I don't think alibaba officially stated about "no qwen3.6 smaller models", and according to the patt…

What's the best setup to link Obsidian with a local LLM?

My specs: MacBook Air M2 24 GB RAM I’m looking for the best way to connect all my Obsidian notes t…

Built a fully offline suitcase robot around a Jetson Orin NX SUPER 16GB. Gemma 4 E4B, ~200ms cached TTFT, 30+ sensors, no WiFi/BT/cellular. He has opinions.

Sparky runs entirely on the Jetson. Gemma 4 E4B at Q4_K_M via llama.cpp with q8_0 KV cache and flash…

Are the rich RAM /poor GPU people wrong here?

Hello Guys, I know everyone has his definition of local models, but for me i see 2 "reasonable" type…

GitHub - pwilkin/openmoss: OpenMOSS pure C++ pipeline based on GGML

I'm uploading a full GGML-based pipeline for OpenMOSS (https://huggingface.co/OpenMOSS-Team/MOSS-TTS…

[FOUNDING] SupraLabs - real open-source AI models for you!

https://preview.redd.it/k6lub2ypva1h1.png?width=1500&format=png&auto=webp&s=cd44452c86b5216fec17113a…

Gemma 4 + LiteRT-LM on mobile: much better memory/perf than my llama.cpp setup

Hi r/LocalLLaMA - I've been paying close attention to the edge AI ecosystem because it's an area whe…

Evaluated a RAG chatbot and the most expensive model was the worst performer. Notes on what actually moved the needle.

We had a customer support RAG bot. Standard setup: ChromaDB, system prompt, an LLM doing generation.…