r/LocalLLaMA

Been watching real adversarial input hit my detection API for six months. Here's what's actually landing.

Disclosure: I built Bordair, a prompt injection detection API. This post is about attack patterns we…

Meddies PII: An Open Multilingual De-identification Model for Clinical Text

A clinical AI model does not need to know who the patient is to reason clinically. It needs the sym…

Windows keeps crashing on rtx 3090

Recently bought used 3090. Under heavy stress tests and gaming it's fine. When I load any model, it'…

vllm-doctor — a CLI tool to diagnose and monitor vLLM inference servers

vllm-doctor reads metrics from a vLLM server's /metrics endpoint or a Prometheus instance and runs r…

mindlab-research/Macaron-V1-Preview-749B • Huggingface

https://huggingface.co/mindlab-research/Macaron-V1-Preview-749B https://preview.redd.it/g64u0fyts06h…

Weird to get near linear scaling by adding another GPU?

Single steam benchmarks (club-3090) model: qwen3.6-27b-autoround-int4 BEFORE: 1x3090 *Their default…

Gemma 4 12b QAT is a regression for my use case, despite all the hype.. Not my main Squeeze

I spent the last few days trying to get consistent tool calling out of the new Gemma 4 12b QAT model…

Thoughts on Gemma4 12b vs 26a4b, which one is better?

Not talking about 31b. In terms of creative tasks, writing, chatting, not necessarily coding but ca…

Kokoro - Local installation. Multilingual.

Hello, I hope everyone is doing well. I'd like to consult with someone who has experience using Koko…

QATs Q4_0 from Google have more precision than Q4_K_XL from Unsloth (at least some)

I wanted to try new QATs and opened two collections on HF (which HF found for me): https://huggingfa…

Gemma4_31b_fp8 keeping up with Sonnet_4.6_medium in my harness.

https://preview.redd.it/9t0qvx6k5z5h1.png?width=1400&format=png&auto=webp&s=88dd83cdd6aa484dcf102bf0…

Best Local TTS solution

So I have been testing a bunch of different solutions for local TTS - nothing so far comes close to …