I tried to fire my LLM API. Open models passed one task and failed the other.
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle What I Built Open-Switch answers one question: can an open model replace your closed LLM API on YOUR data? It runs your
This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle
What I Built
Open-Switch answers one question: can an open model replace your closed LLM API on YOUR data?
It runs your workflow through open-weight models, scores them with bootstrap 95% CIs, and gives a verdict: Safe Switch or No Safe Switch. It also tells you if a LoRA fine-tune pays for itself.
Who pays: B2B SaaS teams cutting LLM bills, and consultants selling migration audits. Hypothesis: $49/mo self-serve, $199/mo agency.
Demo
Runs fully local (no hosted link, by design: your data never leaves your machine):
git clone https://github.com/SynthReaper/ghw-hacktoberfest26.git
cd ghw-hacktoberfest26/switchable && pip install -e .
openswitch run --models gemma-2b-local,gemma-9b-local --task specs/task.yaml
Code
MLH x Hacktoberfest Challenge Projects (2026)
Workspace containing hackathon projects for DEV Hacktoberfest Challenges.
Projects
-
paper-trail/: Paper Trail â Offline Local Walk Generator & Analog Field Card (DEV Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass). -
switchable/: Open-Switch â Autonomous evaluation and migration engine to verify whether open-weight models (Gemma 2, Llama 3, Qwen 2.5) can safely replace expensive closed LLMs with honest statistics (10k bootstrap 95% CIs), default-deny provenance guard, calibrated pairwise judging, and LoRA fine-tuning break-even economics (DEV Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle).
The project lives in the switchable/ folder. Key files:
-
provenance.py: default-deny guard against training on closed-API outputs -
evaluator.py: AB/BA position-bias defense -
stats.py: 10k-resample bootstrap CIs
How I Built It
- Models: Gemma 2 (2B/9B), Llama 3.1/3.2, Qwen 2.5 7B via Ollama
- Judge: every pair is scored in AB and BA order; disagreements are flagged (90% agreement with human labels, Îș = 0.85)
- Provenance guard: blocks fine-tuning on closed-API outputs, then regenerates references with an open teacher
- 109 tests, 10k-resample bootstrap CIs on every metric
| Task | Best open model | Score [95% CI] | Verdict |
|---|---|---|---|
| Support triage | Qwen 2.5 7B + LoRA | 90.0% [80, 98] | â Safe switch, 85% cheaper |
| Banking intent | Gemma 2 2B | 41.2% [27.5, 54.9] | â Not safe |
The tool says "no" when the answer is no. That's the point.
Why Does Open Innovation Matter?
Closed APIs rent you a fine-tune. Open models give you a LoRA adapter you own forever. A $1.03 fine-tune broke even at ~10k requests/month, while 3-shot prompting quadrupled per-request cost.
My Agent Session
Built with Antigravity under a strict AGENTS.md: pre-flight gates, TDD, and a no-fabricated-metrics rule.
Prize Categories
- Best Use of Gemma: 2B runs locally; 9B is the open teacher
- Best Use of Tinker: LoRA training in 85s for $1.03
Originally published by Dev.to AI. Aggregated on AIWithGhost for educational purposes â full credit and traffic to the original publisher.