16B dense on 16GB GPU vs 32B dense on 2x 16GB GPU
I'm currently trying to plan a build to run big(-ish) LLMs locally, and was wondering the following: I'm able to run a 16B dense model at Q4 with reasonable context size on a single 16GB VRAM GPU (9070 XT). If I were to
π
This source provides headlines only. Use the button below to read the complete article on the original site.
π° Read the original article on r/LocalLLaMA
Originally published by r/LocalLLaMA. Aggregated on AIWithGhost for educational purposes β full credit and traffic to the original publisher.