28 TPS on Qwen2.5-7B across two separate cloud regions over public WAN using speculative decoding + CUDA Graphs [P]
been building ShardFlow for the past few months, a distributed LLM inference framework that splits any HuggingFace transformer across N GPU machines and uses neural speculative decoding to deal with WAN latency. the se
📄
This source provides headlines only. Use the button below to read the complete article on the original site.
📰 Read the original article on r/MachineLearning
Originally published by r/MachineLearning. Aggregated on AIWithGhost for educational purposes — full credit and traffic to the original publisher.