本地推理llm硬件
RT @TeksEdge: A single RTX 5090 reportedly hit 390 tps running Qwen3.8-Flash-Next on a brand-new inference engine.
Peak? 452 tps
Here is the setup
🧠 Qwen3.8-Flash-Next
🔥 125B total / 6B active
🎮 NVIDIA RTX 5090 32GB
💾 128GB DDR5 system RAM
🪟 Windows 11
📚 256K configured context
⚙️ INT8 KV cache
Results (decode)
🚀 Infernix + DFlash2 + Q2 → 390 tps
⚡ Strata + MTP + Q2 → 348 tps
⚡ Infernix + native MTP → 304 tps
🟢 NVFP4 configuration → 209 tps
🔵 Qwen FP8 configuration → 60 tps
That is a 6.5× difference between the fastest and slowest reported configurations on the same GPU!
DAMN! How? Brand new Infernix!
🔧 Patched Infernix inference engine
🧠 GSQ-RCO Q2_0 quantization
⚡ DFlash2 speculative decoding
💾 INT8 KV caching
However, reported perplexity
FP8 → 3.487
Q2 → 3.930
That's ~12.7% higher perplexity. The developer reports relatively small coding degradation but larger losses for prose and Chinese.
Link in ALT.