⚡Megatron LLM Hub
←返回搜索
Twitter

单张 RTX 5090 跑 Qwen3 125B 达 390 tps 的本地推理配置与引擎对比

本地推理llm硬件
RT @TeksEdge: A single RTX 5090 reportedly hit 390 tps running Qwen3.8-Flash-Next on a brand-new inference engine. Peak? 452 tps Here is the setup 🧠 Qwen3.8-Flash-Next 🔥 125B total / 6B active 🎮 NVIDIA RTX 5090 32GB 💾 128GB DDR5 system RAM 🪟 Windows 11 📚 256K configured context ⚙️ INT8 KV cache Results (decode) 🚀 Infernix + DFlash2 + Q2 → 390 tps ⚡ Strata + MTP + Q2 → 348 tps ⚡ Infernix + native MTP → 304 tps 🟢 NVFP4 configuration → 209 tps 🔵 Qwen FP8 configuration → 60 tps That is a 6.5× difference between the fastest and slowest reported configurations on the same GPU! DAMN! How? Brand new Infernix! 🔧 Patched Infernix inference engine 🧠 GSQ-RCO Q2_0 quantization ⚡ DFlash2 speculative decoding 💾 INT8 KV caching However, reported perplexity FP8 → 3.487 Q2 → 3.930 That's ~12.7% higher perplexity. The developer reports relatively small coding degradation but larger losses for prose and Chinese. Link in ALT.