关系到可本地部署 27B 模型的实际规格、量化占用与推理速度预期。
llm本地部署推理
RT @teortaxesTex: This is becoming interesting
- let's say Qoduck is Qwen4-27B
- I'll accept that it's ≈35GB in Q8
- 3.8-Flash-next was arch preview for 4
- Qwen 3.x-27B in fp8/q8 is ≈29 GB
- Engram is also 8bit, so 6Gb, ratio checks out
- it goes at 480 tps
- it's an extreme yapper. Qwen's old 35B 3AB was also a yapper.
Now, DeepSeek's original Engram paper used a 27B 4AB MoE (their default test shape)… I'm not sure how Engram plays with dense models or if Alibaba would be willing to check…
in short, what I'm saying is that Qwen4-27B may be a sparse MoE, unlike all previous Qwens-27B. At the level of capability we're seeing, this would be kind of astonishing and unlocking a lot of use cases. But unsurprising (3.8-Flash-next is 125B 6AB + 51B n-gram embedding and 4B MTP)
I will assign this very little probability, this is more of a wish