对 LLM 自动挖洞能力的现实校准,避免高估本地模型产出。
llmai_agentpoc
there's still a considerable gap between open and frontier models, and an even bigger gap between frontier models alone and frontier models in the hands of skilled researchers.
a huge number of people threw tokens at the bug and produced mostly slop PoCs.
the benchmark saturation you see online often doesn't translate into frontier level offensive capability. sure, deepseek flash and kimi k3 can probably develop exploits for a good number of vulnerabilities, but the capability gap is still substantial. they mostly failed for this php bug. prove me wrong.
having a more capable model than your adversary is an advantage, and pairing it with a skilled researcher can make that advantage even greater for defenders. everyone involved in this research had access to the best cyber models.
i personally think we should preserve that capability gap rather than dismiss its defensive value without evidence.