为评估 AI 参与安全工程的实际边界提供分层判断依据
ai_agentllm议题
this is why i'm pessimistic(ik lots of pessimism last two days) about formal verification as a solution to securing the entire existing software stack, and much more optimistic about reducing complexity and attack surface first, then formally verifying the critical parts.
with llms, we can definitely formally verify a lot more software, and we absolutely should. but claiming we can verify the entire existing software stack for $ X sounds absurd to me, almost like throwing resources at a lost cause.
the problem is you have billion states in complex software with insane amount of interactions between components and humans and specifying what exactly "secure" means is way harder
look at this example of how things can go wrong. tldr is @AnthropicAI and @theoremlabs formally proved a asan based memory safe property for some gif encoding library's compiled fuzz target, for inputs under 64 MiB.
but the fuzz harness rejects images exceeding 10k total pixels, so the verified program never reaches the libraries large image states including >10k <64mib which they intended to prove and there is a 32bit integer overflow. (ik the input required to crash is way beyond 64mb limit and outside the theorem, but you get the gist of what can go wrong.)
the proof appears to be correct, claude proved exactly what it was asked to prove but it doesn't prove what they intended, all inputs below 64 mib to "library"
now imagine doing this for browsers, kernels, servers, and applications running on top of them. even when you prove something correct, how do you know your next model won't find a security critical case that your specification missed?
it still looks like the same cat and mouse game to me