⚡Megatron LLM Hub
←返回搜索
Website

Doyensec 对 Aikido 与 XBOW 两款 AI 渗透平台做横向对比

AI AppSec 工具采购时的横向参照,但由厂商赞助
工具ai_agent报告
Our friends at Doyensec ran a head-to-head comparison of Aikido and XBOW, two AI application security testing platforms. I think this is a great initiative. Even though the comparison was sponsored by Aikido, I trust that Doyensec ran it fairly. The hard part for buyers is that a pentest is basically a black box when you’re purchasing it. If you are narrowly looking at one vendor, you have no idea what you’re actually getting, and a strong sales team can close the deal regardless of how good the underlying product is. The only way to really know whether a vendor did a good job is to put it side by side with a competitor on the same target. But pentests are expensive, so almost nobody gets the chance to run that comparison. So a third party comparison like this is a great way to inform the buyer. Benchmark setup Doyensec picked two open-source projects to run head-to-head, Fider and Photoview . Each repo cost $4,000 to scan on both platforms: Aikido’s Standard Pentest and XBOW’s Plus tier come out to the same price. I decided to run the scan on Fider at the same version which Doyensec benchmarked. Unlike Aikido’s and XBOW’s fixed pricing, we calculate cost dynamically from estimated token and compute usage. Since the Fider repo is smaller, the estimate came out to $350: Hacktron scan wizard on the cost calculation step, showing 350 calculated credits for msrkp/fider v0.33.0 So I started a scan on Fider v0.33.0 at 18:14:59, right before leaving for the gym, and by the time I got back it had already finished, at 18:41:45. Here’s the finished scan: Completed Hacktron scan of Fider with 32 confirmed findings and 60 security agents, run from 18:14:59 to 18:41:45 There were 60 agents launched and 31 findings were confirmed. You can view the final scan . Triage I manually validated every finding against the source and against a live Fider v0.33.0 I deployed. 27 held up as true positives, 4 were false positives. Here is how that looks next to Doyensec’s numbers on the same target: Hacktron Aikido XBOW Cost $350 $4,000 Wall-clock 27 min ~8h40m ~1 week Reported 31 19 25 True positives 27 17 24 False positives 4 2 1 So on the same app, Hacktron found the most real bugs (27 vs 24 vs 17) at roughly a tenth of the cost and a fraction of the time: True positives False positives One annoying part with AI agents is that they overstate the severities. Hacktron reported a lot of high severity findings, which when I triaged, many dropped to medium (a CSP-blocked XSS is not a high severity issue). Aikido and XBOW did the same thing per Doyensec, just less aggressively. This is something we need to work on. Hacktron rated Reviewer (real) The scanner called 6 things critical and 14 high; but after grading against real impact, it came out to 3 critical, 3 high, 3 medium, and 18 that were really low or info (13 low, 5 info). Only 22% of its severities matched my review, vs 76% for Aikido and 71% for XBOW. So it’s great at finding bugs and bad at ranking them. Here’s the real severity mix once you grade the 27 true positives properly: Critical3 High3 Medium3 Low13 Info5 The bugs that actually matter Now the important question is, did the $350 pentest find all the important bugs? Doyensec didn’t provide all the bugs Aikido and XBOW found for Fider, but we can look at what the Fider maintainers recently patched and use it as ground truth. Here are the security fixes Fider shipped, and whether this one Hacktron run caught them: Bug Fider patched Commit / advisory Hacktron SSRF in webhook URLs f7db8603 / GHSA-g445-xwm7-594r ✅ Mass-assignment verification key, pre-auth account takeover 74a26a31 ✅ No rate-limit on sign-in code, brute-force ATO b41d1b83 ✅ Cross-tenant verification/invite key reuse, tenant takeover ce4f44bb ✅ XSS in markdown rendering and ATOM feed d28a838d / GHSA-wm2w-gfh7-qg69 ✅ Server-side JS injection in React SSR d5a80ea5 ✅ HTML escaping in rendered emails 2f7aa747 ✅ DoS via unbounded HTTP response read da89c502 ✅ IDOR / moderation bypass on a single comment d74a643d ❌ Authenticated arbitrary blob overwrite 7b047158 / GHSA-vxp5-mf8m-grg9 ❌ So Hacktron found 8 of the 10 bugs Fider recently fixed. More importantly, it caught every critical and high severity bug in the list: the pre-auth account takeover, the SSRF, the cross-tenant tenant takeover, the brute-force ATO, and the markdown/ATOM XSS. The two it missed were not critical or high. Here are all the top findings: Critical Auth bypass via action-binding key overwrite (mass-assignment) Action structs expose VerificationKey / VerificationCode / LinkKey with no json:"-" tag. Attacker can overwrite email verification code and takeover the account. Critical Cross-tenant verification key reuse / private-tenant registration bypass email_verifications keys are looked up globally, and they are not scoped to tenant. A key issued for tenant A activates tenant B. Bypasses private-tenant invite gating. Critical No rate limit on sign-in verification code (brute-force ATO) /_api/signin/verify has no lockout or attempt cap. High SSRF via webhook (+ response leakage) Admin webhook URLs fetched with no SSRF guard. High Infinite loop DoS in log placeholder parsing log.Parse re-scans the rebuilt string; a URL containing @{URL:magenta} re-injects itself into an infinite loop. One unauthenticated GET drove CPU 0 to 569%. High SSRF via custom OAuth provider config Same unguarded http.DefaultClient sink as #11, reached through admin OAuth provider URLs. Medium Server-side JS injection in React SSR URL RawQuery interpolated verbatim into the ssrRender("<URL>", ...) JS string. A double quote in the query breaks out and runs in the V8 SSR context. Medium Synchronous blocking on background worker queue (DoS) Buffered queue, enqueuers block once it fills faster than it drains. Medium Unbounded request body read (memory exhaustion) WrapRequest reads the body with no size cap. Takeaway For $350 and half an hour, on the exact same app, Hacktron found more real vulnerabilities than either $4,000 platform. As we mentioned in Why Mythos Doesn’t Matter for Us , most application-layer bugs do not need the most expensive model on every step. A good harness routes between stronger and cheaper models (Opus, Gemini Pro, and Gemini Flash), with similar results at lower compute cost: Capability lower bound (100% detection) Capability upper bound (0% detection) References Hacktron’s full scan of Fider Doyensec’s comparison report (PDF)