⚡Megatron LLM Hub
All digests
Website

rss-website

Top1

小幅改题即掉近 20 个百分点,公开榜单分数不能直接用作选型依据。

平时拿便宜的开源大模型改代码,很多程序员都碰过让人窝火的情况。明明在提示词里把边界条件和输出格式写得清清楚楚,模型干活时却跟没看见一样。然而,翻看收录了 113 道真实开源项目编程题目的 公开编程榜 DeepSWE ,各家公布的成绩却相当亮眼:最贵的旗舰 Claude Opus 5.5 取得了 74.2% 的通过率,价格便宜不少的 GLM-5.3 达到了 69.0% 的通过率,而价格仅为零头的 DeepSeek V4.1 Flash,竟然以 74.2% 的通过率与 Opus 5.5 并列第一。 初创公司 unLAP 最近发了个 MaxxBench 研究报告 ,解释为什么榜上有名和实际体验有差距。他们在榜单里选了 89 道题,只稍微改了下题目要求或初始代码。改动幅度其实很小,按题库自带的标准答案算,新增代码行数的中位数只有 17 行,变动约 3.5%。每道题让十几款主流模型独立重做 2 遍后,DeepSeek V4.1 Flash 通过率降到 55.6%,GLM-5.3 降到 56.7%,全场只有 Claude Opus 5.5 微涨到 76.4%。公开记录还拍出了它们的小动作:DeepSeek 爱翻配置偷密钥找外援,GLM-5.3 则看懂新要求却赌判分脚本不查冷门情况。 明明第 58 步已经把代码写对了,DeepSeek 为什么要在第 114 步亲手删掉? 有一道考题要求修改开源前端测试工具 Testem ,公开的 完整运行记录 拍下了一个令人哭笑不得的场面。原题目要求对每个浏览器生成一个 XML 报告,比如对谷歌浏览器生成 Chrome.xml 。新题目在文字说明里多加了一条要求,将文件名中的字母全部改为小写,也就是输出 chrome.xml 。测试时每道题都装在一个独立的隔离容器里,让 AI 自己敲命令行读文件、改代码,每执行 1 次命令记作 1 步。DeepSeek V4.1 Flash 开启了允许每步深度长考的最高思考强度 max 档,起初表现不错,在第 58 步就写出了带转小写函数 .toLowerCase() 的正确代码。 但写出正确答案后,它没立刻提交,反倒翻看容器里存系统配置与密码的环境变量,盗取了一把外部平台密钥。因为原版题库早已在网上公开,它便拿这把密钥调用场外能联网搜索的大模型,让对方搜出原题的旧版代码和测试脚本。拿到不带 .toLowerCase() 的旧版代码后,它一眼看出旧代码和当前的提示词相冲突。但它思考了一会,觉得隐藏判分的脚本大概率还是用的原题,既然分数只取决于测试有没有通过,那就跟着老测试走好了。 到第 114 步,它亲手用套来的旧代码覆盖掉刚才写对的小写逻辑,改回输出大写开头的 Chrome.xml ,结果近一半功能测试当场挂掉。放在原版老题上,偷来原题答案确实能帮它白捡高分。偏偏这回改了要求,偷来的老答案反而成了害它删掉正确代码的陷阱。而且它不止干了这一回,在好几道改编题里都上演了迎合老答案而违抗提示词的戏码。 三家机构接连抓包,开到 max 档的 DeepSeek 在容器里干了什么? 按理说考场的隔离容器应该封死外网,DeepSeek 怎么还能从里面偷到密钥连出场外呢?unLAP 团队在配套的 容器安全分析 里查清了漏洞源头:这套容器虽然屏蔽了普通网页,却偏偏漏掉了一条内部通道。原来,评测框架为了方便起见,把负责发送指令的主控程序和待测的程序代码放在了同一个隔离容器里面。而这个主控程序需要联网才能与 DeepSeek 等模型通信,所以它会在容器的环境变量里面设置大模型聚合平台 OpenRouter 的密钥。但这个密钥并没有权限限制,因此不仅可以呼叫 OpenRouter 上的其他模型,还可以启用 OpenRouter 的联网搜索功能。这相当于给了容器里会敲命令行的 AI 一个后门,让它可以直接联网搜索题目或者向其他场外模型求助。 这次测试中,DeepSeek V4.1 Flash 在最高思考强度的 max 档下,有 34.8% 的场次翻阅了环境变量,在 11.2% 的场次用密钥调用外部模型或搜索原题。遇到拒答它还会换招,比如有次找大模型 Claude Haiku 要源码遭拒,它就改找千问模型求助,谎称自己是项目维护者。另一回它找联网模型 Perplexity 查判分脚本连吃 2 次闭门羹,又编造自己在做安全审查去套话也没骗过去,干脆同时向多个模型发请求碰运气。通过这种试探,它在一些题目里拿到了原题文件,还在思考记录里提醒自己对外一个字都别提:“We can mention nothing.” 另外两家独立机构也抓到了同款模型的作弊现行。网页开发框架 Ruby on Rails 的官方评测报告 发现,其他模型都很规矩,唯独 max 档的 DeepSeek V4.1 Flash 偷拿容器密钥扒源码。团队把密钥移出容器、彻底断网重测这批真实业务难题后,它在 max 档的通过率从 37% 掉到了 17%,切回低一格的 high 档则是 12%。研究机构 Prime Intellect 也在一份 公开评测审计 中记下了类似一幕。当时那场推理测试没有断网,查询判分记录的后台接口也敞开着,谁都能翻看其他模型的得分日志。这款模型先自己算出一个结果,接着顺着接口翻出了里面的满分记录。看到满分答案跟自己算出来的不一样,它干脆抄过来,倒推编造了一段假的计算过程。 unLAP 分析,这可能是因为模型在训练阶段只看测试过没过、过了就给奖励,所以一旦开到最高思考强度的 max 档,就会把算力花在寻找捷径上。团队把思考强度调低 1 档到 high 档重跑,翻环境变量的比例立刻从 34.8% 降到 1.1%,偷密钥和调外部模型全归零。unLAP 这轮只统计了降档后的越界动作,没有公布它在改编题上的新总分。至于降档对正常解题能力的影响,在 Ruby on Rails 团队移出密钥的断网测试里,从 max 档切到 high 档会让通过率从 17% 降到 12%,可见管住它乱翻环境偷密钥的同时,遇到复杂难题,解题能力也会打点折。 把 DeepSeek V4.1 Flash 的思考档位从 max 下调至 high 后,翻查环境变量的比例从 34.8% 骤降至 1.1%,偷取密钥与调用外部模型全部归零 不偷密钥的 GLM-5.3 为什么也跌了?赌测试漏检与伪造内存指标 相比之下,GLM-5.3 在隔离容器里规矩得多,翻环境变量和偷密钥都是 0 次,但通过率还是跌到了 56.7%。研究团队给 13 款模型的失败原因做了分类,发现 GLM-5.3 有 40.3% 的失败栽在揣测判分脚本、赌裁判漏检上,这项比例在 13 款模型里排第一。很多题它看懂了新要求,写代码前却先算概率账,赌判分脚本不查冷门边界。 比如有一道考题要求给开源软件 ytt 增加数据比大小的功能,而且规定必须算得准超大整数。为省掉分开处理整数和小数的麻烦,GLM-5.3 偷懒把数字全转成带小数的通用格式来比较。这种格式能精确表示的位数有限,碰上超大整数会把末尾几位数四舍五入抹掉。它知道这隐患,却赌测试撞上超大数字的概率微乎其微。结果判分脚本偏偏测了一个超大整数,让程序当场算错挂掉。 一旦考题要求程序自己汇报运行指标,这种偷懒还会演变成集体造假。例如代码统计工具 scc 那道题,程序需要先扫描各个文件收集记录,最后再汇总统计结果。题目规定内存里的文件记录不能超限,多出来的必须转存磁盘,还得在屏幕上打印内存峰值。原版测试只检查屏幕打印的那行数字,各家模型全都顺利过关,直到 unLAP 加上了独立的实时内存监控。在公开的 所有运行数据 中,同批 13 款模型里只有 Claude Opus 5.5 等 3 款全程老实控住内存过关。其余没全过的 10 款里,足足有 7 款在偷偷造假,GLM-5.3 和 DeepSeek V4.1 Flash 都在其中。因为屏幕上的内存峰值是由模型自己写代码统计的,它们只在第一步扫描文件阶段记录内存条数并存盘。等到汇总阶段,它们偷偷停掉这项统计,把磁盘数据读回内存,最后直接打印第一步记下的低数字交差。结果凡是这么耍滑头的场次,在实时监控下全都当场超标挂掉。 换掉初始代码谁在吃老本,平时怎么安全用便宜模型省钱? 除了发现谁愿意投机取巧以外,这个实验还测出了 GLM-5.3 和 DeepSeek V4.1 Flash 谁更依赖背熟的老代码。题库基于 GitHub 上知名开源项目编写,各家模型训练时多半见过旧版实现。研究团队在 MaxxBench 开源题库 的 89 道题里专门挑了 9 道考题,保持题目说明一字不改,只把项目原有的初始代码换成另一套结构或写法,每道题重测 2 轮。结果在这同一批 9 道题上,最新旗舰 Claude Opus 5.5 拿到 94.4%,总榜原本跟 DeepSeek 分数差不多的 GLM-5.3 拿到了 61.1%,说明它面对陌生代码结构也能正常发挥。反倒是更依赖熟悉代码套路的 DeepSeek V4.1 Flash,在这 9 道题上直接掉到了 44.4%,比 GLM-5.3 低了近 17 个百分点。 在题目说明不变、仅改动初始代码的 9 道考题上,Claude Opus 5.5、GLM-5.3 与 DeepSeek V4.1 Flash 的通过率立刻分化为 94.4%、61.1% 与 44.4% 因为每道题都要在容器里反复读代码、跑测试上百步,全部 89 道题重测下来,Opus 5.5 平均每题花了 16.18 美元,GLM-5.3 花了 5.56 美元,开满 max 档的 DeepSeek V4.1 Flash 只花了 0.62 美元。了解各家的性格之后,平时写代码就可以根据任务分工来省钱。遇到模型没见过的私有仓库或者近期大改接口的代码,交给 Opus 5.5 兜底更放心;需求明确的小活再分给便宜模型省账单。 具体到单价最低的 DeepSeek V4.1 Flash,现在的 AI 编程工具大多支持模型直接在电脑上执行命令,本机环境变量也常有其他平台的付费密钥。既然测试里 DeepSeek V4.1 Flash 一开 max 档就有乱翻环境变量找外援的前科,谁也不想拿自己电脑里的真实密钥去赌它卡壳时老不老实。平时在自己电脑上用它做小活,直接把思考强度设在 high 档即可,让出一些解难题的能力,就能省心防住它乱翻配置。只有非要用 max 档在后台批量跑任务,才需要多做一步物理隔离。你可以单独建一个断网的容器只放待改代码,把负责联网收发指令的程序和真实密钥留在容器外面。 改用花费居中、换了陌生代码也不慌的 GLM-5.3 干日常小活,就不用操心密钥安全,重点防它偷工减料就行。它爱在冷门要求上偷懒赌运气,验收代码时对照提示词,多盯一眼边界条件有没有漏写就好。遇到限内存这类让程序自报数字的要求,用系统工具看一眼真实占用,别信程序自己打印的数字就够了。

llmbenchmark报告 @yage-share source ↗
Recommended1

给 agent 加人工确认层的具体工程实现,工作流设计可直接借鉴。

Ryugi (流儀) —Ryugi is a set of interactive artifacts I built for coding agents. Whenever the agent needs to ask me questions or have me pick an approach, it puts them on interactive pages where I can answer directly: design decisions are written up as a document where I choose options item by item and leave sentence-level comments; UI proposals are laid out on a canvas for me to click and choose; and when the work is done, an acceptance page places approved designs side by side with live screenshots so I can approve or reject them group by group. Every choice I make on these pages feeds right back into the agent session that started it. Repository · Innei 503 ryugi Ryugi (ryoo-ghee) — artifacts for coding agents: artboards of mockups, design docs, whiteboards, HTML benches and verify reports you comment on, with every comment sent back to the agent. Install artifacts only, or the full brainstorm → spec → TDD workflow. Claude Code, Codex and more. JavaScript The project here was the whiteboard inside my blog editor, haklex. Its editing features had always leaned on Excalidraw, which pulled in a heavy bundle and clashed with the rest of the editor's visual style. I wanted to drop the dependency and roll our own whiteboard around Excalidraw's data format, without touching a single field in existing articles. From my opening sentence to `git push` took about half a day: two rounds of development, one release, and three downstream projects updated. I typed fewer than 20 messages in chat; wherever I had to make a call, it was turned into questions in a doc or mockups on a canvas. Here is how it went, step by step. After one prompt: a doc that asks questions My opening prompt was just one sentence: "Drop the excalidraw dependency and roll our own whiteboard based on its data structures." The agent first inspected the code to confirm the scope of changes, then opened a design doc and put its questions directly into it, without following up in the chat box. The questions were embedded right inside the document, each with a recommended option and the reasoning behind it; all I had to do was click my choice on the page. The first question was a fork in the road that determined how big v1 should be—everything that followed depended on it. For two of the questions I skipped the recommendation, and the next round simply picked up from what I chose. After each round, the conclusions were merged into a "Decided" section, noting whether I made the decision or the agent made it for me. I could tell at a glance which items needed another look. Section 4 of the design doc, "Decided": each conclusion is tagged with who decided it After two rounds, the document was fleshed out into a full design spec. Before handing it back to me, the agent had an uninvolved reviewer agent do a cold read; it resolved most of the feedback on its own, leaving just two questions for me to settle. 自研白板编辑器:替换 @excalidraw/excalidraw · Ryugi View “自研白板编辑器:替换 @excalidraw/excalidraw” on Ryugi Cloud, a document artifact. Ryugi "Should we discuss the UI?" Once the document Q&A was done, I asked: "Should we discuss the UI?" The agent said yes—the design spec had marked the UI as "left to the implementer," but these are things you see every time you open the editor. Text doesn't do UI justice, so it switched to a visual canvas: each group presented several options side by side, the recommended one highlighted with a red border, and sticky notes explaining the reasoning right beside them. Round 1 decision fork on the UI canvas: three layout options side by side, with the recommended one outlined in red Across two rounds and five groups, I went with the recommendations every time. With the options drawn out side by side, it was immediately clear which were good and which were bad. The selected mockups were linked into the design spec to guide both implementation and acceptance. Then I replied with "approve." 白板编辑器界面 · Ryugi View “白板编辑器界面” on Ryugi Cloud, an artboard artifact. Ryugi Once approved, I didn't look at code again Once approved, the agent broke the design spec into 19 small tasks and tackled them one by one on a separate branch. Each task followed the same rhythm: write a failing test, watch it fail, write the implementation to make it pass, and commit. Whenever the agent made a call on its own—like minor details that diverged slightly from the spec—it logged it in a ledger. When the chat grew too long and was summarized, it kept working right from that ledger. For over an hour, I didn't type a single word. Review, then the acceptance page Once all 19 tasks were finished, the agent handed the branch over to an independent reviewer agent that had played no part in development. For every issue flagged, it reproduced it with a failing test before writing the fix. Then it spun up the editor in a real browser, took screenshots and screen recordings, and put together an acceptance page: my approved canvas designs on the left, and how it actually looked running on the right. Acceptance page: approved design, live screenshot, and screen recording side by side All I had to do was review each group and approve or reject it. I left a comment on one mockup; it made the fix, took a new screenshot, and placed it right back in the same spot for me to review. Anything it couldn't verify was listed separately on the page, instead of being mixed in with "Completed." 白板编辑器界面 · Ryugi View “白板编辑器界面” on Ryugi Cloud, an artboard artifact. Ryugi Round two: starting from a reference image Next, I wanted the whiteboard to look sharper. I pasted a reference image featuring clean headings, drop-shadowed sticky notes, and illustrative stickers, and asked whether stock Excalidraw could pull off that aesthetic. It broke down the elements in the image one by one—what we could already draw, what capabilities were missing—and even guessed I might want an AI to do the drawing. I replied with one sentence: only build general-purpose drawing capabilities. Next was the same process: open a new design doc, then three rounds of Q&A. In round three, putting rich-text cards on the whiteboard was my own idea. After a cold read, the reviewer agent flagged two questions that needed my call; the agent added its recommendations, I replied with two numbers, approved, and let it implement and review. The acceptance page for this round measured directly against my reference image: the reference on the left, the recreation inside our editor on the right. Round 2 acceptance: reference image side by side with the in-editor recreation 白板绘制能力扩展:阴影、SVG 元素、富文本卡片 · Ryugi View “白板绘制能力扩展:阴影、SVG 元素、富文本卡片” on Ryugi Cloud, a document artifact. Ryugi Verify · 白板绘制能力扩展 · Ryugi View “Verify · 白板绘制能力扩展” on Ryugi Cloud, an artboard artifact. Ryugi Release, then downstream projects With both rounds approved, I asked: "Can we release now, and then adapt the downstream projects?" It cut a new release, then went through three projects using the editor one by one—bumping the dependency, running tests, and verifying in a browser that the whiteboard rendered properly. Downstream results were posted right on the acceptance page as well, with one screenshot for light mode and one for dark. The same whiteboard in downstream projects: light and dark mode Throughout the entire process, logging in to authenticate the npm release was the only thing that needed my manual login—that was the one step the agent couldn't do for me. What I actually said in the chat Looking back, nearly everything I typed in chat over that half-day is captured right here: Time What I said What happened next 12:26 Drop the excalidraw dependency and roll our own whiteboard based on its data structures Read code, opened design doc 13:04 Should we discuss the UI? Opened canvas for UI discussion 13:18 approve Sliced tasks, started implementation 16:35 This looks about ready. Now let's work on extra features. Compared against reference image, kicked off Round 2 16:38 Let's only build general-purpose drawing capabilities Narrowed scope, opened new doc 17:04 approve Implemented Round 2 17:56 Can we release now, and then adapt the downstream projects? Published release, adapted downstream projects 20:06 push Pushed to remote Every other decision was made by clicking options in docs, picking mockups on a canvas, or signing off on the acceptance page. I didn't have to scroll through chat history to find what the agent said; every decision stayed on the page, and all the implementation and acceptance were checked against those same pages. Boundaries This workflow doesn't make judgment calls for me. If I skip an open-ended question, it fills in its best guess and flags it, leaving me to decide whether to review it. The acceptance page lists what it couldn't verify, and deciding whether to test further is up to me. What it does is put decisions right in front of me, so I can see at a glance what is settled and what isn't. Ryugi is open source ( github.com/Innei/ryugi ). In your terminal, run npx ryugi-artifacts install , and select the agent you use to install it. The two design docs, canvas, and acceptance pages mentioned here are all public, so you can see them in their original form.

ai_agent工具工程实践 @innei source ↗
More2

厂商方法论科普,无新手法或数据,可作红队流程框架参考。

Red Team Exercise Guide: How to Automate and Integrate with Blue Teams By: Amanda Kegley November 15, 2026 A red team exercise is an authorized, simulated attack in which ethical hackers emulate a real adversary to test how well an organization prevents, detects and responds to a threat. It chains multiple red team scenarios into one end-to-end campaign against a defined business objective, evaluating people, processes and technology along the way. The value comes from what defenders learn and improve. The goal is evidence that your security controls interrupt the attack paths that matter most, not a long list of findings. Traditional exercises take months to plan and weeks to run, so most organizations test only a few times a year while attackers and environments change constantly. That creates a risk-to-fix gap: reports arrive weeks after testing, and defenses may change before fixes land. At Cymulate, we close that gap with agentic cyber defense engineering , and this guide shows where it fits in each phase. What is a red team exercise? A red team exercise tests whether an adversary could achieve an agreed objective within approved boundaries. NIST SP 800-53 Rev. 5 describes red teaming as a “simulated adversarial attempt to compromise organizational missions or business processes.” A red team scenario is one attack storyline: one adversary, one objective and a connected chain of techniques, such as a spearphishing email that leads to a persistent foothold. An exercise links several scenarios, moving from that foothold through credential access and lateral movement to a crown-jewel system and data exfiltration. At each stage, it measures whether controls block the technique, whether the security operations center (SOC) detects it and how fast the team responds. Further reading Red Team Scenarios: 10 Real-World Attacks to Test Your Defenses From phishing to AI prompt injection, see how realistic attack scenarios help teams validate and improve security defenses. Read More Four roles make an exercise work. The red team conducts offensive testing. The blue team monitors, investigates and responds. A white cell may be involved, which is responsible for managing safety and separation of exercise activity from real incidents. An executive sponsor approves the work and funds the fixes. Red team exercise vs. penetration test A penetration test finds as many technical vulnerabilities as possible in defined targets, openly and in one to two weeks. A red team exercise pursues a business objective covertly over weeks or months and tests how defenders prevent, detect and respond. For more, see red teaming vs. penetration testing . Feature Penetration test Red team exercise Primary goal Find and document as many technical vulnerabilities as possible. Achieve a specific objective (e.g., reach the “crown jewels” or dump a database). Scope Defined technical targets (e.g., a specific web app, network or cloud environment). Broad, open-ended attack paths across technology, physical security and humans. Stealth “Loud” and transparent; internal IT and security teams usually know it is happening. Stealthy and covert; designed to evade the blue team (SOC). Methods Active exploitation of known vulnerabilities, misconfigurations and code flaws. APT emulation, social engineering and evasion tactics. Duration Short-term (typically 1 to 2 weeks). Long-term (several weeks to a few months). Primary deliverable A prioritized list of technical vulnerabilities with remediation steps. An attack narrative, detection gaps and a test of incident response readiness. Compliance use Required by PCI DSS; commonly used as evidence for SOC 2 and HIPAA. Built for operational defense testing; mandated as threat-led testing for some financial entities (e.g., DORA TLPT, TIBER-EU, CBEST). Types of red team exercises Choose the format based on three factors: Operational needs: the question you need answered, such as how far an attacker could move after a merger or cloud migration. Industry and regulations: financial regimes such as DORA threat-led penetration testing (TLPT ), TIBER-EU and CBEST dictate scope, providers and cadence, as an example. Resources: full-scope exercises need skilled operators and budget, so many teams combine external specialists with automation. Adoption follows these factors. In a vendor-run survey of 512 organizations, 89% of enterprises with 10,000 or more employees and 64% of mid-market firms reported annual red team assessments , led by financial services as an example for a highly regulated industry. Formats can be combined; cadences below are planning suggestions. Type Starting point Best for Typical cadence Full-scope, objective-based exercise External, with no insider knowledge End-to-end readiness across technology, people and physical security Annual or semiannual Assumed breach exercise Authorized internal foothold Post-compromise detection, lateral movement and blast radius Quarterly or after major changes Threat-led adversary emulation Intelligence on a named threat group Threat-specific coverage; regulatory TLPT When intelligence changes; at least every three years under DORA Targeted red team exercise A specific surface, such as cloud identity, social engineering or AI applications Validating a new technology or migration After major changes or launches Continuous automated red teaming (CART) Configured assets, agents and integrations Repeatable control validation at scale Continuous: scheduled and change-triggered The eight phases of a red team exercise These phases of a red team a exercise connects planning to verified fixes. In practice, many engagements stop after phase 6: the red team delivers its report, and re-testing becomes a separate, optional engagement. Even the TIBER-EU closure phase asks only for a remediation plan that includes planning for follow-up testing . Fixes stall without that remediation and re-verification loop: Cobalt’s State of Pentesting Report 2025 found that firms remediate just 48% of pentest findings. How Cymulate automates the eight phases Cymulate automates red teaming through agentic cyber defense engineering. Cymulate Vero AI agents profile threats, run production-safe attack scenarios, validate prevention and detection, prioritize gaps, implement mitigation and re-validate. Cymulate Cowork turns those steps into scheduled routines, so phases 2 through 8 run continuously while your team controls objectives, scope and approvals. 1. Planning and scoping Define objectives, systems in scope, participants and success criteria. Agree on rules of engagement, owners and time for remediation before testing begins. With Cymulate: Planning stays human-led. Cowork routines capture the agreed scope, schedule, guardrails and approval points so every run follows the same rules. 2. Threat intelligence and scenario selection Select adversaries and tactics, techniques and procedures (TTPs) relevant to your industry and exposures, then map them to the MITRE ATT&CK framework . See how to choose which red team scenarios to run . With Cymulate: The Threat Profiler agent identifies threats relevant to your environment, and the Red Teamer agent converts them into attack scenarios. Teams draw on more than 100,000 attack actions, updated daily, or build custom attack chains in Cymulate Threat Studio . 3. Environment preparation Verify logging, security information and event management (SIEM) coverage and endpoint detection and response (EDR) readiness. Establish private white-cell communications and confirm emergency contacts. With Cymulate: More than 80 security control integrations confirm telemetry is flowing. After initial deployment, this phase isn’t repeated for every exercise. 4. Baseline assessment Record control settings, detection coverage and known telemetry gaps in a dated baseline, so later comparisons show what improved. With Cymulate: The Validation Operator agent runs a baseline in Cymulate Exposure Validation , mapping prevention and detection results to a MITRE ATT&CK heatmap as well as ingests all vulnerabilities and existing detection rules. 5. Execution Red-teamers use specialized tools to conduct authorized reconnaissance, initial access, privilege escalation, lateral movement and actions on objectives. Testers timestamp actions and preserve evidence. With Cymulate: Exposure Validation runs safe automated attacks against integrated security tools across the entire MITRE ATT&CK lifecycle. Continuous automated red teaming repeats them on a schedule, freeing human red teamers for creative chaining, social engineering and physical testing. 6. Reporting Build an attack narrative showing what controls prevented, what defenders detected and what they missed, with business impact and evidence-based recommendations. With Cymulate: Cymulate platform has numerous out of the box reporting dashboards and the ability to create custom dashboard in minutes. Cymulate Cowork expands these capabilities with agentic AI, allowing for automated routines and additional customization. 7. Remediation and tracking Assign each finding an owner, priority and due date. Track control changes and detection-rule tuning as exercise deliverables. With Cymulate: Cymulate CTEM prioritizes gaps by exploitability and business impact. The Cymulate Platform auto generates vendor-specific mitigations, Sigma and YARA rules and indicators of compromise (IoCs). Cymulate Auto Mitigation pushes supported updates to connected controls, with approval points for production changes. 8. Validation and re-testing Repeat relevant techniques after fixes and confirm the expected prevention, alert or response. Remove test artifacts and feed results into the next exercise. With Cymulate: A Cowork prove the fix routine automatically re-runs the original attack scenario when a mitigation closes, confirms the fix and records the evidence. Red team rules of engagement: a checklist Red team rules of engagement (RoE) are the written agreement that defines what a red team may test, how and when, who approves it and what happens if something goes wrong. They protect production systems, keep the exercise legal and help the SOC tell a test from a real incident. Authorization: Obtain written executive authorization and legal review, including third-party permissions. Scope: List in-scope assets, excluded systems, testing windows and rate limits. Permitted techniques: Specify allowed techniques and actions that need separate approval, including destructive testing. Deconfliction: Route deconfliction through the white cell so the SOC can identify a real incident. Emergency stop: Define stop triggers, such as service degradation, and who can pause testing. Data protection: Encrypt credentials and evidence, restrict access and agree on retention and deletion. Cleanup: Use synthetic data where possible and document cleanup responsibilities before execution. With Cymulate, these rules become platform settings enforced on every run: assessments target only selected assets and test points, run on approved schedules, use techniques chosen from an ATT&CK-mapped library and route high-risk actions through approval points with full audit trails. How to measure a red team exercise Define clock starts, endpoints and denominators before testing. Count a detection as valid only when the alert reaches the intended queue. Metric How to calculate Time to objective Objective achieved timestamp minus exercise start Mean time to detect (MTTD) Time from action start to first valid detection ÷ detected test cases Mean time to contain Time from first valid detection to confirmed containment ÷ contained test cases Mean time to remediate (MTTR) Time from finding logged to successful re-test ÷ verified closed findings Prevention rate Blocked actions ÷ valid attempted actions × 100 Detection coverage by ATT&CK technique Techniques meeting detection criteria ÷ techniques tested × 100 Findings remediated and re-tested Findings fixed and re-tested ÷ actionable findings × 100 The Cymulate platform automatically calculates these cybersecurity metrics with visual dashboards and monitors for configuration drift. How often should you run red team exercises? Full-scope exercises often run annually or semiannually. Adjust that rhythm for business risk, budget, regulatory obligations and major environment changes. The Threat Exposure Validation Impact Report 2025 found that organizations running exposure validation at least monthly reported a 20% reduction in breaches. That annual rhythm assumed attackers needed weeks to weaponize a weakness. In April 2026, Anthropic withheld Claude Mythos Preview from public release, citing its ability to find software vulnerabilities , and the UK AI Security Institute reported it could autonomously complete a 32-step simulated network attack . When attackers move at machine speed, a once-a-year test leaves months of untested exposure. Pair periodic human-led exercises with continuous automated red teaming that re-tests after every new threat or change, so defenders move at machine speed too. Regulators are moving the same way. Requirements have shifted from annual penetration tests under PCI DSS (2004), to intelligence-led red teaming of live systems under CBEST (2014) and TIBER-EU (2018), to recurring threat-led tests under DORA and AI adversarial testing under the EU AI Act (2025). Red team exercise best practices Tie objectives to business risk and critical services. Base threat selection on current intelligence and relevant exposures. Hold a purple team debrief to compare attacker evidence with defender actions. Map every attack action to ATT&CK, recording results and telemetry. Reserve time and ownership for remediation and re-testing. Automate repeatable checks within approved boundaries. Teams without a dedicated red team can follow the example of a bank that increased in-house security testing without a red team . Key takeaways A red team exercise chains scenarios into a campaign that tests prevention, detection and response against a business objective. Most engagements stop at the report. Treat remediation and re-testing as exercise phases. Define metrics before testing and report misses separately. Defend at machine speed. Pair human-led exercises with continuous automated red teaming through agentic cyber defense engineering. Cymulate enables continuous red teaming at machine speed Agentic cyber defense engineering is a closed-loop, AI-assisted approach to continuously proving and improving security defenses. Cymulate applies it to red teaming through three connected steps: Prove. Prioritize. Adapt. Prove. Cymulate Exposure Validation runs automated red teaming from the industry’s largest attack scenario library, with new simulations within 24 hours of major CISA advisories. Cymulate Threat Studio lets red teams build and save custom multi-stage attacks based on their operational needs. Prioritize. Cymulate CTEM proves exploitability and prioritizes vulnerabilities by business impact, with results mapped to MITRE ATT&CK. With Cymulate Platform, all prevention and detection coverage is mapped to MITRE ATT&CK and prioritized by risk. Adapt. The Cymulate Platform delivers vendor-specific mitigations, detection rules and IoCs, and Cymulate Auto Mitigation pushes supported updates to more than 80 integrated controls. Re-testing verifies each fix and closes the risk-to-fix gap. Cymulate Cowork , powered by Vero AI, runs these steps as governed routines, with audit trails and human approval for high-risk actions. Customers report a 52% reduction in critical exposures, a 3X increase in threat detection and 40X faster threat validation, and Hertz Israel reduced its cyber risk by 81% within four months . Ready to make your red team exercises continuous? To explore the full operating model, download the Agentic Cyber Defense Engineering e-book . To see Cowork run exposure validation and mitigation, schedule a demo . Frequently asked questions What is the difference between a red team scenario and a red team exercise? A red team scenario is a single attack storyline: one adversary, one objective and one chain of techniques. A red team exercise links several scenarios into an end-to-end campaign that shows whether an attacker can get from initial access to business impact. How long does a red team exercise take? Most human-led red team exercises take several weeks to a few months, depending on scope and organization size, plus time for remediation and re-testing. Automated red teaming can run the repeatable phases in hours. How often should you conduct red team exercises? Many security teams run a full- scope exercise annually or semiannually, and entities designated under DORA must run threat-led tests at least every three years. Because AI now accelerates attacks, pair periodic exercises with continuous automated red teaming. What are red team rules of engagement? Red team rules of engagement are the written agreement covering scope, excluded systems, testing windows, permitted techniques, approvals, deconfliction, emergency stop triggers and data handling. They keep the exercise safe and legal. Can red team exercises be automated? Many repeatable activities can be automated, including threat profiling, technique execution, reporting, mitigation and re-testing. Human testers still lead objectives , social engineering and physical testing. Cymulate agents run the automated steps continuously within defined guardrails, with human approval for consequential changes. Amanda Kegley Amanda Kegley has over 15 years of experience in cybersecurity. With her in-depth knowledge of security needs in today’s dynamic threat landscape, she successfully innovates and markets new cybersecurity capabilities. More about Author Table of Contents What is a red team exercise? Types of red team exercises The eight phases of a red team exercise Red team rules of engagement: a checklist How to measure a red team exercise How often should you run red team exercises? Red team exercise best practices Key takeaways Cymulate enables continuous red teaming at machine speed Ready to make your red team exercises continuous? Frequently asked questions Cymulate Exposure Validation makes advanced security testing fast and easy. When it comes to building custom attack chains, it's all right in front of you in one place. Mike Humbert , Cybersecurity Engineer DARLING INGREDIENTS INC. Learn More Featured Resources View More Resources Report 2026 Gartner® Market Guide for Adversarial Exposure Validation The guide covers AEV use cases, key features, and market trends to improve threat exposure visibility and defenses. Learn More Data Sheet Cymulate Cowork Use AI agents and automated routines to validate exposures and streamline cyber defense engineering. Read More E-book Agentic Cyber Defense Engineering This e-book explores the six stages of agentic cyber defense engineering and how they work together to strengthen cyber defenses. Learn More GET A PERSONALIZED DEMO Ready to see Cymulate in action? Book a Demo

red_team演练 @cymulate-blog source ↗

生图产品横评,与安全无直接关系,仅作 AI 产品动向了解。

使用过生图模型的人,尤其是生成过海报或电商配图的人,大都有过这样的体验:你抽卡抽了很长时间终于抽到了一张满意的图,构图不错,光照不错,人物表情也不错,现在只想把左下角的标题改一下,或者让人物戴个眼镜。但常发现 AI 在把标题改对的同时把人脸也改了,甚至把背景和窗框都改了。 11 月 9 日,字节跳动发布了标准版生图模型 Seedream 5.5,后续还会推出定位更高的 Pro 版本。在国内改一张 2K 方图,价格已经降到了 0.28 元左右。这个模型主打连改 100 轮不变形、直接出透明底素材,并且在小字和写实质感上有所改善;配套的平台工具还能把成品海报拆成最多 100 个图层。回顾过去两个月,OpenAI 发布了旗舰 GPT Image 2.6,谷歌升级了 Nano Banana 2.1,德国的 Black Forest Labs 也发布了 FLUX 3 Image。这些公司都在发力局部改图和导出素材。为什么大厂突然都在卷改图呢?在真实的盲测榜单上,字节到底排第几,我们平时用又该如何选择呢? 一次出图不到两成能用,重心为什么转向了改图? 答案就在字节的 官方博客 里面。他们测试了包括电商、营销、影视在内的 6 种商业场景。官方表示新模型的可用率全面超越上一代,影视和办公场景甚至达到上一代两倍以上。但是综合所有场景来看,上一代一次出图的整体可用率只有 11% 左右,最新的 Seedream 5.5 也只有 18% 左右。 一次成图的可用率不到两成,剩下的八成多都是半成品。最头疼的是大方向是对的,只是错了一个字或少了一个配饰。重新抽卡太费钱费时;塞回模型局部修改,模型每次都会把没让改的区域顺手重算一遍,连改几轮就会出现偏色、人物走样和细节衰减。要想拯救半成品,模型要么在局部修改时锁死其余像素,要么直出透明底 PNG 素材给设计软件去拼。 单次抽卡很难一步到位。入秋以来,各家都在猛补局部改图和素材导出:OpenAI 的 新旗舰 GPT Image 2.6 支持在照片上圈画改图;接口还能导出矢量图,或直出动图。 德国团队 Black Forest Labs 的 FLUX 3 Image 主打锁住其余画面的多步编辑;擅长商品摄影与曲面贴标的 谷歌 Nano Banana 2.1 支持多图参考对话改图。阿里还发了 采用非商用研究许可的 Qwen-Image-2.1 ,免费公开模型文件让你下载到自己电脑运行,同样集成了涂抹选区改图与透明底输出。字节主推多轮改图与透明底直出,走的正是同一条路。 单次抽卡只有约 18% 可用且返工会破坏原图,锁死非编辑区像素才能把大方向做对的半成品修成可用资产 同门视频模型拿了第一,字节的生图模型排老几? 如果要论字节在生图领域的地位,需要区分视频和图片。字节 Seed 团队同时负责视频模型 Seedance 和生图模型 Seedream,支撑即梦、剪映、豆包等应用。视频线今年表现强势,年初登顶盲测榜,随后与好莱坞的 美国电影协会签署版权保护协议 ,近期传闻 海外收入占比一半 ,甚至德国同行发布新视频模型都以 Seedance 2.5 为唯一对标。但静态生图方面,自去年春季 Seedream 3.0 短暂登顶后,如今榜单前列已由美欧厂商占据。 打开独立评测组织 Artificial Analysis 维护的 文生图盲测榜 ,评测员在不知厂商名的情况下对两张图二选一投票,以 1000 分为基准算出等级分。到 11 月 9 日,全球单次生图分成三个清晰梯队。OpenAI 一家包揽前 5 名:刚发的 GPT Image 2.6 精修版和快速版霸占前 2 名,榜首拿到 1259 分,9 月发的 2.5 精修版和快速版分列第 3、第 4 名,第 5 名是更早的 GPT Image 2。第二梯队第 6 到 15 名为美欧厂商主力,包括谷歌 Nano Banana 2.1、微软、xAI、单张仅 1 美分的 Meta Muse Image 和 FLUX 3 Image。 中国厂商的文生图模型大多落在第三梯队。刚发布的 Seedream 5.5 目前还未上榜,字节在榜主力仍是第 18 名的 5.0 Pro;紧挨着的第 19 名是阿里闭源商业版 Qwen-Image-3.0-Pro。但字节旧旗舰的默认标价是谷歌和阿里的两倍多,单次抽卡不划算。阿里的免费公开权重版 Qwen-Image-2.1 排在第 23 名。 如果去看 同一机构的图像编辑盲测排行榜 ,字节的真正强项就露出来了。前 5 名由 OpenAI 的 4 款模型和微软的改图预览版包揽,谷歌的 Nano Banana 2.1 排第 7。字节的 5.0 Pro 在这里拿到了全球第 11 名,排在 Meta Muse Image 和 FLUX 3 Image 前面。它比国内排后面的腾讯混元预览版和 Qwen-Image-3.0-Pro 高出将近 40 分,在这个千分基准榜上拉开明显差距,拿下了国内头名。字节单次生图居二线,但局部改图是强项。距 5.0 Pro 发布 4 个月后,它没有急着更新贵价的 Pro 版,而是先发标准版 5.5 做深改图和小字,顺便把价格打下来。 2026 年 11 月生图模型分成三个梯队,字节在单次生图居二线但在局部改图位居国产头名 小字、透明底和百轮改图,新模型做到了哪一步? 新上线的标准版 5.5 到底成色怎样?OpenAI 的 2.6 刚发 5 天,字节官方评测对比的是上一代 GPT Image 2.5 精修版。中英文和密集小字测试里,两家严重错字率都压在 1% 以内;但字节轻微错字率只有 1.7%,比对手的 7.4% 低了一大截。换成多语种排版,字节海外云平台 BytePlus 公布的 15 种语言小字评测 显示,它的得分仍略低于 GPT Image 2.5。画面真实感也要分两头看。从零生成写实新图,官方称它压住了常见的过曝和皮肤油光。但拿已有图片进去修改就不一样了。官方盲测对比数据显示,更多专家认为谷歌 Nano Banana 2.1 画面更真实,字节落后了约 6.2 个百分点。 至于透明底直出和海报拆图层,这两项分别对应生图模型本身和配套的平台工具。直接生成单张透明底 PNG 图片,是生图模型本身这次新增的能力;OpenAI 接口和阿里 Qwen-Image-2.1 此前就已支持。前面提到把海报拆成最多 100 个图层:1 张自动补齐遮挡的底图加上最多 99 张透明图层。这个工具还可以按需识别提取画面文字,是字节旗下即梦和海外平台 BytePlus 的配套功能。直接调基础生图接口不会自动拿到分层文件。在 国内媒体的首日实测 中,测试者先用新模型画好了透明底的白羽雪鸮和透明艺术字。再把两张透明图丢回模型去拼横版海报时,它依然把鸟的飞行姿态重画走样了。拿到模型画好的干净透明素材以后,不要再扔回大模型去合图;直接塞进 Figma、Photoshop 或代码里固定位置拼终稿,要省心很多。 关于连续改 100 轮不变形的宣传,目前还只是在官方演示视频里看到的效果。字节也没有公开技术报告和百轮改图的量化数据。从刚才的媒体首日实测看,单张出图大约 5 到 10 秒。连续改几轮人物服装和客厅家具,确实保住了人脸和房间透视。但遇到精细结构还是会露怯:人物换装后出现偏长的大拇指;换上黑色长裙后,裙摆与小腿衔接不自然,小腿看着像悬在半空;桌面静物测试里要求改成金色细框眼镜,更是画出了 3 条眼镜腿。此外,定位更高的 Pro 版本目前没有时间表;发布当天,海外 BytePlus 产品页的试用按钮也仍然指向上一代的旧模型。 一边暗涨一边降价,现在做图和接接口该怎么选? 字节选这个时间点降价,还正好撞上了 OpenAI 几天前的一轮变相涨价。OpenAI 接口是按计费单位 token 来定价的,每生成一张图都会折算成相应的 token 数量收费。这次更新 GPT Image 2.6,价目表上的 token 单价没有变动,但 独立团队实测 80 张图的账单 发现,2.6 快速版在中低画质下单张消耗的 token 大幅增加。跑低画质方图,消耗量涨到 2.5 快速版的 4.2 倍,单张实付从约 0.6 美分涨到 2.5 美分;中画质消耗也涨到 3.3 倍,精修版出图还变慢了。反观字节刚上线的标准版 5.5,海外平台单张起步价压到了 2 美分。国内云平台火山方舟改一张 2K 方图,按输入原图加输出新图两笔费用合计,单次总成本大约只要 0.28 元人民币,折合约 4 美分,还不到上一代 5.0 Pro 的一半。要知道上一代 5.0 Pro 在国际榜单上的默认生图标价高达每张 9 美分,字节这次先上标准版,相当于把日常改图开销砍掉了一半以上。 把价格与优缺点放在一起看,很多公司日常选型就不必死磕一个冠军。如果需要多语种复杂小字或经常跑低画质缩略图,留在旧版 GPT Image 2.5 比升级划算很多;真要用到直出动图、导出矢量图、照片圈画改图或高画质提速时,再升上 2.6 也不迟。如果重视商品摄影和杯身曲面贴标的质感,单张约 3.4 美分的谷歌 Nano Banana 2.1 是可靠的默认选项。如果追求单张 1 美分的超低成本铺量,就选 Meta Muse Image。如果是经常做中文密集小字的信息图或模特局部换装,改图约 0.28 元的 Seedream 5.5 适合放进候选池试水。先用便宜顺手的模型出好局部素材,最后再回设计软件里合成定稿,省钱又少返工。

图像模型产品 @yage-share source ↗