⚡Megatron LLM Hub
←返回搜索
Twitter

实测:前沿模型配检索在安全任务上最优,微调开源模型垫底

自建安全 Agent 选型参考:检索增强胜过微调,开源模型落后
llmai_agent报告
these are the systems I tested: - vanilla open-weight model, no fine-tuning - open-weight model, fine-tuned on security corpus - open-weight model + (RAG)truth oracle for security corpus - Frontier model, no oracle - Frontier model + oracle guess which one performed better? across multiple runs. 🥇Frontier model + oracle 🥈Frontier model, no oracle 🥉open-weight model + (RAG)truth oracle for security corpus yes, open-weight model, fine-tuned on security corpus came at last place. So if you must use open-weights models, you are better off designing a truth oracle for security corpus with some v good retrieval/ranking system, instead of fine-tuning.