自建安全 Agent 选型参考:检索增强胜过微调,开源模型落后
llmai_agent报告
these are the systems I tested:
- vanilla open-weight model, no fine-tuning
- open-weight model, fine-tuned on security corpus
- open-weight model + (RAG)truth oracle for security corpus
- Frontier model, no oracle
- Frontier model + oracle
guess which one performed better? across multiple runs.
🥇Frontier model + oracle
🥈Frontier model, no oracle
🥉open-weight model + (RAG)truth oracle for security corpus
yes, open-weight model, fine-tuned on security corpus came at last place.
So if you must use open-weights models, you are better off designing a truth oracle for security corpus with some v good retrieval/ranking system, instead of fine-tuning.