Editorial analysis: This result matters because it isolates model capability from engineering scaffolding. Practitioners building vulnerability scanners often trade developer effort in harnessing, orchestration, and multimodal preprocessing for model performance. Semgrep’s experiment implies that, on some narrow tasks, an off-the-shelf open-weight model plus lightweight prompting can approach or exceed performance of a frontier coding agent, at materially lower per-finding cost.
What happened – Semgrep’s published benchmark compares multiple models on an IDOR (Insecure Direct Object Reference) detection task using the same prompt and dataset. Semgrep reports that `GLM-5.2` from Zhipu AI scored 39% F1, outperforming `Claude Code` at 32% F1; Semgrep also reports a cost of about $0.17 per vulnerability found for the GLM run. Semgrep additionally reports that its internal multimodal pipeline, which runs inside a purpose-built harness, achieved 53-61% F1, and that the models evaluated in this test were run in a simple harness without endpoint discovery or guided navigation.
Editorial analysis – technical context: Semgrep frames the experiment as a prompting-versus-harness comparison. That distinction matters: a harness that enumerates endpoints, narrows context, and post-processes model outputs can substantially boost end-to-end detection rates. Semgrep’s numbers show the harnessed multimodal pipeline still outperforms raw-model prompting by a wide margin, even when an open-weight model beats a frontier agent on prompt-only runs.
For practitioners: The takeaway is twofold. First, open-weight models such as `GLM-5.2` may be a cost-effective choice for probing large codebases where building a full harness is infeasible. Second, engineering investment in a well-designed harness remains likely to deliver the largest single-lift in detection performance, per Semgrep’s reported 53-61% F1 for its pipeline. Observers should treat the GLM result as a signal to re-evaluate prototype tooling choices, not as definitive proof that harnessing is unnecessary.