Search Results

Background. Agents have recently improved LLM-based vulnerability detection through retrieval, iterative analysis, and verifier modules. However, existing evaluations mainly report improvements in detection performance without determining whether these improvements reflect more reliable vulnerability reasoning or access to retrieved context. Aims. This study investigates how different agent components affect the reasoning-oriented evaluation dimensions of agent-based vulnerability detection. Method. We present a controlled measurement framework with six conditions and evaluate it on six base models. The conditions progress from plain single-pass prompting through matched-context replay and ReAct-style tool use to self-review, trace-isolated verification, and multi-role decomposition, allowing us to separate context effects, iterative orchestration, and added agent modules under a shared protocol. Beyond vulnerability detection performance metrics, our framework evaluates three reasoning-oriented dimensions: robustness under semantics-preserving perturbations, resistance to misleading non-causal context, and grounding in concrete code evidence. Results. The full agent often achieves higher accuracy or F1 than plain single-pass prompting, but these improvements do not consistently extend to the three measured behavioral dimensions of robustness, resistance, and grounding. The models are not consistently more stable under harmless code changes, are often misled by non-causal prompt cues, and rarely cite the exact code lines that explain the vulnerability. In the strongest improvement case, much of the accuracy change already appears when the model receives the same retrieved tool outputs without the full agent process; after that context is held fixed, the remaining agent effect is mainly associated with predicting vulnerable more often. Conclusions. Higher detection performance should not be taken as direct evidence of better vulnerability reasoning. In our experiments, better performance metrics can reflect retrieved context or changes in how often the model predicts vulnerable, rather than a clear improvement in stable, evidence-based vulnerability reasoning. Future evaluations should report these behavior checks alongside performance metrics.