Short paper: Evaluating the capabilities of AI-based penetration testing tools

Advances in AI are extending the capabilities of tools for penetration testing. However, due to a fragmented market and rapid technical developments the extent of capabilities and maturity of available tools are not well understood. This short paper provides an overview of this area by reviewing recent academic literature and proposing several assessment criteria. The literature review identifies an active and growing research field, with numerous advancements in the past 18 months. A focus on LLM agents has progressed capabilities towards exploiting zero-day vulnerabilities. Frontier models show superior performance when combined with a focused knowledge base and multi-agent architectures. However, in most cases human involvement is still required, and fully autonomous solutions are not yet evident. To evaluate the maturity of tools, 12 assessment criteria within four broad categories are proposed: AI sophistication, action capabilities, features, and requirements. Maturity levels are proposed for each criterion which enables an objective benchmark of tool capabilities.