Simon Willison's report highlights a critical, second-order tension in AI security testing. The fact that an AI model can autonomously exploit common vulnerabilities like weak passwords or exposed credentials is less surprising than the corporate decision-making around it. Google's reported choice to withhold disclosure until prompted, justified by a lack of 'harm,' appears to set a concerning precedent.
It suggests companies may be tempted to treat AI security incidents as internal R&D matters rather than events with broader implications for trust and policy. The industry's next challenge is defining what constitutes a reportable AI safety event before a regulator does it for them.