The Cybersecurity and Infrastructure Security Agency (CISA) made public a report Monday detailing the findings of a pilot that examined whether current Federal vulnerability detection software products that use AI – including large language models – are more effective.
According to CISA, one key finding of the pilot highlights that the amount of time needed for analysts to learn how to use the AI capabilities is “substantial” but the “incremental improvement gained may be negligible.”
The report – which was required by President Biden’s October 2023 AI executive order (EO) – was delivered to the White House on July 26.
President Biden’s AI EO directed the Department of Homeland Security to develop plans for, conduct, and complete an operational pilot using AI capabilities to aid in the detection and remediation of vulnerabilities in critical Federal software, systems, and networks.
From late 2023 to early 2024, CISA performed the pilot to examine whether current Federal vulnerability detection software products that use AI are more effective at detecting vulnerabilities than those that do not use AI.
The CISA pilot team used two scenarios for testing the AI tools: security assessments of Federal partner networks, and tests within a controlled environment. The pilot primarily focused on newer AI product types that were available for use prior to Dec. 31, 2023.
CISA’s key findings from the pilot include:
- The best use of AI for vulnerability detection currently lies in supplementing and enhancing as opposed to replacing existing tools;
- The amount of time needed for analysts to learn how to use the new capabilities is substantial and the incremental improvement gained may be “negligible”; and
- AI tools can be unpredictable in ways that are difficult to troubleshoot.
According to CISA, this operational pilot met the agency’s final non-recurring requirement under the AI EO.
Looking ahead, CISA said, “AI tools are improving constantly, and the CISA team will continue to monitor the market and test tools to ensure CISA’s vulnerability detection capabilities remain state-of-the-art.”
CISA’s report from came as the administration announced Friday that it has hit the nine-month mark of its AI EO, and Federal agencies remain on track with their required directives – with two agencies reporting they are ahead of schedule.
