Dr. Jane Pinelas, Chief of Testing, Evaluation, and Assessment (TE&A) at the Defense Department’s (DoD) Joint Artificial Intelligence Center (JAIC), said on August 12 that the Centers of Excellence (CoE) engagement between JAIC and the General Services Administration (GSA) is paying dividends at the organization’s product testing needs ramp up.
Speaking at GSA’s Impact Summit 2020 event, Dr. Pinelas said that GSA – whose Technology Transformation Services (TTS) organization inked the AI CoE engagement with JAIC in September 2019 – “has been helpful” in efforts to create “automated and repeatable testing” methods for the TE&A organization.
As JAIC ramps up, so does the work of the TE&A organization, she said.
“We are the plank holders to make sure AI systems in the department perform like they are supposed to, and don’t do what they are not supposed to,” Dr. Pinelas said. She added, “we evaluate all of the JAIC products according to their mission,” and then “productize the knowledge that we develop” and drive that across the rest of DoD.
Part of that mission is undertaken through policy discussions, and TE&A has a seat at the table. She said her organization “doesn’t dictate policy on AI, but we participate in a lot of those discussions” with a focus on safety and ethics for JAIC systems.
To spur faster development and cut down on bureaucratic challenges that exist in traditional product testing environments, TE&A eschews “linear” testing processes and instead works through an agile model where “you keep iterating until the product is good enough,” Dr. Pinelas said.
“We work as an integrated product team” at JAIC, she said, adding, “this is a new process within DoD.”
In keeping with the cutting-edge nature of AI development, testing and evaluation is also breaking new ground as it goes along, she said. With “testing at the edge,” she said, “continuous testing and development must take place,” and incorporate the idea that “you can make a product and then change it.”
And in the bigger picture, that testing has to drive progress toward the bottom line of utility – or what Dr. Pinelas called “testing at the speed of relevance.”
“It is absolutely imperative that we balance testing” with practical product concerns, because “we don’t want to take so long that the system becomes irrelevant to the warfighter.”
On the ethics front, Dr. Pinelas said ensuring adherence to JAIC’s AI Ethical Principles is “incredibly important” to JAIC, which is looking to expand the team of the organization’s chief ethicist. Replying to a question about managing ethics into the process, she said “think of DevSecEthOps . . . with ethics built inside.”
