Scale artificial intelligence to evaluate large language models for the Pentagon

The US Department of Defense has selected Scale AI to help it test and evaluate genetic artificial intelligence for military applications.

The California-based company announced the partnership with the Chief Digital and AI Office on February 20, the same day the CDAO was set to kick off its Washington, DC conference with discussions on the topic.

Generative AI is capable of generating text, images or other data output using algorithmic models in response to user prompts. Large language models are a type of genetic artificial intelligence that uses statistical relationships between text documents or other inputs, either by themselves or through a supervised learning process, to mine essays, computer code, human-like conversations, and more. Scale AI will produce benchmarks for such systems under the contract.

The Department of Defense has expressed growing interest in genetic artificial intelligence, but its uses remain debated. While a smart assistant or chatbot could efficiently find files, answer FAQs or discover contact information, such tools can also fuel disinformation campaigns, spoofing attempts and cyber-attacks. CDAO launched Task Force Lima in August to study and guide genetic artificial intelligence for national security purposes.

Alexandr Wang, founder and CEO of Scale, said in a statement that the company’s work will enable the adoption of verifiable and secure models in cyberspace. Scale’s testing and evaluation processes can detect misinformation, bias, leaked data and unqualified advice such as medical, legal or financial information.

“Testing and evaluating genetic AI will help the Department of Health understand the strengths and limitations of the technology so it can be developed responsibly.” he said. “Scale is honored to partner with the DOD in this regard.”

See also  Coordinating Instructions for the Conduct of Preliminary Examinations for ASEI-ASSY Candidates

Tuesday’s announcement did not include a dollar value for the project.

Wang told Congress in July that outdated data retention and management policies were plaguing the Defense Department. What is needed, he said then, is to “double down some of these fast procurement methods and ensuring we continue to innovate.”

“AI systems are only as good as the data they are trained on,” he added.

Scale in 2022 won a nearly $250 million contract to provide federal agencies with access to its technologies. The general purchase agreement was issued by the Joint AI Center, which was under CDAO.

Colin Demarest is a reporter at C4ISRNET, where he covers military networks, cyber and IT. Colin previously covered the Department of Energy and the National Nuclear Security Administration – specifically Cold War liquidation and nuclear weapons development – ​​for a daily newspaper in South Carolina. Colin is also an award winning photographer.

Read the original at

Related Posts