NIST staffers revolt against expected appointment of ‘effective altruist’ AI researcher to US AI Safety Institute

The National Institute of Standards and Technology (NIST) is facing an internal crisis as staff members and scientists have threatened to resign over the anticipated appointment of Paul Christiano to a crucial position at the agency’s newly-formed US AI Safety Institute (AISI), according to at least two sources with direct knowledge of the situation, who asked to remain anonymous.

Christiano, who is known for his ties to the effective altruism (EA) movement and its offshoot, longtermism (a view that prioritizes the long-term future of humanity, popularized by philosopher William MacAskill), was allegedly rushed through the hiring process without anyone knowing until today, one of the sources said.

The appointment of Christiano, which was said to come directly from Secretary of Commerce Gina Raimondo (NIST is part of the US Department of Commerce) has sparked outrage among NIST employees who fear that Christiano’s association with EA and longtermism could compromise the institute’s objectivity and integrity.

Many say EA — defined by the Center for Effective Altruism as an “intellectual project using evidence and reason to figure out how to benefit others as much as possible” — has turned into a cult-like group of highly influential and wealthy adherents (made famous by FTX founder and jailbird Sam Bankman-Fried) whose paramount concern revolves around preventing a future AI catastrophe from destroying humanity. Critics of the EA focus on this existential risk, or “x-risk,” say it is happening to the detriment of a necessary focus on current, measurable AI risks — including bias, misinformation, high-risk applications and traditional cybersecurity. 

The US AI Safety Institute was established in November 2023

The AISI was established in November 2023 to “support the responsibilities assigned to the Department of Commerce” under the AI Executive Order. Earlier today, US Senate Majority Leader Chuck Schumer (D-NY) announced that the NIST will receive up to $10 million to establish the US AI Safety Institute.

Last month, VentureBeat reported on criticism about NIST’s lack of transparency around the AISI: In mid-December, House Science Committee lawmakers from both parties sent a letter to NIST that Politico reported “chastised the agency for a lack of transparency and for failing to announce a competitive process for planned research grants related to the new U.S. AI Safety Institute.” That lack of transparency centered around a potential grant to RAND Corporation, “an influential think tank tied to tech billionaires, the AI industry and effective altruism.” (VentureBeat has also reported about a “widening web” of effective altruism adherents in AI ‘safety’ and security circles, including within RAND and leading LLM model company Anthropic).

‘Important to ensure that NIST’s mission is not affected’

Divyansh Kaushik, associate director for emerging technologies and national security at the Federation of American Scientists, told VentureBeat that regardless of who is appointed to the AISI role, “it is important to ensure that NIST’s mission is not affected by it.”

NIST has always done “methodical measurement research that is highly grounded,” he said. “We’re in uncharted territory with trying to establish an evaluation program for hypothetical risks from general purpose models. I totally understand the apprehension NIST scientists would be having and it’s on Commerce leadership to ensure that any appointments stay true to science and true to NIST’s mission and methodology.”

If certain appointments cause NIST’s “superstar scientists” to quit, he added, “that would be a shame — in that case, I’d certainly hope the Secretary looks at what she will lose by appointing one person and reconsider her choice.”

Congressional committees, he pointed out, are already paying close attention to AISI’s work. “I hope they’re able to do good work with good people who are the best measurement scientists and can do this work in a way that leans on NIST’s strengths rather than trying to remake what NIST’s mission is and what it does,” he said.

VentureBeat has reached out to Secretary of Commerce Gina Raimondo, AISI director Elizabeth Kelly, and Paul Christiano. We will update if and when we hear back.