Imagine a world where artificial intelligence, capable of unimaginable feats, could also pose grave threats to humanity. That's the reality, and at the heart of it all is Zico Kolter, a professor at Carnegie Mellon University, who holds a pivotal role in the tech industry. He leads a crucial safety panel at OpenAI, the creators of ChatGPT, with the power to stop the release of potentially dangerous AI systems. But what does this mean? And why is it so important? Let's dive in.
Kolter's panel, composed of just four members, has the authority to halt the launch of new AI models if they're deemed unsafe. This could range from technology that could be weaponized for mass destruction to chatbots that could severely impact mental health. As Kolter himself stated, this isn't just about 'existential concerns'; it encompasses a broad spectrum of safety and security issues that arise with the widespread use of AI.
OpenAI appointed Kolter as chair of its Safety and Security Committee over a year ago. However, the significance of his position amplified recently when regulators in California and Delaware integrated his oversight into agreements allowing OpenAI to restructure and raise capital.
Safety has been a core tenet of OpenAI since its inception as a non-profit research lab a decade ago. The goal was to create beneficial, better-than-human AI. But the rapid commercial success of ChatGPT has led to accusations that the company rushed products to market, potentially compromising safety in the process.
Internal disagreements, including the temporary removal of CEO Sam Altman in 2023, brought these concerns into the spotlight. OpenAI's move to become a more traditional for-profit company also sparked pushback, including a lawsuit from co-founder Elon Musk.
Agreements announced last week by OpenAI, along with the California and Delaware Attorneys General, aimed to address these concerns. At the core of these commitments is the prioritization of safety and security over financial considerations as OpenAI transitions into a new public benefit corporation under the control of its nonprofit OpenAI Foundation.
Kolter will serve on the nonprofit's board but not the for-profit board. However, he will have full access to all for-profit board meetings and information related to AI safety decisions. According to the memorandum of understanding, Kolter is the only individual, besides the Attorney General, named in the document.
Kolter stated that the agreements largely confirm that his safety committee will retain its existing authority. The other members of the committee also sit on the OpenAI board, including former US Army General Paul Nakasone. Furthermore, Altman stepped down from the safety panel last year, which was seen as a move to grant it more independence.
"We have the ability to do things like request delays of model releases until certain mitigations are met," Kolter said. However, he declined to comment on whether the safety panel has ever halted or mitigated a release, citing confidentiality.
Kolter anticipates various concerns about AI agents in the coming years, from cybersecurity threats, such as accidental data exfiltration, to security issues surrounding AI model weights. But there are also new and specific challenges. "Do models enable malicious users to have much higher capabilities when it comes to things like designing bioweapons or performing malicious cyberattacks?" he asks. Additionally, he highlights the impact of AI models on people's mental health and the effects of interacting with these models, all of which must be addressed from a safety perspective.
But here's where it gets controversial... OpenAI has already faced criticism this year, including a wrongful-death lawsuit from parents whose teenage son died by suicide after extended interactions with ChatGPT.
Kolter, who heads Carnegie Mellon's machine learning department, began studying AI in the early 2000s, long before it gained mainstream popularity. "When I started working in machine learning, this was an esoteric, niche area," he said. "We called it machine learning because no one wanted to use the term AI because AI was this old-time field that had overpromised and underdelivered."
Kolter, now 42 years old, has followed OpenAI for years and even attended its launch party in 2015. However, he didn't anticipate the rapid advancements in AI. "I think very few people, even people working in machine learning deeply, really anticipated the current state we are in, the explosion of capabilities, the explosion of risks that are emerging right now," he said.
AI safety advocates are closely monitoring OpenAI's restructuring and Kolter's work. Nathan Calvin, general counsel at the AI policy nonprofit Encode, expressed cautious optimism, particularly if Kolter's group can "actually hire staff and play a robust role." He added, "I think he has the sort of background that makes sense for this role. He seems like a good choice to be running this." However, he also emphasized the importance of OpenAI staying true to its original mission, stating that these commitments could be significant if the board members take them seriously.
And this is the part most people miss...
What do you think? Do you believe Kolter's panel will be effective in ensuring AI safety? Are you concerned about the potential risks of AI, or do you see more benefits? Share your thoughts in the comments below!
