I’ve been frequently sounding the alarm on the path that OpenAI has taken since it started its partnership with Microsoft. I’ve argued that the artificial intelligence lab has gradually swayed from pursuing science to creating profitable products for its main financial backer.
OpenAI CEO Sam Altman put some of my doubts to rest this week with a blog post in which he laid out the lab’s plan for artificial general intelligence (AGI). Regardless of where you stand on the AGI debate, the post includes some interesting points about how OpenAI plans to tackle the challenges of AI research and product development. And I think this is important because many other research labs will be facing similar challenges in the coming years.
Altman also leaves some questions unanswered‚ which might be fair given the constant changes that the field is going through. Here are some of my key takeaways from OpenAI’s AGI strategy.
AGI research might (expectedly) hit a wall
The intro of the post is about the benefits and risks of AGI. On the one hand, AGI can elevate humanity, discover new knowledge, turbocharge the economy, amplify creativity, etc. On the other, it can pose extreme risks, societal disruption, accidents, etc.
But there are a few interesting facts. Altman acknowledges that “our current progress could hit a wall.” I think this is important given the hype surrounding large language models (LLM). Constant progress in LLMs and other deep learning models has led some to believe that we’re on the path to creating AGI. But there are clear signs that LLMs alone cannot solve critical aspects of intelligence and can make fatal mistakes if entrusted with sensitive tasks.
One thing I would have liked Altman to elaborate on is their plans for research in domains other than LLMs. In the past, OpenAI conducted research in various fields, including robotics and different reinforcement learning systems. Those efforts have all but vanished from OpenAI’s research in the past few years. Although those areas might prove to be very expensive and not profitable (in the short term, at least), they might provide valuable information for solving the mysteries of intelligence.
Another important point Altman makes: “We don’t expect the future to be an unqualified utopia, but we want to maximize the good and minimize the bad, and for AGI to be an amplifier of humanity.” Now, this is a good point to make in general, but it leaves a few hard questions: How do we define what is good and bad? Who will be the arbiter of values? And what aspects of humanity do we want to amplify that can maximize the good and minimize the bad?
Gradual transition to AGI
I think the more interesting part of the post is OpenAI’s short-term strategy. Here, Altman’s thoughts reflect what OpenAI has learned from deploying AI systems to millions of users.
First, he speaks of a “gradual transition” being better than a sudden one because it allows us to adjust incrementally.
“A gradual transition gives people, policymakers, and institutions time to understand what’s happening, personally experience the benefits and downsides of these systems, adapt our economy, and to put regulation in place,” Altman writes.
Scientists, researchers, and practitioners that I’ve spoken to agree that given the impact that AI is having on everyday lives, it can no longer be something that is developed by scientists in a lab. It should become a multidisciplinary field that includes people from all sorts of domains, including human sciences, engineering, and legal sciences. This will help us better understand the impact that powerful deep learning models will have on society, economy, and politics.
Altman also speaks about handling AI deployment challenges with “a tight feedback loop of rapid learning and careful iteration.” This means having the mindset that everything you assumed about your technology could be wrong. Accordingly, it requires having the tools and infrastructure in place to constantly gather feedback from users and developers and to regularly update the models. OpenAI has mastered this to near perfection in recent years.
These practices have helped them take their technology in unexpected directions. For example, GPT-3 was initially meant for language-related tasks. Along the way, the OpenAI team learned that the same model could be fine-tuned for generating software code, which has become one of the most successful applications of LLMs so far.
AI is not open
OpenAI has been regularly criticized for not releasing its models to the public and using its research to make a profit and raise funding.
Altman defends the path that the lab has taken by saying, “The optimal decisions will depend on the path the technology takes” and that planning in a vacuum is very difficult. In the footnotes, he explains that their original structure as a non-profit did not work out because they didn’t think scale (i.e., training larger and more expensive neural networks) would be as important as it turned out to be. He also says that “we were wrong in our original thinking about openness,” which means they will continue to keep their more capable models hidden behind APIs.
Altman says, “We believe that democratized access will also lead to more and better research, decentralized power, more benefits, and a broader set of people contributing new ideas.”
I find this part a bit problematic. The API model that OpenAI uses will make it easier for more people to access AI systems without going through the technical difficulties of setting up models. And this will indeed help more people contribute new ideas to the field.
But some kinds of research will require access to the training data as well as the model weights and architectures, which OpenAI is not making available. Transparency and sharing have been among the cornerstones of scientific progress. But unfortunately, with the growing tendency for AI research labs to keep the details of their models secret, it is becoming harder for scientists from different organizations to collaborate.
Also, contrary to what Altman is saying, the kind of policy that OpenAI has pursued is not decentralizing power. It is concentrating it in the hands of Microsoft, which has an exclusive license to OpenAI’s technology. It has also triggered an AI arms race, with other big tech companies pursuing similar deals with other research labs.
Customized AI models
One of the important points Altman makes—which goes back to determining the values to promote with AI—is “creating increasingly aligned and steerable models.” One of OpenAI’s key achievements in this regard was the use of reinforcement learning from human feedback (RLHF) to align LLMs with human intents.
However, there are problems with baking alignment into models through a centralized process. After the release of ChatGPT, many users posted instances of the model apparently “taking sides” on sensitive political and social issues. It wasn’t clear whether the problem was due to the bias in the training data or the guardrails implemented by OpenAI. But what is clear is that you can’t find a universal solution that can satisfy the preferences of all individuals and groups.
Altman believes this can be solved by getting society to “agree on extremely wide bounds of how AI can be used” and giving users discretion to use the AI within those bounds. (I’m not sure if it will be possible to define “wide bounds” in a way that can provide enough flexibility while preventing misuse of the models at the same time. We’ll have to wait and see.)
“The ‘default setting’ of our products will likely be quite constrained, but we plan to make it easy for users to change the behavior of the AI they’re using,” Altman writes. “We believe in empowering individuals to make their own decisions and the inherent power of diversity of ideas.”
OpenAI already provides tools for fine-tuning GPT-3 with custom datasets. I’m hoping that in the future, they will roll out tools for RLHF fine-tuning as well. OpenAI is already doing research on several new AI alignment techniques. And the big takeaway has been that creating better AI does not always depend on inventing more advanced architectures. Finding ways to combine AI with human intuition helps do more with less and take existing technology to the next level.
Balancing AI research and profit-making
Altman says there needs to be a global conversation around three key areas: AI governance, fair distribution of benefits, and fair access. There is no blueprint for starting this global conversation. With so many conflicting interests, I’m not sure if such a conversation would be possible at all. And software is not something that can be regulated easily.
Altman also explains OpenAI’s efforts to align its scientific and commercial incentives. These efforts include a clause in OpenAI’s charter that states “if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project.”
I don’t know how OpenAI plans to implement this clause, given that we do not have any idea what a late-stage AGI system will look like. But it is good to have this broad framework to regularly reassess the company’s direction.
Altman also says that the company’s capped-profit structure will prevent shareholders from capturing value without bound and push the company toward “deploying something potentially catastrophically dangerous.” OpenAI may have set guardrails for itself, but the insane amounts of capital that it has raised and the direction that Microsoft has taken with integrating OpenAI’s technology into its products threaten to lead to the very catastrophically dangerous developments that OpenAI tries to prevent.
Altman makes two other important points. First, the need for independent audits of new systems. This might address some of the transparency issues I raised earlier and the problems caused by not making the models public. But it really depends on the details of the auditing process, which OpenAI will declare later in the year.
And second is the need to limit the rate of compute needed to train models. The computational requirements of models such as GPT-3 make it virtually impossible for smaller companies to carry out their own efforts. Research on increasing the capabilities of smaller models can help democratize the field, though it would also require open-sourcing the models.
Final thoughts
With so many developments in AI, expecting a perfect roadmap from OpenAI would be unrealistic. This is why I appreciate Sam Altman’s article despite all the questions that I would like to see answered.
Of course, there are no perfect answers to many of the questions that we have. But it is important to have these conversations regularly and see where we stand and adjust course. We’ll keep an eye out to see how it unfolds.