Altman's latest revelation: there is an urgent shortage of GPUs, and plans to open source GPT-3, and open multi-modality next year

Source: The Paper

Reporter Shao Wen

While Altman calls for regulation of the future of AI, he doesn't think the current model is dangerous and thinks it would be a big mistake to regulate or ban it.

·OpenAI's internal data shows that the scaling laws of model performance (Scaling laws) are still in effect, making the model larger will continue to produce performance.

Image source: Generated by Unbounded AI tool

Last week, more than 20 developers, including OpenAI CEO Sam Altman and AI development platform HumanLoop CEO Raza Habib, held a closed-door meeting to discuss OpenAI's API. (application programming interface) and product plans. A few days later, Raza Habibi published a detailed summary of the meeting's highlights.

In the discussion, Altman admitted very frankly that OpenAI is currently limited by the GPU (graphics processing unit), and talked about GPT-3's open source plan, OpenAI's current top priority, and the social impact of artificial intelligence. While Altman calls for regulation of the future of AI, he doesn't see the current model as dangerous.

Altman also said that OpenAI's internal data shows that the law that the performance of the model is proportional to the scale still holds true, that is, the larger the model, the stronger the performance. OpenAI will continue to try to make models bigger, but they will probably only double or triple in size every year, not by many orders of magnitude.

Currently severely limited by the GPU

A recurring theme throughout the discussion was that OpenAI's current heavy reliance on GPUs has delayed many of their short-term plans.

OpenAI received the most user complaints about the reliability and speed of the API. Altman understood their concerns and explained that much of the problem was caused by a shortage of GPU resources.

The 32k tokens context function previously supported by ChatGPT cannot be extended to more people, and OpenAI still has some problems to solve, so although they may soon have a 100k-1M tokens context window, they still need to be obtained in research breakthrough.

The fine-tuning API is also currently limited by GPU resources. They have not yet used efficient fine-tuning methods like Adapters or LoRa (two common fine-tuning methods for large models), so fine-tuning operation and management is very computationally resource intensive. There will be better ways of fine-tuning in the future. They might even host a marketplace for community contributed models.

Dedicated capacity provisioning is also limited by GPU resources. OpenAI provides dedicated capacity for customers with private needs, allowing customers to run private data in a dedicated space. To access this service, customers need to commit to an advance of $100,000.

OpenAI Roadmap

Sam shared a tentative short-term roadmap for OpenAI's API.

In 2023, the first task is to achieve cheaper and faster GPT-4; second, a longer context window - in the near future, the context window may reach up to 1 million tokens; third, the fine-tuning API will expand To the latest model, but the specific form will be determined by the actual needs of developers; fourth, a stateful API - when calling the chat API today, it is necessary to repeatedly pass the same conversation history and pay the same tokens repeatedly. In the future, there will be a API version that remembers conversation history.

In 2024, multi-modal capabilities will be opened. When GPT-4 was released, it demonstrated powerful multimodal capabilities, but until GPUs are satisfied, this capability cannot be extended to everyone.

Many developers are interested in accessing ChatGPT plugins through the API, but Altman does not think these plugins will be released in the short term. In addition to browsing, the plug-in system has not yet found PMF (Product Market Fit, the best fit between product and market).

Altman pointed out that a lot of people think they want their app to be inside ChatGPT, but what they really want is ChatGPT within the app.

Altman said that OpenAI will not release more products than ChatGPT.

He said that by convention, a great platform will have a killer app, and ChatGPT is going to make this record-breaking app. ChatGPT's vision is to be a super-smart assistant for work, but there are many other GPT use cases that OpenAI won't touch.

Scaling laws still apply

While Altman calls for regulation of the future of AI, he doesn't think the current model is dangerous and thinks it would be a big mistake to regulate or ban it.

He reiterated his belief in the importance of open source and said that OpenAI is considering making GPT-3 open source. The reason why it hasn't been open source is because they feel that not many people and companies have the ability to properly manage such a large language model.

Many recent articles have quoted Altman saying that "the era of giant AI models is over," but this does not accurately reflect his original meaning. He said that OpenAI's internal data shows that the scaling laws of model performance (Scaling laws) are still in effect, making the model larger will continue to produce performance. The scaling rate is indeed unsustainable, as OpenAI has scaled up the model by millions of times in just a few years, and continuing to do so in the future is not sustainable. That doesn't mean OpenAI won't keep trying to make models bigger, it just means they'll probably only double or triple every year, rather than many orders of magnitude.

The fact that scaling laws continue to hold has important implications for AGI (artificial general intelligence) development timelines, Altman said. The scaling law assumes that we probably already have most of the parts needed to build AGI, and that most of the work left will be scaling up existing methods to larger models and larger datasets. If the era of scaling laws is over, we should probably expect AGI to be a long way off. The fact that scaling laws continue to work strongly suggests a short timeline for achieving AGI.