OpenAI managed to steal Google’s attention in the weeks leading up to Google’s biggest event of the year (Google I/O). When the big announcement came there, all they had to show was a slightly better language model than the previous one with the “magic” part not even in Alpha testing.
OpenAI may have made users feel like a mom getting a vacuum cleaner for Mother’s Day, but it sure managed to minimize press attention for Google’s big event.
The letter O
The first hint that there is at least some trolling going on is the name of the new GPT model, 4 “o” with the letter “o” as in the name of the Google event, I/O.
OpenAI says that the letter O stands for Omni, which means everything, but there seems to be a subtext to that choice.
GPT-4o oversold as magic
Sam Altman in a tweet on the Friday before the announcement he promised “new stuff” that he found “magical”:
“No gpt-5, it’s not a search engine, but we’ve been working hard on some new things that we think people will love! It seems like magic to me.”
Greg Brockman, co-founder of OpenAI he tweeted:
“Introducing GPT-4o, our new model that can reason over text, audio, and video in real time.
It’s extremely versatile, fun to play, and a step toward a much more natural form of human-computer interaction (and even human-computer-computer interaction): “
The announcement itself explained that previous versions of ChatGPT used three models to process audio input. A model for converting audio input to text. Another model to complete the task and output the text version and a third model to convert the text output to audio. The breakthrough of GPT-4o is that it can now process audio input and output within a single model and output it all in the same amount of time it takes a human to listen and answer a question.
But the problem is that the audio part is not online yet. They are still working on getting the rails to work and it will be weeks before an Alpha version is released for a few users to try. Alpha releases are expected to possibly have bugs, while beta releases are generally closer to final products.
Here’s how OpenAI explained the disappointing delay:
“We recognize that GPT-4o’s audio modalities present a variety of new risks. Today we’re releasing text and image inputs and text outputs. Over the coming weeks and months, we’ll be working on the technical infrastructure, usability through further training and the security necessary to release the other modalities.
The most important part of GPT-4o, the audio input and output, is finished, but the security level is not yet ready for public release.
Some disappointed users
It is inevitable that an incomplete and oversold product will generate some negative sentiment on social media.
AI Engineer Maziyar Panahi (LinkedIn profile) he tweeted his disappointment:
“I’ve been testing the new GPT-4o (Omni) on ChatGPT. Not impressed! Not one bit! Faster, cheaper, multi-modal, these are not for me.
Interpreter, that’s all I care about and it’s just as lazy as before!”
He continued with:
“I understand that for startups and companies the cheapest, fastest, audio, etc. they are very attractive. But I only use chat, and it looks almost the same there. At least for the data analysis wizard.
Also, I don’t think I’m getting anything else for my $20. Not today!”
There are others on Facebook and X who expressed similar sentiments, although many others were happy with what they saw as an improvement in the speed and cost of using the API.
OpenAI oversold GPT-4o?
Given that GPT-4o is in an unfinished state, it’s hard not to lose the impression that the release was timed to coincide with and hurt Google’s I/O. Launching it on the eve of Google’s big day with a half-finished product may have inadvertently created the impression that GPT-4o in its current state is a minor iterative improvement.
In its current state it is not a revolutionary step forward, but once the audio part of the model leaves the Alpha testing stage and moves into the Beta testing stage, we can start talking about revolutions in the model big language But by the time that happens Google and Anthropic may have already put a flag on that mountain.
OpenAI’s announcement paints a lackluster picture of the new model, promoting performance on par with the GPT-4 Turbo. The only bright spots are the significant improvements in languages other than English and for API users.
OpenAI explains:
“It matches the performance of GPT-4 Turbo in English text and code, with a significant improvement in non-English language text, while being much faster and 50% cheaper in the API.”
Below are ratings for six benchmarks showing that GPT-4o barely passes GPT-4T in most tests, but falls behind GPT-4T in one important benchmark for understanding reader
Here are the scores:
MMLU (Massive Multitasking Language Understanding)
This is a benchmark for multitasking and problem-solving accuracy in over fifty subjects including math, science, history, and law. GPT-4o (88.7 score) is slightly ahead of GPT4 Turbo (86.9).
GPQA (Grade Level Google Test Q&A Benchmark)
These are 448 multiple choice questions written by human experts in various fields such as biology, chemistry and physics. GPT-4o scored 53.6, slightly beating GPT-4T (48.0).
Maths
GPT 4o (76.6) beats GPT-4T by four points (72.6).
HumanEval
This is the encoding benchmark. GPT-4o (90.2) slightly beats GPT-4T (87.1) by about three points.
MGSM (Mathematics Referent of the Multilingual Primary School)
This tests LLM primary level maths skills in ten different languages. GPT-4o scores 90.5 compared to 88.5 for GPT-4T.
DROP (discrete reasoning over paragraphs)
This is a benchmark consisting of 96,000 questions that tests the understanding of the language model on the content of the paragraphs. GPT-4o (83.4) scores almost three points lower than GPT-4T (86.0).
Did OpenAI troll Google with GPT-4o?
Given the provocative model with the letter o, it’s hard not to consider that OpenAI is trying to steal the media attention ahead of Google’s important I/O conference. Whether that was the intention or not, OpenAI managed to minimize the attention it received at Google’s upcoming search conference.
Is a language model that barely surpasses its predecessor worth all the glamor and media attention it received? The pending announcement dominated news coverage of Google’s big event, so for OpenAI the answer is clearly yes, it was worth the hype.
Featured image by Shutterstock/BeataGFX
[ad_2]
Source link
