Gemini's data analytics capabilities aren't as good as Google claims

One of the selling points of Google's flagship generative AI models, Gemini 1.5 Pro and 1.5 Flash, is the amount of data they can allegedly process and analyze. During press conferences and demos, Google has repeatedly claimed that these models can achieve previously impossible tasks, like summarizing multiple documents spanning hundreds of pages or searching for scenes in movies, thanks to “long context.”

But new research suggests that the model isn't actually very good at those things.

Two separate studies looked at how well Google's Gemini model and other models could make sense of huge amounts of data the length of “War and Peace.” Both studies found that Gemini 1.5 Pro and 1.5 Flash struggled to correctly answer questions on large data sets: In a series of document-based tests, the models got the answers right only 40% to 50% of the time.

“While models like Gemini 1.5 Pro can technically handle long contexts, we have seen numerous examples that show the models don't actually 'understand' the content,” Marzena Karpinska, a postdoctoral researcher at the University of Massachusetts Amherst and co-author on one of the studies, told TechCrunch.

Gemini's context window is missing

A model's context, or context window, refers to the input data (e.g., text) that the model considers before generating an output (e.g., additional text). A simple question like “Who won the 2020 US Presidential election?” can act as context, as can a movie script, show, or audio clip. The larger the context window, the larger the size of the document that can fit in it.

The latest version of Gemini can ingest more than 2 million tokens as context. (“Tokens” are bits of raw data, like the syllables “fan,” “tas,” and “tic” in the word “fantastic.”) That's the equivalent of about 1.4 million words, 2 hours of video, or 22 hours of audio — the most context of any model on the market.

During a briefing earlier this year, Google showed off a few pre-recorded demos to showcase the potential of Gemini's long-context features, including one in which Gemini 1.5 Pro searches through roughly 402 pages of the Apollo 11 moon landing television recording for quotes containing jokes and scenes from the telecast that resembled a pencil sketch.

Oriol Viñals, vice president of research at Google DeepMind, who led the briefing, described the model as “magical.”

“[1.5 Pro] “It does these kinds of inference tasks across every page, across every word,” he said.

That may have been an exaggeration.

In one of the studies that benchmarked these capabilities, Karpinska, along with researchers from the Allen Institute for AI and Princeton University, asked the model to evaluate true and false statements about fiction books written in English. The researchers chose recent works so that the model couldn't “cheat” by relying on prior knowledge, and they peppered the statements with specific details and plot references that would be difficult to understand without reading the entire book.

If there was a statement like “Using his skills as an Apos, Nusis is able to reverse engineer the type of portal that is opened by the reagent key found in Rona's crate,” Gemini 1.5 Pros and 1.5 Flashes who had ingested the relevant book were required to state whether the statement was true or false and explain why.

Image credit: University of Massachusetts Amherst

Testing with a single book of about 260,000 words (about 520 pages), the researchers found that 1.5 Pro answered true/false questions correctly 46.7% of the time, while Flash answered only 20% of the time. This means that Coin is significantly more accurate at answering questions about books than Google's latest machine learning models. Averaging across all benchmark results, neither model was able to beat random chance in answering questions.

“We found that the model had a harder time verifying claims that required consideration of large parts of the book, or even the entire book, compared to claims that could be resolved by obtaining text-level evidence,” Karpinska said. “Qualitatively, we also observed that the model struggled to verify claims about implicit information that was obvious to a human reader but not explicitly stated in the text.”

The second of the two studies, co-authored by researchers at the University of California, Santa Barbara, tested Gemini 1.5 Flash (not 1.5 Pro)'s ability to “infer” from videos, i.e., search them to answer questions about their content.

The co-authors created a dataset that combined images (such as a photo of a birthday cake) with questions for the model to answer about objects depicted in the image (such as “What cartoon character is on this cake?”). To evaluate the model, they randomly chose one image and inserted “distracting” images before and after it to create a slideshow-like video.

Flash's performance wasn't much better: In tests where the model was asked to transcribe six handwritten digits from a 25-image “slideshow,” Flash correctly recognized about 50 percent of the transcriptions, dropping to about 30 percent accuracy at eight digits.

“The real-world question-answering task on images seems particularly difficult for all the models we tested,” Michael Saxon, a PhD student at the University of California, Santa Barbara, and one of the study's co-authors, told TechCrunch. “The subtle reasoning required to recognize that there are numbers in the frame and read them may be what breaks the models.”

Google is overpromising with Gemini

Neither study was peer reviewed, nor did they look at Gemini 1.5 Pro and 1.5 Flash releases in a 2 million token context (both tested 1 million token context releases), and Flash is not as good as Pro in terms of performance, which Google is promoting as a lower-cost alternative.

Still, the two models add to a case in which Google overpromised and then underdelivered with Gemini from the start. None of the models the researchers tested, including OpenAI's GPT-4o and Anthropic's Claude 3.5 Sonnet, performed well. Yet Google is the only model provider that promotes the context window as a top priority in its ads.

“There's nothing wrong with a simple claim that, based on objective technical details, 'our model can process X number of tokens,'” Saxon says, “but the question is, what useful thing can you do with it?”

Generative AI in general has come under increased scrutiny as companies (and investors) grow frustrated with the technology's limitations.

In two recent surveys by Boston Consulting Group, nearly half of respondents (all C-level executives) said they don't expect generative AI to deliver significant productivity gains and are concerned that generative AI-powered tools could lead to mistakes and data breaches. PitchBook recently reported that early-stage generative AI deals have declined for the second consecutive quarter, plummeting 76% from their Q3 2023 peak.

Faced with meeting-summary chatbots that conjure up fictitious details about people and AI search platforms that are essentially plagiarism generators, customers are searching for potential differentiators. As Google races, sometimes clumsily, to catch up with its generative AI rivals, it has been desperate to make Gemini's context one of those differentiators.

But it appears the gamble was premature.

“There is still no established way to actually demonstrate 'reasoning' or 'understanding' of long documents, and essentially each group releasing these models is making these claims based on their own ad-hoc evaluations,” Karpinska said. “Without knowing how long contextual processing has been implemented and the companies not sharing these details, it's hard to judge how realistic these claims are.”

Google did not respond to a request for comment.

Both Saxon and Karpinska believe the antidote to the hype around generative AI is better benchmarking and, likewise, more emphasis on third-party critique. Saxon points out that one common test of long context — the “needle in a haystack” that Google cites frequently in its marketing materials — only measures a model’s ability to retrieve specific information, such as a name or number, from a dataset, but not to answer complex questions about that information.

“All scientists and most engineers who use these models fundamentally agree that the existing benchmarking culture is broken,” Saxon says, “so it's important for the public to understand that they should take these huge reports with a pinch of salt, with numbers like 'benchmark overall general intelligence.'”

Sources 1/ https://Google.com/ 2/ https://techcrunch.com/2024/06/29/geminis-data-analyzing-abilities-arent-as-good-as-google-claims/ The mention sources can contact us to remove/changing this article

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

LOS ANGELES, CA / ACCESSWIRE / June 24, 2020, / Compare-autoinsurance.Org has launched a new blog post that presents the main benefits of comparing multiple car insurance quotes. For more info and free online quotes, please visit https://compare-autoinsurance.Org/the-advantages-of-comparing-prices-with-car-insurance-quotes-online/ The modern society has numerous technological advantages. One important advantage is the speed at which information is sent and received. With the help of the internet, the shopping habits of many persons have drastically changed. The car insurance industry hasn't remained untouched by these changes. On the internet, drivers can compare insurance prices and find out which sellers have the best offers. View photos The advantages of comparing online car insurance quotes are the following: Online quotes can be obtained from anywhere and at any time. Unlike physical insurance agencies, websites don't have a specific schedule and they are available at any time. Drivers that have busy working schedules, can compare quotes from anywhere and at any time, even at midnight. Multiple choices. Almost all insurance providers, no matter if they are well-known brands or just local insurers, have an online presence. Online quotes will allow policyholders the chance to discover multiple insurance companies and check their prices. Drivers are no longer required to get quotes from just a few known insurance companies. Also, local and regional insurers can provide lower insurance rates for the same services. Accurate insurance estimates. Online quotes can only be accurate if the customers provide accurate and real info about their car models and driving history. Lying about past driving incidents can make the price estimates to be lower, but when dealing with an insurance company lying to them is useless. Usually, insurance companies will do research about a potential customer before granting him coverage. Online quotes can be sorted easily. Although drivers are recommended to not choose a policy just based on its price, drivers can easily sort quotes by insurance price. Using brokerage websites will allow drivers to get quotes from multiple insurers, thus making the comparison faster and easier. For additional info, money-saving tips, and free car insurance quotes, visit https://compare-autoinsurance.Org/ Compare-autoinsurance.Org is an online provider of life, home, health, and auto insurance quotes. This website is unique because it does not simply stick to one kind of insurance provider, but brings the clients the best deals from many different online insurance carriers. In this way, clients have access to offers from multiple carriers all in one place: this website. On this site, customers have access to quotes for insurance plans from various agencies, such as local or nationwide agencies, brand names insurance companies, etc. "Online quotes can easily help drivers obtain better car insurance deals. All they have to do is to complete an online form with accurate and real info, then compare prices", said Russell Rabichev, Marketing Director of Internet Marketing Company. CONTACT: Company Name: Internet Marketing CompanyPerson for contact Name: Gurgu CPhone Number: (818) 359-3898Email: [email protected]: https://compare-autoinsurance.Org/ SOURCE: Compare-autoinsurance.Org View source version on accesswire.Com:https://www.Accesswire.Com/595055/What-Are-The-Main-Benefits-Of-Comparing-Car-Insurance-Quotes-Online View photos