Leaked Google Notes admit defeat by open-source AI

A leaked Google memo provides a point-by-point summary of why Google is losing out to open-source AI, and suggests a path back to dominance and platform ownership.

The note begins by acknowledging that competitors will always be open source, not OpenAI.

Can’t compete with open source

Furthermore, they admit that they are not in a position to compete with open source and that they have already lost the battle for AI supremacy.

they wrote:

“We have been looking a lot into OpenAI. Who will cross the next milestone? What will be the next move?

But the uncomfortable truth is that we are not in a position to win this arms race, and neither is OpenAI. While we were arguing, a third faction quietly ate our lunch.

Of course, I’m talking about open source.

To put it bluntly, they are licking us. What we consider to be the ‘major unsolved problems’ have been resolved and are in the hands of the people today. ”

Most of the memo is devoted to explaining how Google is being beaten by open source.

And while Google has a slight edge over open source, the memo’s authors admit that Google is disappearing and never coming back.

The metaphorical card self-analysis they dealt is rather pessimistic.

“Our models still have a slight advantage in terms of quality, but the gap is closing surprisingly quickly.

The open source model is faster, more customizable, more private, and has better features.

They’re doing something with the $100 and $13 billion parameters that we’re struggling with at $10 million and $540 billion.

And they’re doing it in weeks, not months. ”

Large language model size is not an advantage

Perhaps the most chilling realization expressed in the memo is that Google’s size is no longer an advantage.

The ridiculously large size of their models is now seen as a drawback, never the insurmountable advantage they thought it was.

A leaked memo lists a series of events that indicate a possible rapid end to Google’s (and OpenAI’s) control of AI.

I recall that just a month ago, in March 2023, the open source community got hold of a large language model of a leaked open source model developed by Meta called LLaMA.

Within days and weeks, the global open source community developed all the building blocks needed to clone Bard and ChatGPT.

Sophisticated procedures such as instruction tuning and reinforcement learning from human feedback (RLHF) were cheaply and quickly replicated by the global open-source community.

imperative tuning The process of fine-tuning a language model to do specific things that it was not trained to do in the first place. Reinforcement learning from human feedback (RLHF) A technique in which a human evaluates the output of a language model to learn which output is satisfactory to the human.

RLHF is the underlying model of ChatGPT and the technique used by OpenAI to create InstructGPT that allows GPT-3.5 and GPT-4 models to receive instructions and complete tasks.

RLHF is the flame that open source stole

Open source scale threatens Google

What particularly scares Google is the fact that the open source movement can scale projects in ways that closed source can’t.

The question and answer dataset used to create Dolly 2.0, an open source ChatGPT clone, was created entirely by thousands of employee volunteers.

Google and OpenAI relied in part on questions and answers collected from sites such as Reddit.

The open source Q&A dataset created by Databricks is claimed to be of higher quality. This is because the humans who contributed to its creation are experts, and the answers they provided are longer and richer than those found in the general question-and-answer dataset. public forum.

The leaked memo states:

“In early March, Meta’s LLaMA was launched, giving the open source community its first really capable underlying model.

There was no direction or coordination of speech, no RLHF.

Nevertheless, the community quickly understood the importance of what they were given.

A tremendous spurt of innovation followed, with just a few days between major developments…

Just a month later, there are variants such as instruction tuning, quantization, quality enhancement, human evaluation, multimodality, RLHF, and many of them build upon each other.

Most importantly, they solved the scaling problem in a foolproof way.

Many of the new ideas come from ordinary people.

Barriers to entry for training and experimentation have been lowered from the gross output of major research organizations to one person, one night, and a rugged laptop. ”

In other words, what took months and years to train and build for Google and OpenAI took only days for the open source community.

This must be a really scary scenario for Google.

This is one of the reasons I’ve been writing so much about the open source AI movement, as it really shows what the future of generative AI will look like in a relatively short time.

Open source has historically outperformed closed source

This memo cites OpenAI’s recent experience with DALL-E. DALL-E is a deep learning model used to create images compared to open-source Stable Diffusion, a precursor to what is currently befalling generative AI like Bard and ChatGPT.

Dall-e was released by OpenAI in January 2021. An open-source version, Stable Diffusion, was released a year and a half later in August 2022, surpassing Dall-E’s popularity in just a few weeks.

This timeline graph shows how quickly Stable Diffusion overtook Dall-E.

The Google Trends timeline above shows that interest in the open source Stable Diffusion model has significantly surpassed that of Dall-E just three weeks after its release.

A year and a half after Dall-E’s release, interest in Stable Diffusion continued to grow exponentially, and OpenAI’s Dall-E remained stagnant.

The existential threat that similar events overtake Bard (and OpenAI) is a nightmare for Google.

Excellent open source model creation process

Another factor that worries Google engineers is that the process of creating and improving the open source model is quick, cheap, and perfectly suited to the global collaborative approach common to open source projects.

In this memo, new techniques such as LoRA (Low-Rank Adaptation of Large Language Models) enable fine-tuning of language models in days at very low cost, and the final LLM is reduced to a very expensive LLM. You will find that they are comparable. Created by Google and OpenAI.

Another benefit is that open source engineers can build on and iterate on previous work rather than starting from scratch.

Building large language models with billions of parameters the way OpenAI and Google have done is not necessary today.

This may be the point Sam Alton was alluding to when he recently said the days of large language models are over.

The author of a Google note contrasted the cheap and fast LoRA approach to creating LLM with the current Big AI approach.

The author of the memo describes Google’s shortcomings as follows:

“By contrast, training a huge model from scratch discards not only the pre-training, but also the iterative improvements made on top of it. It doesn’t take much time, so full retraining is very expensive.

Each new application or idea requires careful consideration as to whether it really calls for an entirely new model.

…In fact, in terms of engineer time, the pace of improvement from these models has greatly outpaced what the largest variations can do, and the best are already almost indistinguishable from ChatGPT. ”

The authors conclude by recognizing that what they thought was an advantage, the huge model and attendant exorbitant cost was actually a disadvantage.

The global collaborative nature of open source is more efficient and speeds innovation by orders of magnitude.

How can a closed source system compete with the vast majority of engineers in the world?

The authors conclude that they cannot compete and that direct competition is, in their words, a “losing proposition.”

That’s the crisis, the storm, and it’s happening outside of Google.

If you can’t beat open source, join us

The only consolation the author of this memo finds in open source is that open source innovation is free, so Google can take advantage of it.

Finally, the authors conclude that the only approach open to Google is to own the platform in the same way it dominates the open-source Chrome and Android platforms.

They point out how Meta has benefited from releasing the LLaMA large language model for research, and how thousands of people are currently doing the work for free.

Perhaps the key takeaway from this memo is that Google may in the near future seek to replicate the advantages of open source by releasing projects on an open source basis and thereby owning the platform. That’s it.

This memo concludes that moving to open source is the most viable option.

“Google should establish itself as a leader in the open source community and lead the wider debate by collaborating rather than ignoring it.

This probably means taking some unpleasant steps, such as exposing model weights for small ULM variants. This necessarily means giving up some control over our model.

However, this compromise is inevitable.

We cannot expect to both drive and control innovation. ”

Open source leaves with AI fire

Last week, we pitted Prometheus open source against Google and OpenAI’s “Gods of Olympus,” referring to the Greek myth that the human hero Prometheus stole fire from the gods of Mount Olympus.

I tweeted:

“While Google, Microsoft and Open AI are bickering and turning against each other, will open source put out their fire?”

The leaked Google memo confirms that observation, but also points to a possible change in strategy for Google to join the open source movement, embrace and dominate it in the same way it did with Chrome and Android. doing.

Read the leaked Google note here:

Google: “We have no moat, no OpenAI”

Sources 1/ https://Google.com/ 2/ https://www.searchenginejournal.com/leaked-google-memo-admits-defeat-by-open-source-ai/486290/ The mention sources can contact us to remove/changing this article

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

LOS ANGELES, CA / ACCESSWIRE / June 24, 2020, / Compare-autoinsurance.Org has launched a new blog post that presents the main benefits of comparing multiple car insurance quotes. For more info and free online quotes, please visit https://compare-autoinsurance.Org/the-advantages-of-comparing-prices-with-car-insurance-quotes-online/ The modern society has numerous technological advantages. One important advantage is the speed at which information is sent and received. With the help of the internet, the shopping habits of many persons have drastically changed. The car insurance industry hasn't remained untouched by these changes. On the internet, drivers can compare insurance prices and find out which sellers have the best offers. View photos The advantages of comparing online car insurance quotes are the following: Online quotes can be obtained from anywhere and at any time. Unlike physical insurance agencies, websites don't have a specific schedule and they are available at any time. Drivers that have busy working schedules, can compare quotes from anywhere and at any time, even at midnight. Multiple choices. Almost all insurance providers, no matter if they are well-known brands or just local insurers, have an online presence. Online quotes will allow policyholders the chance to discover multiple insurance companies and check their prices. Drivers are no longer required to get quotes from just a few known insurance companies. Also, local and regional insurers can provide lower insurance rates for the same services. Accurate insurance estimates. Online quotes can only be accurate if the customers provide accurate and real info about their car models and driving history. Lying about past driving incidents can make the price estimates to be lower, but when dealing with an insurance company lying to them is useless. Usually, insurance companies will do research about a potential customer before granting him coverage. Online quotes can be sorted easily. Although drivers are recommended to not choose a policy just based on its price, drivers can easily sort quotes by insurance price. Using brokerage websites will allow drivers to get quotes from multiple insurers, thus making the comparison faster and easier. For additional info, money-saving tips, and free car insurance quotes, visit https://compare-autoinsurance.Org/ Compare-autoinsurance.Org is an online provider of life, home, health, and auto insurance quotes. This website is unique because it does not simply stick to one kind of insurance provider, but brings the clients the best deals from many different online insurance carriers. In this way, clients have access to offers from multiple carriers all in one place: this website. On this site, customers have access to quotes for insurance plans from various agencies, such as local or nationwide agencies, brand names insurance companies, etc. "Online quotes can easily help drivers obtain better car insurance deals. All they have to do is to complete an online form with accurate and real info, then compare prices", said Russell Rabichev, Marketing Director of Internet Marketing Company. CONTACT: Company Name: Internet Marketing CompanyPerson for contact Name: Gurgu CPhone Number: (818) 359-3898Email: [email protected]: https://compare-autoinsurance.Org/ SOURCE: Compare-autoinsurance.Org View source version on accesswire.Com:https://www.Accesswire.Com/595055/What-Are-The-Main-Benefits-Of-Comparing-Car-Insurance-Quotes-Online View photos