Connect with us

Tech

Google trained a trillion parameter AI language model

Avatar

Published

on


An anonymous reader cites a report from VentureBeat. Google researchers have developed and benchmarked a method that claims to be able to train a language model with over a trillion parameters. They say the 1.6 trillion parameter model, which appears to be the largest in size to date, has achieved up to four times the speedup of the previous largest Google development language model (T5-XXL). Large-scale training is an effective path to a powerful model, as researchers point out in a treatise detailing their work. A simple architecture, backed by a large dataset and a large number of parameters, outperforms much more complex algorithms. However, effective and large-scale training can be very computationally intensive. That’s why researchers sought a technique called a switch transformer, a “sparsely activated” technique that uses only a subset of the model’s weights, or parameters that transform the input data in the model.

In the experiment, researchers used 32 TPU cores in Colossal Clean Crawled Corpus, a 750GB-sized dataset of text scraped from Reddit, Wikipedia, and other web sources, using several different Switch Transformer models. Was pre-trained. They impose other challenges on the model, such as predicting words that are missing 15% of the words where they are masked, and getting text to answer a list of increasingly difficult questions. did. Researchers have found that the 1.6 trillion parameter model, which includes 20,48 experts (Switch-C), is in contrast to the smaller model (Switch-XXL), which contains 395 billion parameters and 64 experts. There is no training instability. ” However, in one benchmark (Sanford Question Answering Dataset (SQuAD)), Switch-C scores are lower than Switch-XXL (89.6) (87.7), and researchers are uncertain between the quality of fine-tuning and computational requirements. I think it’s due to the relationship. And the number of parameters.

In this case, Switch Transformer has benefited from many downstream tasks. For example, researchers have shown that pre-training can be up to 7x faster while using the same amount of computational resources. Researchers have demonstrated that large sparse models can be used to create smaller, denser models that are fine-tuned with 30% of the tasks. Greater model quality improvements. In one test in which the Switch Transformer model was trained to translate between more than 100 different languages, researchers observed “universal improvements” in 101 languages, with 91% of the languages ​​compared to the baseline model. Benefited from more than four times the speedup. “This work focuses on very large models, but on models with only two experts, performance is improved while easily meeting the memory constraints of commonly available GPUs or TPUs. I also understand, “the researcher wrote in the treatise. “We cannot completely maintain the quality of the model, but we can achieve a compression ratio of 10 to 100 times by distilling the sparse model into a high density model while achieving about 30% of the quality improvement of the expert model. . “

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

LOS ANGELES, CA / ACCESSWIRE / June 24, 2020, / Compare-autoinsurance.Org has launched a new blog post that presents the main benefits of comparing multiple car insurance quotes. For more info and free online quotes, please visit https://compare-autoinsurance.Org/the-advantages-of-comparing-prices-with-car-insurance-quotes-online/ The modern society has numerous technological advantages. One important advantage is the speed at which information is sent and received. With the help of the internet, the shopping habits of many persons have drastically changed. The car insurance industry hasn't remained untouched by these changes. On the internet, drivers can compare insurance prices and find out which sellers have the best offers. View photos The advantages of comparing online car insurance quotes are the following: Online quotes can be obtained from anywhere and at any time. Unlike physical insurance agencies, websites don't have a specific schedule and they are available at any time. Drivers that have busy working schedules, can compare quotes from anywhere and at any time, even at midnight. Multiple choices. Almost all insurance providers, no matter if they are well-known brands or just local insurers, have an online presence. Online quotes will allow policyholders the chance to discover multiple insurance companies and check their prices. Drivers are no longer required to get quotes from just a few known insurance companies. Also, local and regional insurers can provide lower insurance rates for the same services. Accurate insurance estimates. Online quotes can only be accurate if the customers provide accurate and real info about their car models and driving history. Lying about past driving incidents can make the price estimates to be lower, but when dealing with an insurance company lying to them is useless. Usually, insurance companies will do research about a potential customer before granting him coverage. Online quotes can be sorted easily. Although drivers are recommended to not choose a policy just based on its price, drivers can easily sort quotes by insurance price. Using brokerage websites will allow drivers to get quotes from multiple insurers, thus making the comparison faster and easier. For additional info, money-saving tips, and free car insurance quotes, visit https://compare-autoinsurance.Org/ Compare-autoinsurance.Org is an online provider of life, home, health, and auto insurance quotes. This website is unique because it does not simply stick to one kind of insurance provider, but brings the clients the best deals from many different online insurance carriers. In this way, clients have access to offers from multiple carriers all in one place: this website. On this site, customers have access to quotes for insurance plans from various agencies, such as local or nationwide agencies, brand names insurance companies, etc. "Online quotes can easily help drivers obtain better car insurance deals. All they have to do is to complete an online form with accurate and real info, then compare prices", said Russell Rabichev, Marketing Director of Internet Marketing Company. CONTACT: Company Name: Internet Marketing CompanyPerson for contact Name: Gurgu CPhone Number: (818) 359-3898Email: [email protected]: https://compare-autoinsurance.Org/ SOURCE: Compare-autoinsurance.Org View source version on accesswire.Com:https://www.Accesswire.Com/595055/What-Are-The-Main-Benefits-Of-Comparing-Car-Insurance-Quotes-Online View photos



picture credit

ExBUlletin

to request, modification Contact us at Here or [email protected]