Google AI Introduces Performers: A Generalized Attention Framework Based on the Transformer Architecture

Image source: https: //ai.googleblog.com/2020/10/rethinking-attention-with-performers.html

The Transformer model, a deep learning framework, achieves cutting-edge results across domains such as natural language, conversation, images, and even music. The core block of the Transformer architecture is an attention module that calculates the similarity score for all pairs of positions in the input sequence. Longer input sequences are less efficient because they require quadratic computation time and quadratic memory size for the storage matrix.

Therefore, for long-distance attention, one of the most common methods is sparse attention. It reduces complexity by calculating selective similarity scores from sequences based on a variety of methods. There are still certain limitations, such as the lack of efficient sparse matrix multiplication operations on all accelerators, no theoretical guarantees, and insufficientness to address the full range of problems.

Introduction to performers

To solve these problems, Google AI has introduced the Performer, a Transformer architecture with a linearly scaling attention mechanism. The framework is implemented by fast attention via a positive orthogonal random feature (FAVOR +) algorithm, and a scalable low variance and non-bias estimation of the attention mechanism represented by random feature map decomposition (especially normal softmax attention). To provide. Mappings help maintain the complexity of linear space and time.

In the original attention mechanism, the query, the row matrix and the key, the column matrix, are multiplied and passed through the softmax operation to form the attention matrix. However, in this case, the query key product matrix cannot be decomposed into the original components after going through the softmax operation.

https://ai.googleblog.com/2020/10/rethinking-attention-with-performers.html

You can use the matrix factorization factorization to get the linear time attention mechanism. After decomposing the attention matrix, you can reposition the matrix multiplications to approximate the results of the normal attention mechanism without explicitly creating a quadratic-sized attention matrix. This provides FAVOR + (fast attention due to matrix connectivity).

Therefore, using the above algorithm, Performer comes up with the idea of ​​reducing space and time complexity and reducing energy costs by improving inference speed.

Github: https: //github.com/google-research/google-research

Paper: https: //arxiv.org/pdf/2009.14794.pdf

Relation

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

LOS ANGELES, CA / ACCESSWIRE / June 24, 2020, / Compare-autoinsurance.Org has launched a new blog post that presents the main benefits of comparing multiple car insurance quotes. For more info and free online quotes, please visit https://compare-autoinsurance.Org/the-advantages-of-comparing-prices-with-car-insurance-quotes-online/ The modern society has numerous technological advantages. One important advantage is the speed at which information is sent and received. With the help of the internet, the shopping habits of many persons have drastically changed. The car insurance industry hasn't remained untouched by these changes. On the internet, drivers can compare insurance prices and find out which sellers have the best offers. View photos The advantages of comparing online car insurance quotes are the following: Online quotes can be obtained from anywhere and at any time. Unlike physical insurance agencies, websites don't have a specific schedule and they are available at any time. Drivers that have busy working schedules, can compare quotes from anywhere and at any time, even at midnight. Multiple choices. Almost all insurance providers, no matter if they are well-known brands or just local insurers, have an online presence. Online quotes will allow policyholders the chance to discover multiple insurance companies and check their prices. Drivers are no longer required to get quotes from just a few known insurance companies. Also, local and regional insurers can provide lower insurance rates for the same services. Accurate insurance estimates. Online quotes can only be accurate if the customers provide accurate and real info about their car models and driving history. Lying about past driving incidents can make the price estimates to be lower, but when dealing with an insurance company lying to them is useless. Usually, insurance companies will do research about a potential customer before granting him coverage. Online quotes can be sorted easily. Although drivers are recommended to not choose a policy just based on its price, drivers can easily sort quotes by insurance price. Using brokerage websites will allow drivers to get quotes from multiple insurers, thus making the comparison faster and easier. For additional info, money-saving tips, and free car insurance quotes, visit https://compare-autoinsurance.Org/ Compare-autoinsurance.Org is an online provider of life, home, health, and auto insurance quotes. This website is unique because it does not simply stick to one kind of insurance provider, but brings the clients the best deals from many different online insurance carriers. In this way, clients have access to offers from multiple carriers all in one place: this website. On this site, customers have access to quotes for insurance plans from various agencies, such as local or nationwide agencies, brand names insurance companies, etc. "Online quotes can easily help drivers obtain better car insurance deals. All they have to do is to complete an online form with accurate and real info, then compare prices", said Russell Rabichev, Marketing Director of Internet Marketing Company. CONTACT: Company Name: Internet Marketing CompanyPerson for contact Name: Gurgu CPhone Number: (818) 359-3898Email: [email protected]: https://compare-autoinsurance.Org/ SOURCE: Compare-autoinsurance.Org View source version on accesswire.Com:https://www.Accesswire.Com/595055/What-Are-The-Main-Benefits-Of-Comparing-Car-Insurance-Quotes-Online View photos

picture credit