Connect with us

Tech

Google AI Launches Crowdsourcing Harmful Test Set (CATS4ML) Data Challenge for Machine Learning

Avatar

Published

on



Source: https: //ai.googleblog.com/2021/02/uncovering-unknown-unknowns-in-machine.html

Google AI researchers have recently launched a crowdsourcing harmful test set (CATS4ML) data challenge for machine learning. This task focuses on improving machine learning (ML) evaluation datasets by encouraging them to explore existing ML benchmarks for unknown and potentially unfavorable cases.

The performance of machine learning (ML) models is highly dependent on the learning algorithm and the data used for training and evaluation. Researchers around the world are working to improve their data, including a series of workshops on the issue of ML evaluation. However, research and challenges focused on the data used to evaluate ML models are not common.

In addition, many evaluation datasets contain items that are easy to evaluate. Therefore, they overlook the natural ambiguity of the real-world context. Evaluating ML models without real-world examples makes it difficult to reliably test machine learning performance. This can lead to weaknesses in the ML model.

Google AI’s CATS4ML data challenge at HCOMP2020 addresses the difficulty of identifying weaknesses in these ML models. The main purpose of this task is to raise the bar for ML evaluation sets and find new examples of data that machine learning is confident in but is actually misclassifying. The results of this challenge help detect and avoid future errors and provide insight into the explainability of the model.

What are the weaknesses of machine learning models?

Weak Spot is a class of examples where it is difficult or impossible for a model to evaluate accurately. This is because the evaluation dataset does not contain examples of these classes.

These fall into two categories:

Known unknowns: examples where the ML model is uncertain about the correct classification Unknown unknowns: examples where the ML model is confident in the answer but is actually wrong.

Researchers continue to study known unknowns in an area called active learning. The community has found a solution to interactively get new labels from people with uncertain examples. For example, if the model does not know if the subject in the photo is a cat, the person is instructed to confirm it. However, if the system is reliable, the person will not be asked for confirmation. In this case, the reliability of the model correlates with its performance. That is, I know what the model doesn’t know.

Some efforts have also been made to discover the unknown, which has helped reveal the behavior of many unintended machines. In contrast to the method used to find the unknown unknown, Generative Adversarial Networks (GAN) produces the unknown unknown in the image recognition model in the form of an optical illusion. This causes deep learning models to make mistakes that go beyond human perception.

However, real-world examples can provide better insights into model failures in day-to-day performance. Therefore, the CATS4ML Data Challenge aims to collect unmanipulated samples that humans can reliably interpret, but many ML models make mistakes.

cats4ml-2021

First edition of CATS4ML Data Challenge

The first edition of the CATS4ML Data Challenge focuses on visual recognition using images and labels in the Open Images Dataset. Participants can select a target image from the Open Images dataset and select a set of 24 target labels from the same dataset. ML researchers and practitioners are unique in exploring existing publicly available datasets and discovering examples of unknown unknowns in machine learning models, focusing on preselected target label lists. You will be asked to invent a creative method. This also allows researchers to create more balanced, diverse and socially recognized benchmark datasets for ML.

The challenge is open to researchers and developers around the world until April 30, 2021. Those who want to participate in the challenge can register on the Challenge website, download the target image and labeled dataset, and post the photos they find.

Source: https: //ai.googleblog.com/2021/02/uncovering-unknown-unknowns-in-machine.html

Challenge website: https: //cats4ml.humancomputation.com/

Suggestion

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

LOS ANGELES, CA / ACCESSWIRE / June 24, 2020, / Compare-autoinsurance.Org has launched a new blog post that presents the main benefits of comparing multiple car insurance quotes. For more info and free online quotes, please visit https://compare-autoinsurance.Org/the-advantages-of-comparing-prices-with-car-insurance-quotes-online/ The modern society has numerous technological advantages. One important advantage is the speed at which information is sent and received. With the help of the internet, the shopping habits of many persons have drastically changed. The car insurance industry hasn't remained untouched by these changes. On the internet, drivers can compare insurance prices and find out which sellers have the best offers. View photos The advantages of comparing online car insurance quotes are the following: Online quotes can be obtained from anywhere and at any time. Unlike physical insurance agencies, websites don't have a specific schedule and they are available at any time. Drivers that have busy working schedules, can compare quotes from anywhere and at any time, even at midnight. Multiple choices. Almost all insurance providers, no matter if they are well-known brands or just local insurers, have an online presence. Online quotes will allow policyholders the chance to discover multiple insurance companies and check their prices. Drivers are no longer required to get quotes from just a few known insurance companies. Also, local and regional insurers can provide lower insurance rates for the same services. Accurate insurance estimates. Online quotes can only be accurate if the customers provide accurate and real info about their car models and driving history. Lying about past driving incidents can make the price estimates to be lower, but when dealing with an insurance company lying to them is useless. Usually, insurance companies will do research about a potential customer before granting him coverage. Online quotes can be sorted easily. Although drivers are recommended to not choose a policy just based on its price, drivers can easily sort quotes by insurance price. Using brokerage websites will allow drivers to get quotes from multiple insurers, thus making the comparison faster and easier. For additional info, money-saving tips, and free car insurance quotes, visit https://compare-autoinsurance.Org/ Compare-autoinsurance.Org is an online provider of life, home, health, and auto insurance quotes. This website is unique because it does not simply stick to one kind of insurance provider, but brings the clients the best deals from many different online insurance carriers. In this way, clients have access to offers from multiple carriers all in one place: this website. On this site, customers have access to quotes for insurance plans from various agencies, such as local or nationwide agencies, brand names insurance companies, etc. "Online quotes can easily help drivers obtain better car insurance deals. All they have to do is to complete an online form with accurate and real info, then compare prices", said Russell Rabichev, Marketing Director of Internet Marketing Company. CONTACT: Company Name: Internet Marketing CompanyPerson for contact Name: Gurgu CPhone Number: (818) 359-3898Email: [email protected]: https://compare-autoinsurance.Org/ SOURCE: Compare-autoinsurance.Org View source version on accesswire.Com:https://www.Accesswire.Com/595055/What-Are-The-Main-Benefits-Of-Comparing-Car-Insurance-Quotes-Online View photos



picture credit

ExBUlletin

to request, modification Contact us at Here or [email protected]