Google DeepMind unveils 'superhuman' AI system for fact-checking, reducing costs and improving accuracy

Join us in Atlanta on April 10th to explore the landscape of security workers. Explore the vision, benefits, and use cases of AI for security teams. Request an invitation here.

A new study from Google's DeepMind research division finds that artificial intelligence systems can outperform human fact-checkers when evaluating the accuracy of information produced by large-scale language models.

The paper, entitled “Long Sentence Factuality in Large-Scale Language Models” and published on the preprint server arXiv, introduces a technique called Search Augmented Factuality Evaluation (SAFE). SAFE uses a large-scale language model to break down the generated text into individual facts and uses Google search results to determine the accuracy of each claim.

SAFE leverages LLM to decompose a long-form response into a set of discrete facts, a multi-step inference process that includes submitting a search query to Google Search and determining whether a fact is supported by the search. Use to evaluate the accuracy of each fact. The authors explained the results.

Superhuman performance sparks debate

Researchers compared SAFE to human annotators on a dataset of approximately 16,000 facts and found that SAFE's ratings matched human ratings 72% of the time. Even more remarkable, in his sample of 100 disagreements between SAFE and human raters, he found SAFE to be correct in 76% of cases.

VB event

AI Impact Tour Atlanta

Continuing the tour, he headed to Atlanta on April 10th to join the AI ​​Impact Tour. This special, invite-only event in partnership with Microsoft will feature a discussion on how generative AI is transforming the security workforce. Space is limited, so request your invitation now.

request an invitation

Although this paper claims that LLM agents can achieve superhuman evaluation performance, some experts question the meaning of superhuman here.

A cursory reading doesn't tell you much about the human subject, but by superhuman you seem to mean true human fact-checkers rather than low-paid crowdworkers, right? That makes the characterization misleading. (That's like saying 1985's chess software was superhuman).

Gary Marcus (@GaryMarcus) March 28, 2024

Gary Marcus, a prominent AI researcher and frequent critic of exaggerated claims, says superhuman in this case simply means better than poorly paid crowdworkers; suggested on Twitter that it might mean a human fact-checker.

That characterization is therefore misleading, he says. Chess in 1985 is like saying he software was superhuman.

Marcus makes a valid point. To truly demonstrate superhuman performance, her SAFE must be benchmarked against expert human fact-checkers, not just crowdsourced employees. The specific details of the human evaluator, such as qualifications, compensation, and fact-checking process, are critical to properly contextualizing the results.

Cost reduction and benchmarking of top models

One obvious advantage of SAFE is cost. Researchers found that using AI systems is about 20 times cheaper than human fact-checkers. As the amount of information produced by language models continues to explode, it becomes increasingly important to have an economical and scalable way to verify claims.

The DeepMind team used SAFE to evaluate the factual accuracy of 13 top language models across four families (Gemini, GPT, Claude, and PaLM-2) in a new benchmark called LongFact. The results show that larger models generally have fewer factual errors.

However, even the best-performing models produced a significant number of false claims. This highlights the risks of over-reliance on language models that can fluently represent inaccurate information. Automated fact-checking tools like SAFE can play an important role in mitigating these risks.

Transparency and human baselines matter

The SAFE code and LongFact dataset are open sourced on GitHub to allow other researchers to review and build upon the study, but further information regarding the human baseline used in the study is required. Transparency is still needed. Understanding the details of a crowdworker's background and process is essential to assessing the capabilities of her SAFE in the appropriate context.

As tech giants race to develop ever more powerful language models for applications ranging from search to virtual assistants, the ability to automatically fact-check the output of these systems proves crucial. there's a possibility that. Tools like SAFE are an important step in building new layers of trust and accountability.

However, it is important that the development of such important technologies occurs openly, with input from a wide range of stakeholders, beyond the walls of a single company. Rigorous and transparent benchmarking of human experts, not just crowdworkers, is essential to measure true progress. Only then can we assess the real impact that automated fact-checking has on fighting misinformation.

VB Daily

Be sure to stay informed.Get the latest news in your inbox every day

By subscribing, you agree to VentureBeat's Terms of Use.

Thank you for subscribing. Check out his other VB newsletters here.

An error has occurred.

Sources 1/ https://Google.com/ 2/ https://venturebeat.com/ai/google-deepmind-unveils-superhuman-ai-system-that-excels-in-fact-checking-saving-costs-and-improving-accuracy/ The mention sources can contact us to remove/changing this article

What Are The Main Benefits Of Comparing Car Insurance Quotes Online

LOS ANGELES, CA / ACCESSWIRE / June 24, 2020, / Compare-autoinsurance.Org has launched a new blog post that presents the main benefits of comparing multiple car insurance quotes. For more info and free online quotes, please visit https://compare-autoinsurance.Org/the-advantages-of-comparing-prices-with-car-insurance-quotes-online/ The modern society has numerous technological advantages. One important advantage is the speed at which information is sent and received. With the help of the internet, the shopping habits of many persons have drastically changed. The car insurance industry hasn't remained untouched by these changes. On the internet, drivers can compare insurance prices and find out which sellers have the best offers. View photos The advantages of comparing online car insurance quotes are the following: Online quotes can be obtained from anywhere and at any time. Unlike physical insurance agencies, websites don't have a specific schedule and they are available at any time. Drivers that have busy working schedules, can compare quotes from anywhere and at any time, even at midnight. Multiple choices. Almost all insurance providers, no matter if they are well-known brands or just local insurers, have an online presence. Online quotes will allow policyholders the chance to discover multiple insurance companies and check their prices. Drivers are no longer required to get quotes from just a few known insurance companies. Also, local and regional insurers can provide lower insurance rates for the same services. Accurate insurance estimates. Online quotes can only be accurate if the customers provide accurate and real info about their car models and driving history. Lying about past driving incidents can make the price estimates to be lower, but when dealing with an insurance company lying to them is useless. Usually, insurance companies will do research about a potential customer before granting him coverage. Online quotes can be sorted easily. Although drivers are recommended to not choose a policy just based on its price, drivers can easily sort quotes by insurance price. Using brokerage websites will allow drivers to get quotes from multiple insurers, thus making the comparison faster and easier. For additional info, money-saving tips, and free car insurance quotes, visit https://compare-autoinsurance.Org/ Compare-autoinsurance.Org is an online provider of life, home, health, and auto insurance quotes. This website is unique because it does not simply stick to one kind of insurance provider, but brings the clients the best deals from many different online insurance carriers. In this way, clients have access to offers from multiple carriers all in one place: this website. On this site, customers have access to quotes for insurance plans from various agencies, such as local or nationwide agencies, brand names insurance companies, etc. "Online quotes can easily help drivers obtain better car insurance deals. All they have to do is to complete an online form with accurate and real info, then compare prices", said Russell Rabichev, Marketing Director of Internet Marketing Company. CONTACT: Company Name: Internet Marketing CompanyPerson for contact Name: Gurgu CPhone Number: (818) 359-3898Email: [email protected]: https://compare-autoinsurance.Org/ SOURCE: Compare-autoinsurance.Org View source version on accesswire.Com:https://www.Accesswire.Com/595055/What-Are-The-Main-Benefits-Of-Comparing-Car-Insurance-Quotes-Online View photos