The AI ​​Innovation Race: Does It Matter If Your Computer Is Bigger Than Mine?

As AI requires vast data lakes for training, it is assumed that leadership in AI R&D requires huge and growing computational capacity.

Major economies are engaged in an IT arms race. The United States is spending $500 million to build an exascale computer that will double the capacity of its largest existing computer (exascale = 1,000,000,000,000,000,000 operations per second). The European Union has commissioned the European High Performance Computing Joint Undertaking (EuroHPC JU) with an exascale computer in Germany. China alone accounts for 12% of the world’s supercomputer capacity.

The U.S. National Security Commission has concluded that due to the computing and data requirements of advanced AI systems, U.S. AI development is concentrated in fewer organizations in fewer geographic regions. pursuing fewer avenues of research.

What hope then for small economies that cannot afford to invest in massive computing capacities?

What hope then for small economies that cannot afford to invest in massive computing capacities? A recent Australian government document commented that:

Australia has capacity in AI-related areas like computer vision and robotics, and social and governance aspects of AI, but its core core capacity in LLMs and related areas is relatively low. weak. While the Australian government has announced investments of $100 million in AI-related initiatives (for example, including a National AI Centre), the creation of generative AI technologies presents barriers to access particularly high, due to its huge computational and data requirements.

But a recent study from the Georgetown Universitys Center for Security and Emerging Technology (CSET) suggests a more nuanced picture of the success of AI R&D.

CSET undertook its study because there is no comprehensive data available on the use of computation among AI researchers. CSET interviewed AI researchers identified through leading AI journals and LinkedIn entries. Responses came from a wide range of researchers in the public and private sectors: of the 410 responses, 67% said they worked in academia and 29% in industry. Among respondents who said they worked in manufacturing, 70% said they worked for a company with more than 500 employees, while 30% said they worked for a company with 500 employees or less.

Computation is not the main constraint for many AI researchers

Respondents were asked to identify their projects over the past 5 years that contributed the most to research in their field and the projects that consumed the most computing resources. More than two-thirds said the two projects were identical. But when asked what made the project successful, 90% rated specialized knowledge, talents or skills, 52% rated large amounts of computing, and 51% rated access to data. unique.

To check the relative importance of the factors, the researchers were asked to imagine if the budget of their current or most recent AI project doubled, what would be their first priority for spending the extra money? About half (52%) said they would spend more to hire more talent, while only about a fifth of researchers would make buying more or better compute their top priority, and a similar share would prioritize data collection or cleaning.

As a third cross-check, respondents were asked the reasons why they had refused, abandoned or revised AI projects in the past 2 years. Researchers report rejecting and abandoning projects due to a lack of data or availability of researchers more often than due to a lack of computing resources. But when it comes to the reason for revising projects, more than three-quarters of researchers identified issues with accessing sufficient computing power.

Computation less important in the future for AI progress

Respondents were asked, over the past 10 years, which of five factors – data, computation, algorithms, number of researchers and level of support for AI projects was the most important contributor to progress in AI. They were then asked the same question for the next 10 years.

Looking back, 59% of respondents believed that advances in AI were primarily driven by computing power. But going forward, only 40% saw computing power as the main driver, with the focus on algorithms.

While reviewing these results, CSET cautions against the risk of a degree of self-interest and self-importance among researchers:

Predictions that the importance of algorithms will increase while the importance of computation will decrease might be a reflection of researchers’ own interests rather than a developing trend. Researchers tend to regard the advances that come from the brute-force approach of simply using more computation as less interesting and less valuable than the new approaches that require their knowledge and creativity. That more computation often outweighs more ingenuity has been a bitter lesson that perhaps has not yet been fully internalized by the research community.

Public Sector vs. Private Sector Computing

One of the dreaded outcomes of the need for massive computing capacity to support AI is that Big Tech will be better positioned than the public sector, even some countries, to develop this computing capacity.

However, the CSET study shows a less gloomy picture:

• while academic reports of AI spending on computing in their projects are much lower than those of commercial researchers, the hours of computing capacity per project are very similar between the public and private sectors;
• when AI academics were asked if they had thought about moving to the private sector and why, only 35% ranked access to better computing resources as an important deciding factor, far behind the 70% who identified a better pay.
• AI academics express no more concern than commercial AI researchers about the level of access they will have to the computing resources they will need in the future. On the contrary, current heavy users of computing resources, whether academic or commercial, fear that they will have to share more access in the future as the popularity of AI research grows.

start-up

If AI R&D depended on access to large-scale computing, one would expect start-ups to worry about the asymmetric advantages that Big Tech has. However, access to talent was still the top priority for the start-ups surveyed. Interestingly, the start-ups also attributed the project’s success much more to access to data than computing power.

Where governments should invest their AI research funds

The CSET report notes that the U.S. government’s decisions to invest massive sums in computing capacity are driven by the well-intentioned assumption that providing large amounts of computing resources to more researchers could democratize research on AI by allowing researchers with few resources to compete with better ones – those with resources. However, CRET warns that this policy could prove counterproductive:

Such a strategy could actually backfire on you, leading to even greater stratification of the differences in compute usage. Across a large number of metrics, we found that the researchers most eager to get more compute were the same ones who were already using more compute than their peers. this in turn suggests that if more computation were made available to researchers at all levels, it could primarily benefit compute-intensive users, without becoming a major resource for researchers who currently use less computation.

Perhaps more cynically, politicians love groundbreaking events, which makes investing public money in shiny hardware more appealing than investing in less tangible projects like talent growth or access. to better datasets to train AI more accurately and ethically.

Conclusion

The CSET study suggests there is still room for small and medium economies to make strides in AI innovation with a more nuanced and mixed set of policy initiatives. The study concludes that:

Computing cannot be considered an all-purpose lever to promote progress in AI. Advances in AI research.

The punchline is that main resource [in AI R&D] is human

Learn more:The primary resource is people: A survey of AI researchers on the importance of computation