AI Index Report-2024: AI still behind human on complex tasks like competition-level mathematics

KATHMANDU: In the wake of trepidation, optimism and bewilderment lumped together with spurt and influence of Artificial Intelligence (AI), research and studies on AI have surged lately across the globe.

In this connection, the Human-Centered Artificial Intelligence (HAI) at the Stanford University released an annual report on AI on Monday.

The seventh edition report- ‘Artificial Intelligence Index Report, 2024’ has comprehensively studied the AI trend in multiple fronts ranging from education to accountability, research to development, and economy to medicine.

One of the significant messages the report has underlined is: ‘AI beats humans on some tasks, but not on all’. There are ten takeaways in the wide-ranging report fit in 500 plus pages with extensive use of data and comparative analyses.

In the first takeaway it wrote, “AI has surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on more complex tasks like competition-level mathematics, visual commonsense reasoning and planning.”

The report is dovetailed in nine chapters focusing- research and development, technical performance, responsible AI, economy, science and medicine, education, policy and governance, diversity, and public opinion.

AI patent shifting from US to East Asia and Pacific

Taking reference from Centre for Security and Emerging Technology, 2023, the report has showed shifting AI patent. Comparing the data from 2010 to 2022, the report showed granted AI patent hugely concentrating in East Asia and Pacific – 75.2 percent, which is trailed by the North America with 21.2 percent. North America was leading the AI patent till 2011.

However, the US leads the source of top AI model in the world. In the last year, 2023, the institutions based in the US produced 61 modes of AI while it is followed by European Union with 21 models and China 15 models.

Eightfold rise on AI investment

According to the report, the world has witnessed massive surge in investment in Generative AI. “Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds,” it stated.

The ‘Economy’ chapter of the report has brought forth information on investment, global AI labour demand, wages of computer scientists, adaptation of AI capabilities, corporate activity, robot installation and jobs. The report mentioned that China is dominating the installation of robots in industries.

“China dominates industrial robotics. Since surpassing Japan in 2013 as the leading installer of industrial robots, China has significantly widened the gap with the nearest competitor nation. In 2013, China’s installations accounted for 20.8% of the global total, a share that rose to 52.4% by 2022,” it explained.

Similarly, the AI governance initiatives are also on rise, especially in EU countries, the US and China. From AI policy to actions, these are painted as major players in AI governance. Policy makers in the world are not left untouched by the AI, the report argues.

It is also worth noting to have knowledge how the tech giants are investing huge chunk in designing multiple AI models and programmes in the world. The AI Index estimated, “Training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute.”

At a time when many tout technology as a non-discriminatory front, the global tech sprint of 2023, ChatGPT, has been found politically biased. The Index has referenced the researchers and stated ‘significant bias in ChatGPT toward Democrats in the US and the Labour Party in the UK.’

Bearing this in mind, the report has alerted over hijacking of political views of users by the ChatGPT when 2024 is a global election year.

So far, 80 countries have put in place the AI Strategies. They include the US, the UK, Bangladesh, India, Thailand, Rwanda, Ethiopia and Israel, among others. Canada was the first country to have AI strategy- in 2017.

It has argued that AI clinical knowledge has boosted in recent time. “Highly knowledgeable medical AI has arrived. Over the past few years, AI systems have shown remarkable improvement on the MedQA benchmark, a key test for assessing AI’s clinical knowledge,” the Index asserts.

The Index added that since the benchmark’s introduction in 2019, AI performance on MedQA has increased nearly threefold.

AI hiring has also seen upward trends in the countries like Australia, Canada, Singapore, and India since 2023.

Co-Directors of AI Index, Ray Perrault and Jack Clark observed the report, saying, “As AI rapidly evolves, the AI Index aims to help the AI community, policymakers, business leaders, journalists, and the general public navigate this complex landscape. It provides ongoing, objective snapshots tracking several key areas: technical progress in AI capabilities, the community and investments driving AI development and deployment, public opinion on current and potential future impacts, and policy measures taken to stimulate AI innovation while managing its risks and challenges.”

Several IT and AI experts provided information and expertise for the report where a dozen of tech organizations including GitHub, Linkedin, Schwartz Reisman Institute for Technology and Society, Women in Machine Learning, Code.org, and International Federation of Robotics supplied data.