Sber has presented its leaderboard called Russian SuperGLUE (General Language Understanding Evaluation), the first ranking of neural networks for the Russian language. A neural network’s ranking position depends on how well it deals with tasks in logic, commonsense, reasoning, and general language understanding.
Designed by the SberAI team with Huawei and the Higher School of Economics, the leaderboard is an open project for all data scientists working with neural networks in Russian. Ranking results are published on the website.
“Modern neural networks that are used for text processing purposes are so complex that stakeholders use tasks similar to the IQ test for humans to evaluate them. The Russian SuperGLUE project is the first ranking comparing Russian language neural network models. Objective evaluation metrics allow us to measure our progress towards building reliable intelligent systems, which ultimately brings us closer to creating AGI, known as ‘strong’ artificial intelligence.” – Alexander Vedyakhin, First Deputy Chairman of the Executive Board, Sberbank
Compared to the first version of the leaderboard, Russian SuperGLUE 1.1 has updated and expanded sets of tasks devoted to question answering systems and understanding the meaning of texts. Furthermore, the code base has been significantly increased, so now the project presents ready-made solutions to assess the quality of large language models such as GPT-3, BERT, mBERT, mBART, GPT-2, XLM-MLM, XLM-Roberta. The new version of the ranking also has better reliability of the model ranking system.
In mid-November, the project was successfully presented at the largest international conference EMNLP (The Conference on Empirical Methods in Natural Language Processing). On Friday, it will be presented at Huawei University during a Natural Language Processing seminar.
AI development issues will be addressed at AI Journey, the largest platform for obtaining up-to-date information about AI and its application in Russia and abroad. The event is hosted by Sberbank in cooperation with leading Russian and foreign companies. In 2019, AI Journey was held online and attracted more than 150 leading industry experts as speakers, thus hitting the world’s Top 5 DS/AI conferences by the number of participants.
This year, about 200 leading Russian and international experts are expected as speakers. Among the recognized experts who have confirmed their attendance are Mike Davies, director of the Neuromorphic Computing Lab, Intel; Laurens van der Maaten, Research Director, Facebook AI Research; and Jürgen Schmidhuber, IDSIA Lab Director.
