Yandex, in collaboration with Oxford and Cambridge Universities and as part of the NeurIPS conference on machine learning, is launching the global ‘Shifts Challenge’, a three-pronged competition track designed to tackle the problem of distributional shift in ML that features the largest autonomous vehicle (AV) dataset in the industry to date.
This dataset contains 600,000 scenes (or more than 1,600 hours of driving) collected through testing self-driving technology in different cities across the US, Israel and Russia in good weather, rain and snow.
Overcoming distributional shift is a crucial aspect of training ML models and is essential to building robust models that can operate in all circumstances, even those it has not previously been exposed to. This is a prerequisite for models that want to operate in ‘real-life situations’, such as autonomous vehicles roaming the streets of our cities. As such, competitions such as this are a key tool in accelerating research in this area.
The ‘Shifts Challenge’ features three competition tracks, focusing on AV trajectory predication, machine translation and weather forecasting. In addition to the 600,000 scene AV dataset, participants on the other two tracks will have access to high-quality datasets from the Yandex. Translate and Yandex. Weather services.
Participants on the AV track are invited to train their motion prediction models on certain types of scenes and then test them in different conditions (different cities, countries and weather) for further improvement. Their models will then be evaluated by the challenge committee via a new part of the dataset, which will include new shifts in data.
The final ranking will be based on the models’ prediction accuracy as well as on the ability of models to estimate the uncertainty of its predictions in any given case. Uncertainty estimation shows how sure the model is about its decisions and is equally as important as the accuracy of predictions made by the models. This is crucial for AV technology which must be both robust and reliable.
Yandex, Cambridge and Oxford universities team up to launch international machine learning competitionRead More“As deep-learning approaches become more powerful, they are being applied in ever more interesting and diverse areas. It is increasingly important for these systems to “know when they don’t know”, to prevent bad decisions. Through participation in the global Shifts Challenge, researchers have an unprecedented opportunity to evaluate on large-scale, real-world data their models’ ability to measure confidence in their own predictions.” — Professor Mark Gales (Cambridge University)
“The main obstacle to the development of robust models which yield accurate uncertainty estimates is the availability of large, diverse datasets which contain examples of distributional shift from real, industrial tasks. Most research in the area has been done on small image classification datasets with synthetic distributional shift. Unfortunately, promising results on these datasets often don’t generalize to large-scale industrial applications, such as autonomous vehicles. We aim to address this issue by releasing a large dataset with examples of real distributional shift on tasks which are different from image classification. We hope that this will set the new standard in uncertainty estimation and robustness research.” — Andrey Malinin, Yandex senior research scientist and Shifts Challenge lead.
