The leaderboard repo contains predefined routes (.xml files) and scenarios (.json files). Will the script for generating these .xml and .json files be provided?
Hey aprakash, as far as I know, they were generated by hand, so probably not
The provided json files contain 3 types of scenarios - Scenario1, Scenario3 & Scenario4 whereas the scenario manager has 13 supported scenarios. Is the distribution of the evaluation scenarios in the leaderboard similar to that of the provided json file or does it have all these extra scenarios to test out of distribution generalization?
Not all of the 13 supported scenarios are present in the leaderboard, only 10 out of these 13 are used, and can be viewed here.
Not all of them, however are scenarios per se. For example, Scenario2 is to follow a vehicle, but as routes have in-built traffic, there is no need to create one specific moment, as it will constantly have to do so to avoid crashing.
Thanks for the clarification.