Questions about Running Evaluation


I just tried to follow the “Getting Started” instructions for the leaderboard and run into a couple issues:

  • pip install *.egg doesn’t work for me, but using easy_install *.egg works. I think I got some issue about not being able to parse the encoding format. Is this okay? I am using a Python3.7 base conda environment.

  • I think the evaluation bash script requires the track environment variable to be specified, else the argparse thinks the track is the empty string. Is there a default track that should be used?

  • I am able to successfully run the evaluation for one episode but it seems like the server crashes at the end with this error:
    LowLevelFatalError [File:Unknown] [Line: 772]
    VulkanRHI::vkQueueSubmit(Queue, 1, &SubmitInfo, Fence->GetHandle()) failed, VkResult=-4
    at /home/jenkins/UnrealEngine_4.24/Engine/Source/Runtime/VulkanRHI/Private/VulkanQueue.cpp:71
    with error VK_ERROR_DEVICE_LOST
    Signal 11 caught.
    Malloc Size=65538 LargeMemoryPoolOffset=65554
    CommonUnixCrashHandler: Signal=11
    Malloc Size=65535 LargeMemoryPoolOffset=131119
    Malloc Size=111328 LargeMemoryPoolOffset=242464
    Engine crash handling finished; re-raising signal 11 for the default handler. Good bye.
    Segmentation fault (core dumped)

Is this due to using Vulkan? I have a Nvidia 1070 GPU so maybe it’s not powerful enough or has insufficient memory?

  • Finally, I am wondering about the default mode used for evaluation. Is it synchronous (so run_step can potentially take an unbounded amount of time) or asynchronous so the control loop delays matter?

Thanks for the help!

So regarding the crashing, I think it was just Vulkan. Using OpenGL when launching CARLA seems to resolve that issue and I can run multiple scenarios smoothly.

About the synchronous mode question, I do see in the terminal that there is a sim time which isn’t quite at 1x speed. So is it correct to assume that the simulation isn’t real-time? Is there any time budget which we are expected to adhere to?

Hey @govvijay and sorry for the delay. We just updated the leaderboard docs in preparation for the CARLA challenge and it now has more detailed information.

As you noted at your first two bullets points, that information was incorrect and is now fixed.

Lastly, yes, the simulation is run synchronously so it will wait for all calculus to be completed. Currently, the max time budget is of 200 hours and up to 20 submissions per month.

1 Like