Video

CARLA (Car Learning to Act) is an open-source simulator desgined within Academia as an autonomous driving research tool. Developed by the Computer Vision Center, along with Intel Labs and the Toyota Research Institute, it is a platform in which to support the development, training and validation of autonomous urban driving systems. CARLA was presented at the First Conference in Robot Learning at Mountain View, CA by CVC/UAB PhD candidate Felipe Codevilla.

Training an autonomous car to drive is a challenge that is being tackled in research all over the world. Cars are performing simple driving tasks on real, actual roads. However, teaching these cars to drive with zero incidents and in the most varied scenarios as possible isn’t trivial. There are plenty of rare and odd situations of which one sole car might never encounter and it needs to know how to react real-time.

“Imagine a child running towards the road, or a very dusty evening with the sun lying low and frontally into the car’s cameras,” explains Felipe Codevilla, co-author of the paper  ‘CARLA: An open urban driving simulator’. “You expect the car to be able to respond to these situations, but you need to have trained it first”. CARLA enables researchers to trigger the different, unexpected situations a car might come up against. As added by Dr. Antonio López, head of the ADAS team at CVC and also co-author of the paper. “CARLA allows us to drive in different environments, lighting conditions, weather changes or urban scenarios”.

The physical world represents clear difficulties for autonomous driving research, not only infrastructure costs and logistic difficulties, but funds and manpower involved are high and costly. Furthermore, a single vehicle is far from sufficient for collecting the requisite data that cover the multitude of corner cases that need to be processed for both training and validations. CARLA has been developed to overcome such challenges and give researchers a new, open source, research-oriented platform.

Although the use of simulators for autonomous driving is not new, and videogame technology has been used to train autonomous cars in the past, existing simulation platforms are limited, lacking numerous basic elements such as pedestrians, traffic rules, intersections, or other complications that may arise constantly in real life driving.

Commercial videogames, such as The Grand Theft Auto have also been tested in autonomous driving research, but the privileged information that the car needs to comprehend its environment remains unavailable in videogames due to their commercial nature. CARLA, being built from zero for autonomous driving research purposes, gives the car access to privileged information such as GPS coordinates, speed, acceleration and detailed data on a number of infractions.

The sensors that the virtual car has within the simulator are composed by RGB cameras and by pseudo-sensors that provide ground-truth depth and semantic segmentation. Camera parameters include 3D location, 3D orientation with respect to the car’s coordinate system, field of view and depth of field. The semantic segmentation sensor provides a total of 12 semantic classes: road, lane-marking, traffic sign, sidewalk, fence, pole, wall, building, vegetation, vehicle, pedestrian and other.

The simulator not only recreates a dynamic world but also provides a simple interface between this world and the agent that interacts with it. The platform has a highly realistic environment and enables users to use a set of sensors to guide the car. By not using metric maps, visual perception becomes a crucial asset for the vehicle.

The authors carried out three approaches when testing autonomous driving in CARLA: Firstly, a classic modular pipeline; secondly, an end-to-end model trained via imitation learning, and finally, an end-to-end model trained via reinforcement learning. The first approach, a classic modular pipeline, structured the driving task into three subsystems: perception, planning and continuous control.

In the second approach, the imitation learning in an end-to-end model, researchers used a dataset of driving traces recorded by human drivers, collecting a total of 14 hours of driving data for training. The third and last approach, the reinforcement learning model, trained the deep network based on a reward signal provided by the environment, with no human traces. Conclusions were that the performance of two of the systems (modular pipeline vs the imitation learning approach) was very close under most of the testing conditions, differing by less than 10%.

When performance was compared between the imitation learning and reinforcement learning models, they realised that the agent instructed with reinforcement learning performed significantly worse than the one trained by human imitation. The model based on reinforcement learning was prepared using a significantly larger amount of data; thus, the results suggested that an out-of-the-box reinforcement learning algorithm is not sufficient for the driving task and more research needs to be developed further within this line of study.

“Performance isn’t optimal in any of the tested methods” states Felipe Codevilla when asked for a conclusion. Results showed that giving cars new environments and a set of situations they hadn’t encountered in previous training poses still a serious challenge. Experts now expect that CARLA, being open source, will enable a broad community to actively engage in autonomous driving research.

More information at Carla.org.

Reference:

A. Dosovitskiy, G. Ros, F. Codevilla, A. López, V. Koltun (2017): CARLA: An Open Urban Driving Simulator

---

Related articles: 

CARLA in the Media

Towards A No Driver Scenario: Autonomous And Connected Cars At The Computer Vision Center

The Future Of Autonomous Cars: Understanding The City With The Use Of Videogames

Compitiendo Contra La Inteligencia Artificial Del Coche Autónomo