A driver is constantly learning and develops a kind of intuition over time. For example, if you are driving in the fast lane and see a car ahead in the right lane slowly pulling to the left, you will automatically let off the gas – even if the other car has not yet put on its turn signal. Any experienced driver suspects that the other car is about to change lanes.
How can an autonomous vehicle learn from experience in the same way and also react intuitively? Porsche Engineering addressed this question together with Porsche AG and Cariad, a software and technology company of the VW Group, as part of a Big Data Loop proof of concept. The software specialists at Cariad are now pushing ahead with series development within the Group. The proof of concept was intended to demonstrate how all functions based on artificial intelligence (AI) can be continuously developed in the future. The solution resembles a circuit: Data from the vehicle is transmitted wirelessly to the cloud, where it is used to further train the AI. Afterwards, the improved algorithm is tested and fed back again.
Detecting lane changes at an earlier stage
The test object in this project is a vehicle equipped with an enhanced Adaptive Cruise Control (ACC) system. This driver assistance system ensures that a safe distance is always maintained from the vehicle in front, detecting early when other road users are cutting in, for example. The aim now is to use AI to detect this behaviour precisely, at an earlier stage
In the test vehicle, a neural network developed in-house takes over this task, which is continuously further trained with real scenes from the test drives. This creates an endless cycle of observation and learning that continuously improves the performance of the ACC. “A likely lane change is detected half a second to a second earlier – the equivalent of 30 metres of driving on the motorway,” explains Dr. Joachim Schaper, Senior Manager AI and Big Data at Porsche Engineering.
Every modern vehicle with assistance systems produces enormous amounts of ‘Big Data’, including evaluated camera signals or information from radar sensors – so there is plenty of material to train a neural network. What appears to be a simple idea at first glance, however, turns out to be a real challenge when it comes to implementation. “For example, we only want to record the data that really helps the system move forward,” says project manager Philipp Wustmann, an expert in longitudinal and lateral control at Porsche Engineering. “That's no easy task, because radar sensors and cameras generate an immense amount of data, most of which is not relevant to the function under consideration.” Driving on an empty highway, for example, offers no learning opportunities for a distance controller. Moreover, evaluating all the data would be far too time-consuming.
That's why we select specific scenes from which the AI can learn something. This task is performed by something called a SceneDetector in the Taycan test vehicle: this algorithm uses the interpreted camera signals on the vehicle bus – the system that transfers the data. These are not raw video images, but information about which objects are at what distance from the vehicle. The SceneDetector filters out those scenes from the current traffic situation in which the ACC is not yet reacting optimally – for example, when the vehicle cutting in is detected too late or incorrectly. In addition, it is technically possible to have the programme record what are known as corner cases – borderline situations that rarely occur in everyday driving. For example, if a vehicle in front swerves in its lane without actually changing it, the algorithm could mark this scene. The same applies to a situation in which the camera does not detect the lane markings. This detection of specific scenes is handled by special software called Automated Measurement Data Analytics (AMDA).
Propagation of data by simulation
Once the SceneDetector has found five potentially instructive cut-in events, it transmits the associated data to a server via mobile radio. In the cloud, the amount of illustrative material is increased: to do this, the data is first fed into a simulation that uses a game engine – the same technology that computer games use to generate their images. With the help of the Porsche Engineering Virtual ADAS Testing Center (PEVATeC), virtual test drives can be produced in which the vehicles in the computer physically behave like their real counterparts on the ground. The simulation results in measurements that correspond to those of the real vehicle bus.
Case examples
True positive (left): the cut-in detection system should correctly predict whether another vehicle is about to change lanes. If so, the Adaptive Cruise Control (ACC) can brake gently and early.
False positive (right): cut-in detection should also detect that a vehicle is drifting to the right, but will still not cut in. This helps to avoid unnecessary braking manoeuvres.
In the PEVATeC simulation environment, different variants of the recorded cut-in process are created, again automatically, on the basis of the real measurements – in other words, a re-simulation of the real situation takes place. In each case, the simulated cut-in processes differ only minimally: in one version the other road user pulls to the left more quickly, in another he is traveling at a greater distance. These variations generate more training data within a very short time without additional test drives. It also improves the generalisability of the AI model. It recognises not only standard situations, but also those that occur less frequently. That’s essentially the nature of the technology: neural networks acquire new skills exclusively through observation. The more examples they see, the better they become. The simulation environment also allows critical or atypical situations to be recreated to broaden the range of training data.
“A likely lane change is detected half a second to a second earlier—equivalent to 30 meters of travel on the motorway.” Dr. Joachim Schaper, Senior Manager AI and Big Data at Porsche Engineering
After all the visual scenarios have been created, the actual learning begins: all the real and simulated cut-in events are used to train a neural network in the cloud. By observing, it learns to recognise the signs of an approaching lane-change in a similar way to a human driver. This allows the ACC to brake smoothly, almost exactly as a real person would. Or as the expert Schaper puts it: “We are replicating intuition in AI.” In the long run, the vehicle could literally develop a sense for the behaviour of other road users and, for example, recognise an aggressive driving style that suggests risky lane changes. Porsche Engineering uses the Volkswagen Group’s cloud platform GroupMDM (MDM stands for Measured Data Management) to store and process the data.
The driver can activate a new release
Once training is complete, new software for the adaptive cruise control system is automatically created and validated. What that means is that it must be able to reliably recognise cut-ins in a large number of unknown test scenarios. Only if the software proves itself unequivocally is it transferred to the vehicle. Then the driver sees a dialogue box with the text: “A new release is available. Do you want to activate it?" If the driver then presses “OK,” the enhanced adaptive cruise control starts working.
Training and validation of new AI models
The new AI model is automatically trained with an improved dataset and validated with an existing validation dataset. If it is better than the previous model, it is used for further testing in the vehicle.
It would also be possible to test the new ACC module in the background (ShadowMode) in the vehicle first: while driving, the enhanced model receives the same sensor data as the existing on-board system and must react accordingly. However, the old ACC continues to control the accelerator and brake. Meanwhile, software monitors the quality of the predictions. If, for example, the “advanced” AI predicts a cut-in process that does not take place (a “false positive”), it would be disqualified. Only when it becomes clear that the neural network trained in the cloud is really superior in its predictions does it go into live operation.
.png)
In this proof of concept, the only manual step in the learning cycle involves the driver pushing a button to activate the new version of the ACC. “What's new is that everything is automated,” Schaper says. The recognition of the relevant scenes in the vehicle happens without human intervention, as does the propagation of the learning scenes in the simulation environment. A computer also controls the training of the neural network in the cloud and the update back to the vehicle. The developers only check things. “The vehicle optimises itself,” Wustmann says.
Proof of concept in just four months
Porsche Engineering was able to implement the self-learning adaptive cruise control system in just four months. A patent application was filed for part of the technology. The proof of concept demonstrated that the technical approach works. The architecture used here is already being used in other development projects, for example to test and validate a new sensor generation as a replacement for existing sensors. The next step will be to bring the Big Data Loop into series production. There are still a few challenges to be met here, for example corner case detection or fully automated function optimisation. Cariad is currently working on a series application of the Big Data Loop in the overall context of automated driving for VW Group brands. This proof of concept will provide valuable insights.
Once the necessary technology is standard in all delivered vehicles, learning will also be faster – because there will be more digital visual material. After all, while today a single test vehicle is on the highway collecting cut-in manoeuvres, in the future every vehicle will be able to send data back to the manufacturer if the customer agrees. Project manager Wustmann is excited by this prospect: “Getting direct feedback from customer fleets in different countries would save an immense amount of time, money and testing."
“The technology could also be interesting for lateral guidance, for example for the lane keeping system." Philipp Wustmann, Project Manager for Longitudinal and Lateral Control at Porsche Engineering
Cut-in recognition represents only one part of the road to a lifelong learning vehicle. In the future, neural networks will be used in many places in the vehicle, and they could all be improved by automatic feedback loops. “The technology could also be interesting for lateral guidance, for example for the lane keeping system,” says Wustmann.
In brief
The Big Data Loop allows AI-based vehicle functions to be continuously developed. The approach has proven itself in a proof of concept and is already being used in other development projects. It could be an important step towards the lifelong learning vehicle.
Cariad
The basis for the best possible optimisation in the Big Loop is an extensive database. Here, Porsche has an advantage due to its cooperation with other brands of the Volkswagen Group: the huge shared foundation of data software development is centrally bundled in the specially founded Cariad (I Am Digital) company. Supported by this data, systems and functions are developed that are given their distinctive DNA by Porsche engineers: assistance systems increase safety and convenience. They can relieve congestion, help with parking and even act as trainers. However, the decision remains an individual one and pure self-driving is always an option with Porsche.
Info
Text: Constantin Gillies
Contributors: Philipp Wustmann, Dr. Joachim Schaper
Illustrations: Florian Müller, Design Hoch Drei
Text first published in the Magazine Porsche Engineering, issue 2/2021.
