The Big Loop

However, automatic feedback loops are also interesting for other functions. CARIAD is actively advancing the development of the Big Data Loop for a wide variety of complex application purposes. A driver is constantly learning and develops a kind of intuition over time. For example, if you are driving in the fast lane and see a car ahead in the right lane slowly pulling to the left, you will automatically let off the gas—even if the other car has not yet put on its turn signal. Any experienced driver suspects that the other car is about to change lanes.

Detecting lane changes at an earlier stage

The test object in the project is a vehicle equipped with an enhanced Adaptive Cruise Control (ACC) system. The regular series-production driver assistance system ensures that a safe distance to the vehicle in front is al- ways maintained. To do this, the ACC must detect early when other road users are cutting in. The aim is now to use AI to detect precisely this behavior at an earlier stage: In the test vehicle, a neural network developed in-house takes over this task, which is continuously further trained with real scenes from the test drives. This creates an endless cycle of observation and learning that continuously improves the performance of the ACC. “A likely lane change is detected half a second to a second earlier—the equivalent of 30 meters of driving on the motorway,” reports Dr. Joachim Schaper, Senior Manager AI and Big Data at Porsche Engineering.

Every modern vehicle with assistance systems pro- duces enormous amounts of data (Big Data), including evaluated camera signals or information from radar sensors—so there is plenty of material to train a neural network. What appears to be a simple idea at first glance, however, turns out to be a real challenge when it comes to implementation. “For example, we only want to record the data that really helps the system move forward,” says project manager Philipp Wust- mann, an expert in longitudinal and lateral control at Porsche Engineering. “That's no easy task, because radar sensors and cameras generate an immense amount of data, most of which is not relevant to the function under consideration.” Driving on an empty highway, for example, offers no learning opportunities for a distance controller. Moreover, evaluating all the data would be far too time-consuming.

That's why we select specific scenes from which the AI can learn something. This task is performed by something called a SceneDetector in the Taycan test vehicle: This algorithm uses the interpreted camera signals on the vehicle bus. These are not raw video im- ages, but information about which objects are at what distance from the vehicle. The SceneDetector filters out those scenes from the current traffic situation in which the ACC is not yet reacting optimally—for example, when the vehicle cutting in is detected too late or incorrectly. In addition, it is technically possible to have the program record what are known as corner cases, meaning borderline cases that rarely occur in everyday driving. For example, if a vehicle in front swerves in its lane without actually changing it, the al- gorithm could mark this scene. The same applies to a situation in which the camera does not detect the lane markings. This detection of specific scenes is handled by special software called Automated Measurement Data Analytics (AMDA).

Propagation of data by simulation

Once the SceneDetector has found five potentially in- structive cut-in events, it transmits the associated data to a server via mobile radio. In the cloud, the amount of illustrative material is increased: To do this, the data is first fed into a simulation that uses a game engine, the same technology that computer games use to generate their images. With the help of the Porsche Engineering Virtual ADAS Testing Center (PEVATeC), virtual test drives can be produced in which the vehicles in the computer physically behave like their real counterparts on the asphalt. The simulation results in measurements that correspond to those of the real vehicle bus.

Case examples

True positive (left): the cut-in detection system should correctly predict whether another vehicle is about to change lanes. If so, the Adaptive Cruise Control (ACC) can brake gently and early.

False positive (right): cut-in detection should also detect that a vehicle is drifting to the right, but will still not cut in. This helps to avoid unnecessary braking maneuvers.

In the PEVATeC simulation environment, different variants of the recorded cut-in process are created, again automatically, on the basis of the real mea- surements—in other words, a re-simulation of the real situation takes place. In each case, the simulated cut-in processes differ only minimally: In one version the other road user pulls to the left more quickly, in an- other he is traveling at a greater distance. These vari- ations generate more training data within a very short time without additional test drives. It also improves the generalizability of the AI model. It recognizes not only standard situations, but also those that occur less frequently. That’s essentially the nature of the tech- nology: Neural networks acquire new skills exclusively through observation. The more examples they see, the better they become. The simulation environment also allows critical or atypical situations to be recreated to broaden the range of training data.

“A likely lane change is detected half a second to a second earlier—equivalent to 30 meters of travel on the motorway.” Dr. Joachim Schaper, Senior Manager AI and Big Data at Porsche Engineering

After all the visual scenarios have been created, the actual learning begins: All the real and simulated cut-in events are used to train a neural network in the cloud. By observing, it learns to recognize the signs of an approaching lane change in a similar way to a human driver. This allows the ACC to brake smoothly, almost exactly like a human driver. Or as the expert Schaper puts it: “We are replicating intuition in AI.” In the long run, the vehicle could literally develop a sense for the behavior of other road users and, for example, recognize an aggressive driving style that suggests risky lane changes. Porsche Engineering uses the Volkswagen Group’s cloud platform GroupMDM (MDM stands for Measured Data Management) to store and process the data.

The driver can activate a new release

Once training is complete, new software for the adap- tive cruise control system is automatically created and validated. What that means is that it must be able to reliably recognize cut-ins in a large number of un- known test scenarios. Only if the software proves itself unequivocally is it transferred to the vehicle. Then the driver sees a dialog box with the text: “A new release is available. Do you want to activate it?" If the driver then presses “OK,” the enhanced adaptive cruise control starts working.

Training and validation of new AI models

The new AI model is automatically trained with an improved dataset and validated with an existing validation dataset. If it is better than the previous model, it is used for further testing in the vehicle.

It would also be possible to test the new ACC module in the background (ShadowMode) in the vehicle first: While driving, the enhanced model receives the same sensor data as the existing on-board system and must react accordingly. However, the old ACC continues to control the accelerator and brake. Meanwhile, software monitors the quality of the predictions. If, for example, the “advanced” AI predicts a cut-in process that does not take place (“false positive”), it would be disqualified. Only when it becomes clear that the neural network trained in the cloud is really superior in its predictions does it go into live operation.

In this proof of concept, the push of a button by which the driver activates the new version of ACC represents the only manual step in the learning cycle. “What's new is that everything is automated,” Schaper emphasizes. The recognition of the rele- vant scenes in the vehicle happens without human intervention, as does the propagation of the learning scenes in the simulation environment. A computer also controls the training of the neural network in the cloud and the update back to the vehicle. The developers only check things. “The vehicle optimizes itself,” Wustmann summarizes.

Proof of concept in just four months

Porsche Engineering was able to implement the self-learning adaptive cruise control system in just four months. A patent application was filed for part of the technology. The proof of concept demonstrated that the technical approach works. The architecture used here is already being used in other development projects, for example to test and validate a new sensor generation as a replacement for existing sensors. The next step will be to bring the Big Data Loop into series production. There are still a few challenges to be met here, for example corner case detection or fully automated function optimization. CARIAD is currently working on a series application of the Big Data Loop in the overall context of automated driving for the brands of the VW Group. This proof of concept will provide valuable insights.

“The technology could also be interesting for lateral guidance, for example for the lane keeping system." Philipp Wustmann, Project Manager for Longitudinal and Lateral Control at Porsche Engineering

Once the necessary technology is standard in all delivered vehicles, learning will also be faster—be- cause there will be more digital visual material: After all, while today a single test vehicle is on the highway collecting cut-in maneuvers, in the future every vehi- cle will be able to send data back to the manufacturer if the customer agrees. Project manager Wustmann is excited by this prospect: “Getting direct feedback from customer fleets in different countries would save an immense amount of time, money and testing."

Cut-in recognition represents only one part of the road to a lifelong learning vehicle. In the future, neural networks will be used in many places in the vehicle, and they could all be improved by automatic feedback loops. “The technology could also be interesting for lateral guidance, for example for the lane keeping system,” Wustmann reports.

In brief

The Big Data Loop allows AI-based vehicle functions to be continuously developed. The approach has proven itself in a proof of concept and is already being used in other devel- opment projects. It could be an important step towards the lifelong learning vehicle.

Text: Constantin Gillies
Contributors: Philipp Wustmann, Dr. Joachim Schaper
Illustrations: Florian Müller

Text first published in the Magazine Porsche Engineering, issue 2/2021.