Előadás

Artificial Intelligence-based Perception in Self-Driving Vehicles

Beyond Artificial Intelligence Research: a New Innovation and Industrial Domain

2025. november 7. 13:00Babeș-Bolyai University, Central Building, Aula Magna · Kolozsvár · Románia

Átirat · 1:12:19

uh remote. It works. Super. So thank you uh thank you for uh the invitation and

Átirat · 1:12:26

uh I'm really honored to be here. Um, my name is Stefan Mateman. I'm half

Átirat · 1:12:32

Hungarian, half Romanian. Uh, I currently work at Robert Bosch, but I have an academic background. Actually,

Átirat · 1:12:39

uh, I did my PhD in Toronto in University of Toronto. Uh, why I mention this is because I'm kind of with a foot

Átirat · 1:12:44

on in two worlds in the industrial applied part, but also in the fundamental research part a little bit.

Átirat · 1:12:51

And during my presentation, I'm going to talk today about perception in autonomous and assisted driving. I'm

Átirat · 1:12:58

going to kind of try to point out the relationship between fundamental research on one side and the applied uh

Átirat · 1:13:04

industrial setting. Okay. So um my presentation will have three parts.

Átirat · 1:13:10

First I want to show you a little bit what the setup is in autonomous driving. And here I'm in the hardware and also

Átirat · 1:13:16

the main let's say architecture that we would have also the typical approach

Átirat · 1:13:21

that we would have in autonomous driving. And then I would like in the second part of my presentation to dive

Átirat · 1:13:27

into the challenges that we actually have. Now there's a lot of challenges. I'm just going to pick four of them. And

Átirat · 1:13:32

with each of these challenge I'm going to show you some a few short deep dives on some techniques that we can use to

Átirat · 1:13:38

mitigate those challenges. And what I hope you will see that there's a pattern. There's a pattern of methods

Átirat · 1:13:44

that keep reappearing in different contexts and certain ideas that are kind

Átirat · 1:13:50

of resisting the uh time. So even they are here from the beginning of the field and still they keep reappearing.

Átirat · 1:13:58

So let's talk about the setup the context for autonomous driving. So

Átirat · 1:14:03

usually I mean you can imagine if you have a car that has to drive on the road it has to perceive all around. So 360

Átirat · 1:14:11

perception right you have to see everything around you if you want to really have a safe uh vehicle right. So

Átirat · 1:14:18

um what do we have? What are we armed with? what what what is our hardware? Of course, we'll have some sensors and we

Átirat · 1:14:25

will have cameras which give you a lot of rich dense information and capture appearance. Uh and you also usually have

Átirat · 1:14:33

some radars or leaders which will give you more depth information usually but less of the of the details right of the

Átirat · 1:14:40

appearance. And then you feed all this into some specialized embedded hardware

Átirat · 1:14:45

that is not that embedded as on a drone as you've seen before from Levy. It's maybe a little you have the luxury of

Átirat · 1:14:51

having more compute power in a car but not too much. So you still have to be very careful about how you optimize your

Átirat · 1:14:58

system and at the end what do you want out of this? Well you will have the three Ps as they say from autonomous

Átirat · 1:15:05

driving that is first you want to perceive right you have to see. So once

Átirat · 1:15:10

you see what's around you, the cars, the static environment and so on, then you want to predict what's going to happen

Átirat · 1:15:16

next. What are the other traffic participants are going to do? So that's the second P, prediction.

Átirat · 1:15:22

Once you did the prediction, then you are ready to do planning. Okay? So you can actually plan what you are going to

Átirat · 1:15:29

do next. You're going to accelerate, break, steer left, steer right, and so on. Okay? Um these are the three main uh

Átirat · 1:15:36

outputs that you would want from the system and today we're just going to talk about perception because that's a

Átirat · 1:15:42

is a short presentation. Uh but uh many of the techniques I'm going to talk about can be applied for the other

Átirat · 1:15:49

stages as well. Okay. So now that we know what the inputs and outputs are, what is the typical uh typical

Átirat · 1:15:55

architecture? Well, for each modality, you probably want to somehow extract the

Átirat · 1:16:01

information from it using an encoder. So, you have an encoder for the images or for the cameras and you have an

Átirat · 1:16:07

encoder for uh the leader rather for example or for each other modality that

Átirat · 1:16:12

you you have to you have on the car. What do you get out of this? You get a representation. Now, this element is the

Átirat · 1:16:20

key. If you have a good representation of the world, then you will have a very

Átirat · 1:16:25

easy job of solving your perception, prediction, and planning tasks. So, it's very important to have a good

Átirat · 1:16:30

representation. What does it mean to be good? What's a good representation? Well, um if you've

Átirat · 1:16:36

seen actually Ard Karpati has this nice nice slide uh with um cats that are you

Átirat · 1:16:43

know dark and white and so on. You can see that what you want is actually a representation in which each element of

Átirat · 1:16:51

your feature vector right it's just a bunch of numbers right is sensitive to

Átirat · 1:16:56

an orthogonal property of the input one of them captures geometry the other captures motion the other captures I

Átirat · 1:17:02

don't know texture and so on right this is called disentangling the factors of variations in the input so that's what

Átirat · 1:17:09

we would like that's the holy grail of representation this is a challenge and you'll see later that it's not so easy to to get this once you represent the

Átirat · 1:17:16

world and by the way this representation I said it's a vector but usually it's more than that in autonomous driving we

Átirat · 1:17:23

have this kind of voxal so imagine the space like this room being filled up with little cubes each cube has its own

Átirat · 1:17:28

feature vector so you al already associate the vectors with some threedimensional position usually that's

Átirat · 1:17:34

what you you usually have for a real um um production ready system so once you

Átirat · 1:17:40

have that representation then from these features you start attaching heads. So

Átirat · 1:17:46

these heads can are other software modules that take those features and can predict um the location of vehicles.

Átirat · 1:17:52

You've seen 3D bounding body detection already from Levy. So that was actually an example for seeing where the cars are

Átirat · 1:17:58

and also their orientation. Okay, so those are the head here. I'm showing just 3D object detection and occupancy.

Átirat · 1:18:05

Semantic occupancy is similar to the vauels I mentioned before. But inside the featurel, I know whether it's

Átirat · 1:18:10

occupied by vegetation, it's occupied by a road or by nothing. It's a free voxal or it's occluded. Maybe I don't even see

Átirat · 1:18:16

it. Okay. You can see there an example on the slide. Okay. So, how do we

Átirat · 1:18:21

approach this? So, the simplest idea is to try to write it yourself, right?

Átirat · 1:18:26

That's what we did at the beginning a long long time ago. So the problem with perception is that perception is

Átirat · 1:18:33

something that psychologists term an inconspicuous task. Inconspicuous means

Átirat · 1:18:38

that as a human you you perceive you do it all the time every day, right? But you don't know how the heck you're doing

Átirat · 1:18:44

it. You can just do it automatically, right? So that means that if you want to write a program for it, you cannot do

Átirat · 1:18:51

it. It's just it's impossible because you don't know how you do it, right? So then trying to devise your own program

Átirat · 1:18:58

by hand will not work. So the first stage was try to devise your own features by hand and on top of that

Átirat · 1:19:05

attach a head with unknown parameters. It's a it's one candidate from a family of programs and

Átirat · 1:19:12

try to find the best program. This is classical machine learning. I'm sure you're familiar with it. So this was the first stage and that's classical machine

Átirat · 1:19:18

learning. So we engineer the feature and we just learn the head, right? such that the head responds in the way we want.

Átirat · 1:19:25

Maybe by giving the right uh vehicle locations or giving the right labels depends or maybe even fancier setups

Átirat · 1:19:31

like reinforcement learning but don't want to go into that right now. Okay. So uh this was good but not good enough. Uh

Átirat · 1:19:39

usually the performance wasn't that good because still we cannot get the right features. We have the same inconspicuous

Átirat · 1:19:45

features. Now how do we represent the world? How do we disentangle shape from motion and stuff using a handwritten

Átirat · 1:19:52

program? We don't know how to do that. So the next stage was to go a little bit further and then actually this was the

Átirat · 1:19:59

deep learning stage. Try to come up with a vocabulary of layers. So primitive

Átirat · 1:20:05

operations with unknown parameters and you just you know connect these into into something that's called an

Átirat · 1:20:10

architecture and just search for the best feature extractor by fitting these to the data. Right? So that was deep

Átirat · 1:20:17

learning. Um and that was um another step into making this field more

Átirat · 1:20:23

homogeneous. If in the previous era we had various types of features like we

Átirat · 1:20:28

had hogs, ACFs, those of you who worked in those days, right? We had uh different kinds of uh u machine learning

Átirat · 1:20:35

models like support vector machines, you had uh decision trees and so on. They look very different. Each solution was

Átirat · 1:20:40

tailored for a specific application. With deep learning that is gone. Now we have a single uh vocabulary of layers.

Átirat · 1:20:49

We just specify the architecture and out of that we just train it and we get we

Átirat · 1:20:54

get uh the results. So this was another step towards having things more homogeneous more the same for all applications and it sped up a lot the

Átirat · 1:21:01

field and also got you good results. Uh this was by the way as a parenthesis inspired from something um that's called

Átirat · 1:21:08

um connectionism. It comes from u cognitive psychology which said that intelligence comes from little

Átirat · 1:21:14

components and the way you interconnect them because the components are very simple. If you speak about layers in

Átirat · 1:21:19

your networks, you don't have a lot of them, right? You have 20 types of layers maybe that you use daily, right? So now

Átirat · 1:21:25

we got another level of homogeneity with deep learning. We could come up with an architecture for uh I don't know um the

Átirat · 1:21:33

head for occupancy, the head for for uh vehicle detection. we could come up with an architecture for backbone and we are

Átirat · 1:21:39

happy but maybe we can go even further so that that's where the field is I put a star on the slide and now we're going

Átirat · 1:21:46

to a next stage which are foundation models so what is the promise of these foundation models the promise of

Átirat · 1:21:52

foundation models is that actually you can have a single model that can solve a lot of tasks basically well any task

Átirat · 1:21:58

they say maybe not any task but so so many things it can solve so many things that you could reuse it for different

Átirat · 1:22:03

domains okay and you've seen already them popping up these days. Okay. So now

Átirat · 1:22:09

we got even more homogeneous uh way of working with the same approach even

Átirat · 1:22:14

across domains now. Okay. So this was the path from classical computer vision,

Átirat · 1:22:20

classical machine learning, deep learning and then foundation models. So we're kind of in the in the middle from

Átirat · 1:22:25

deep learning to foundation models in in the field of autonomous driving. Okay. So now I've shown you like you know

Átirat · 1:22:33

roughly what the techniques are. Let's go into details. So what are the uh challenges? And the first one, as I

Átirat · 1:22:39

mentioned before, I hinted representations. Okay, so it's critical to have a good representation. I hope I

Átirat · 1:22:45

convinced you a little bit. Uh now, uh why do we care? Maybe if I put a lot of

Átirat · 1:22:51

tests, it's just going to learn to represent the world, right? If it has to detect vehicles with enough data, it's

Átirat · 1:22:57

okay. It's going to learn to separate geometry from motion because it has to work, right? Usually that doesn't

Átirat · 1:23:02

happen. Why? Well, you have noise. You have accidental correlations between features. So, uh, usually you would need

Átirat · 1:23:10

an huge amount of data even to hope even with no noise to actually disentangle the factors of variation. So, you kind

Átirat · 1:23:16

have to come up with more bias, more inductive bias in order to learn to separate things. So, I'm going to to

Átirat · 1:23:23

show you just two random techniques and just to give you a hint of where what's uh what can be used. So, the first idea

Átirat · 1:23:29

is is self-supervised learning. Maybe you heard about this. So this is an idea that's actually related to how we were

Átirat · 1:23:35

designing features in the first era. Actually remember we were trying to deise uh features that are invariant to

Átirat · 1:23:43

certain transformations of the image. And you've already seen again from Levy some examples of invariance actually with the leaves and different

Átirat · 1:23:49

orientations of the leaf. It's still a leaf if I I apply a homography to it. Right? The same idea here. If I do a

Átirat · 1:23:56

transformation on the input then I and I encode it with my feature encoder and I do another transformation

Átirat · 1:24:03

and encode it should give the same features or at least they should be

Átirat · 1:24:08

similar in a certain way right it should be invariant to those transformations okay so this is self-supervised learning

Átirat · 1:24:14

now what transformation doesn't matter it can be masking it can be uh in this case I'm showing masking actually it can

Átirat · 1:24:20

be shifting it can be anything you want it can be even even moving temporarily to another frame actually. So this is uh

Átirat · 1:24:27

actually what I'm showing here is an instantiation of this called JEPA. It's the joint embedding prediction

Átirat · 1:24:32

architecture. And if you look at the approach here, you'll see those red red

Átirat · 1:24:38

labels. There are some tricks. So assume you are the deep learning model and you have to represent the input in such a

Átirat · 1:24:44

way that it doesn't care about about masking. Let's say so no matter what I mask it you should give the right

Átirat · 1:24:50

feature. Well, you would happily predict zero and you're done. You give the same output, right? you would not even look

Átirat · 1:24:55

at the input. This is called mode collapse, right? It's a good example of the need to regularize. You need to come

Átirat · 1:25:02

up with a bias that comes from you as a designer and force it into the model such that the model doesn't go to

Átirat · 1:25:08

trivial solutions. In this case, the idea would be for example to force the

Átirat · 1:25:13

model to encode some information from the input, not just predict constant by saying that there should be mutual

Átirat · 1:25:19

information maximized between the input and the representation that it's it's building. Right? So you see there is a

Átirat · 1:25:27

bias an inductive bias and it doesn't come from free it comes from you. it's come and this is true through all the

Átirat · 1:25:34

eras of deep learning machine learning classical computer vision always always the bias is there from the designer

Átirat · 1:25:40

that's what I'm trying to my main message at least probably today that there there's this biases that you

Átirat · 1:25:46

really have to communicate they are sometimes hidden they are you know like okay there a huge language model it

Átirat · 1:25:52

doesn't have bias it does have a bias even by the way I I put the attention models that I do auto reggressive training there is bias there all the

Átirat · 1:25:58

time another example apologies is uh you could steal bias. So if you have an

Átirat · 1:26:04

off-the-shelf model, you could have that this is actually a paper from Bosch. Uh

Átirat · 1:26:09

you could actually generate uh you know ask your language model or visual

Átirat · 1:26:14

language model in this case to describe certain objects given an image. And the features that the language model has in

Átirat · 1:26:21

order to answer your prompt should contain information that should also be captured

Átirat · 1:26:28

by the uh model that you're you're looking for the the vision only model. Right? So what's happening here is that

Átirat · 1:26:34

the language model is a teacher to the for the uh classical um or for for the

Átirat · 1:26:40

target model. So this is an example in which bias is being transferred from one domain to the other and because in

Átirat · 1:26:46

language maybe it's easier to get certain biases then you get an improvement in your uh vision encoder

Átirat · 1:26:52

for autonomous driving. So bias can be encoded directly or even transferred across domains. This is my main message

Átirat · 1:26:58

here. And there's other techniques but I will resume myself to these two. Okay. Um yeah I put some notes for you guys if

Átirat · 1:27:04

you get the slides. [snorts] Okay. Uh by the way this can be combined. So you can you can combine

Átirat · 1:27:10

both methods inject your own bias let's say with Japa at the same time also have a teacher from another domain

Átirat · 1:27:16

simultaneously during the training process. So it's pretty flexible

Átirat · 1:27:21

safety. Now this is [snorts] an example of a field where there's a gap. the the literature the fundamental literature is

Átirat · 1:27:28

not really focusing that much on this topic but industry is really really really interested in safety because in

Átirat · 1:27:36

order to to sell a vehicle you need to meet very very strict ISO standards

Átirat · 1:27:42

safety you have to show that you'll always get the uh the system to to break

Átirat · 1:27:47

and stop in case of an emergency so can we do it with machine learning can we just have a system that you know always

Átirat · 1:27:53

works there is a consensus in the field that you can't currently with the current technology inherently safe AI

Átirat · 1:28:00

it's impossible at at least in the maybe even the next few years that's kind of what I've I've I've seen so we cannot

Átirat · 1:28:08

really hope that the machine learning system is going to generalize on each situation just because in the lab it

Átirat · 1:28:13

worked okay so what can we do well what we could do is actually sa fail safely

Átirat · 1:28:19

so have the machine learning system like you know your uh feature extractor and

Átirat · 1:28:25

then the the task heads be able to tell you something is wrong. I'm not really sure what's going on here. Please ask

Átirat · 1:28:31

the driver to put it his or her hands on the wheel. Okay. So this um is one field

Átirat · 1:28:37

of research which really helps a lot. Uh the other idea is to uh during the

Átirat · 1:28:42

training process to already uh you know make the model resilient to certain um

Átirat · 1:28:48

out of distribution scenarios that are never present in the laboratory. Okay. uh I will not have time to go into

Átirat · 1:28:54

details. I've shown you here some techniques that can be used. Um what if you look at these techniques there is

Átirat · 1:29:01

maybe some already look familiar like multitask learning I just mentioned having several tasks attached to the same representation

Átirat · 1:29:07

also um for example in distribution error error detection is focused on

Átirat · 1:29:12

trying to see already from training time which samples are outliers and here you see the same idea I need a bias. How do

Átirat · 1:29:20

I define an outlier? How do I think about clustering for example simplest approach right? my cluster my my

Átirat · 1:29:25

training set what how many clusters do I have has to come from you again the same team without bias you cannot learn okay

Átirat · 1:29:32

so um again and you'll see several techniques actually coming up that can be used for safety and also for other

Átirat · 1:29:39

challenges you'll see immediately efficiency very important we're talking

Átirat · 1:29:44

about an embedded system okay bigger than maybe on a mobile phone or a drone you have more power on a car but not too

Átirat · 1:29:49

much so you want it to be real time and this is related to safety as well because if it's not fast enough, it's

Átirat · 1:29:56

not going to react enough fast enough to to the environment. Okay? So, it has to be fast. It has to have low bandwidth.

Átirat · 1:30:02

We have a lot of data moving around in a car. It has to consume of course not too

Átirat · 1:30:08

much power. And you have to accommodate a very limited operator set in such in such embedded systems. And here I

Átirat · 1:30:15

already talk about just a subset of the layers that are supported also like maybe four bits for each feature not a

Átirat · 1:30:22

floatingoint number. So how can we we uh deal with this? One idea similar to the

Átirat · 1:30:28

teacher model that you've seen before already when we're talking about language teaching the uh the vision

Átirat · 1:30:34

model for ad is knowledge distillation. Uh and maybe I can show you briefly this

Átirat · 1:30:40

one. So actually here we have the large model. So we train a good model

Átirat · 1:30:45

floatingpoint model that works well in the cloud or on my computer and then I

Átirat · 1:30:50

have a certain candidate embedded model that works in the car which is smaller quantized you know it it's not that

Átirat · 1:30:57

powerful. So what I can do is I can distill the big model into the small model by teaching teaching it at which

Átirat · 1:31:03

level do I teach it? I can teach it at the level of features or at the level of the task outputs.

Átirat · 1:31:09

So uh the key here is that the task level for example usually has to be soft

Átirat · 1:31:15

labels. So the teacher is not saying there's a car here. It's saying also how confident it is that that is a car. 80%

Átirat · 1:31:21

that it's a car and the softness has certain seems to be the key. So there's

Átirat · 1:31:26

research also in self distillation where the teacher and student are the same. So not exactly this setup where actually

Átirat · 1:31:33

they show that this softness of the labels from the the big model to the smaller model is very very important as

Átirat · 1:31:40

a regularizer and they even have some mathematical guarantees on this. Other techniques are other techniques

Átirat · 1:31:46

are related to finding the best architectures. This is code architecture search best means efficiency versus

Átirat · 1:31:52

accuracy. So balance of the two uh quantizing the model properly as I mentioned we have few bits and also for

Átirat · 1:31:58

scalability being able to predict how how fast the model would be in your search space. Okay, cost. I'm not going

Átirat · 1:32:05

to go into much details. There's not much time left. Um, but here I just want to mention that uh, you know, you want

Átirat · 1:32:11

to use all the data including unlabelled data. You know, labeling is prohibitive in today's world. So, you can do a lot

Átirat · 1:32:18

of smart stuff. Uh, you can actually uh label only uh the interesting examples.

Átirat · 1:32:24

For example, you start with a small set of label data. You run the first training. get a model and you ask your model where do you think you're you're

Átirat · 1:32:31

not very certain. Uncertainty estimation is also related to safety. You see a link between the

Átirat · 1:32:37

two. The same techniques can be used uh you know getting a model to determine how confident it is in a certain example

Átirat · 1:32:45

and there's a certain problem there open problem in the research how you do that properly the calibration problem for

Átirat · 1:32:51

those of you who are familiar with that. Uh what you can do further here is also have several annotators and each

Átirat · 1:32:57

annotator has a different noise model. You know each person has has certain

Átirat · 1:33:02

idiosyncrasies. You can also have automatic annotators. So you might want to also select which annotator you're going to send an image to. Maybe even

Átirat · 1:33:08

ask a second annotator a second opinion like when you go to the doctor, right? So then you have to estimate the noise

Átirat · 1:33:14

model and there's another field of learning with noisy label that's related also to active learning. Again a connection in terms of techniques

Átirat · 1:33:20

between them. Um yeah and there's other like federated learning actually means training from different locations when I

Átirat · 1:33:26

know China doesn't want you want to let you get the the data out of uh of the country and then you can do certain

Átirat · 1:33:32

tricks to train at both locations. I won't go into details but this is actually quite challenging

Átirat · 1:33:38

and my last slide. Thank you. Um so basically what I my main message as I

Átirat · 1:33:44

mentioned before is that you will uh get this biased team everywhere. So remember

Átirat · 1:33:51

that your experience from the previous era is actually transferable to the new era. Right? If you're trying as I

Átirat · 1:33:57

mentioned before to get the right uh the right hawk features let's say designed them now you're going to ask a certain

Átirat · 1:34:02

invariance and inject it or you can do for example that levy shown right he was doing date augmentation with certain

Átirat · 1:34:08

homographies that's a bias also I'm saying the homography is what I want to you to be invariant too. So the same

Átirat · 1:34:15

insight from the old days gets still lives today. And then there's what the

Átirat · 1:34:21

last message that I want to give you is that there's a gap. There is a gap between the research community and the

Átirat · 1:34:26

industrial community. Certain areas like for example safety or the calibration problem probably need more attention or

Átirat · 1:34:33

at least we need more help from you. So that's why this this I think this is

Átirat · 1:34:38

very important so that uh you you see these applications and still focus on the fundamentals at the same time because that's where the key is.

Átirat · 1:34:45

I think that's it on my side. Thank you very much.

Átirat · 1:34:52

[applause] Thank you very much. And uh we came to a kind of a conclusion

Átirat · 1:34:58

uh now uh before the break and uh I think what I would like to ask first

Átirat · 1:35:05

maybe we have some questions uh to the first four speakers uh before the break.

Átirat · 1:35:10

Anyone has some certain questions or we can ask uh we will be around here so uh

Átirat · 1:35:15

we can ask it in here as well. Is that better? It sounds better. Okay. And uh

Átirat · 1:35:21

please don't go away. Uh there's outside a little bit of uh discussion room. Uh

Átirat · 1:35:27

we'll come back uh 4:00 is that right? So we have uh 20 minutes of break

Átirat · 1:35:33

something like that. And I would also like to have all the speakers already who spoken and who are going to speak uh

Átirat · 1:35:41

to hear for a common photo. Okay. So see you after the break. Thank you.

Témák

MI-támogatott

Összefoglaló

MI-támogatott

Ștefan Máthé az autonóm és támogatott vezetés percepciós rendszerét mutatja be az érzékelőktől és beágyazott számítási platformoktól a reprezentációkig és feladatspecifikus fejekig. Áttekinti a klasszikus gépi tanulástól a mélytanuláson át a foundation modellekig vezető fejlődést, majd négy ipari kihívást emel ki: reprezentáció, biztonság, hatékonyság és költség/adatigény. Példái között szerepel az önfelügyelt tanulás, a JEPA, a tudásdesztilláció, a kvantálás, az aktív tanulás és a federált tanulás; fő üzenete, hogy az induktív bias minden korszakban megmarad, és a kutatás és ipar közötti rések – például biztonság és kalibráció – továbbra is nyitott problémák.

Videó navigáció

Fejezetek

MI-támogatott