Előadás

3D Foundation models and Car Design

Beyond Artificial Intelligence Research: a New Innovation and Industrial Domain

2025. november 7. 13:00Babeș-Bolyai University, Central Building, Aula Magna · Kolozsvár · Románia

Átirat · 0:00

Everyone uh this is the conference for uh artificial intelligence uh beyond

Átirat · 0:07

artificial intelligence research a new innovation and industrial uh domain. So

Átirat · 0:12

this is what we are talking about and after the break the first speaker is

Átirat · 0:17

Bonze from Foxswagen uh data lab Munich

Átirat · 0:23

who is a local uh anyway but uh come come uh came here to us to share recent

Átirat · 0:30

ideas. So the floor is yours. Thanks a lot. Um thanks for the invitation. Happy to be here. Um my talk

Átirat · 0:36

is going to be a bit um more technical um because I think I've prepared it with

Átirat · 0:43

a different room in mind. So um I'm going to talk about recent advances in

Átirat · 0:49

uh 3D generative models. Um okay. So um uh I'm going to show you two

Átirat · 0:56

papers and where the actual what's the actual state of this domain and

Átirat · 1:03

basically these are two foundation models that are used almost everywhere lately and we are also using it within

Átirat · 1:09

the company to uh build applications around and to fine-tune them and so on.

Átirat · 1:16

So it's not going to be anything particular to Volkswagen but I'm going to touch upon the things we are going uh

Átirat · 1:24

and we are doing there with these models. Um, so if you just open your

Átirat · 1:29

phone and then you have a very nice picture of yourself and you would like to know how this code is going to look

Átirat · 1:36

on me, then you can go to Google and

Átirat · 1:42

take the guy from the website, take yourself and then basically just ask transfer the code to myself or something

Átirat · 1:48

like that and then there you go. You can have a code which basically is transferred for the first one to the

Átirat · 1:55

second one. There are a bit of glitches. As you can see, the position is fine, but it also transfer the hands with the

Átirat · 2:02

code. So, it's a bit tricky, but it still does something meaningful. And so, uh there are also other models,

Átirat · 2:09

for example, Flux, which is like uh one of the top image generation models, text

Átirat · 2:14

to image generation model, and there's also multi model lama, where you can basically plug in a picture and ask

Átirat · 2:21

questions about the picture. So um these are models that are trained with um

Átirat · 2:27

billions of image and text pairs and um this is one of the disadvantages of the

Átirat · 2:36

uh 3D models because they actually don't have this amount of data. So people

Átirat · 2:41

don't go around during their holidays create 3D models upload them on Instagram and Facebook. So whatever um

Átirat · 2:50

these companies have in terms of data, it's um very unlikely to

Átirat · 2:56

be gathered in the same way. Uh for 3D data also it's a bit tricky because

Átirat · 3:02

everything that surround us from the uh components in your mobile phone to the

Átirat · 3:07

rubber on your shoes has a 3D model somewhere because someone has created a CAD model had to print it had to

Átirat · 3:14

manufacture it in some way. So these models are sitting around. We just can't use them because evidently these are

Átirat · 3:22

under uh patents licenses and people don't give don't want to give them away for free. So what can we do then? There

Átirat · 3:29

are some data sets which are like objiverse which has a bunch of 3D models

Átirat · 3:35

but these are mainly from uh video games. Then also there's an

Átirat · 3:41

Amazon black objects data set. There's toys. Some people have tried to reverse

Átirat · 3:46

autos basically take the cars take them apart uh scan the components

Átirat · 3:52

and then we also have a bunch of uh CAD models which are probably the most important ones uh but there's very

Átirat · 4:00

they're very simple and there's also a very limited amount of data so there's only for example you have 10 million

Átirat · 4:06

toys I mean models from games and you only have around 700,000 of CAD models

Átirat · 4:12

which are simple components like screws or small little boxes and

Átirat · 4:17

box-like objects. So what can we do with this data? So people nonetheless try to

Átirat · 4:23

do something with it and um uh came up with some nice models. So

Átirat · 4:30

first of let's focus on what exactly the data is. So data can come in a lot of different flavors. Um one of them is um

Átirat · 4:38

so the easiest one is just a bunch of pictures from different angles and then

Átirat · 4:43

a scene can be represented as a collection of these images uh with known positions. I mean and uh

Átirat · 4:50

the second one this is can be used for learning scenes and rendering them but

Átirat · 4:57

it's not that useful. What is most useful is is for example point clouds that are gathered by light liars and all

Átirat · 5:04

kinds of other uh scanning procedures and often come as points uh with

Átirat · 5:12

sometimes they come with normals sometimes they come with intensity values and so on. So the next most

Átirat · 5:18

important thing that's really um useful in industry are these uh two. So there

Átirat · 5:25

are the meshes which basically um have a proper geometry. So there are faces, vertices, edges and faces. And

Átirat · 5:33

the best ones are obviously used the CAD models that use in that they use in computer data design. So there you have

Átirat · 5:39

vertices, edge curves, faces and these are all specified by exact models like

Átirat · 5:45

spline patches and so on. Um so this is the this is how the data comes to us and

Átirat · 5:51

then uh the question is like what can we do with this? How can we represent these things in a computer? Right? So um so

Átirat · 5:59

then we come to representation which is a bit different than what Stefan said earlier. So this is going to be my

Átirat · 6:07

representation and the representation he kindly introduced is in my talk is going to be something called the latent space.

Átirat · 6:14

Um so these are like um implicit fields which are basically like um a surface in

Átirat · 6:21

geometry. it has positive values on a positive side of the surface and negative um side on the negative side of

Átirat · 6:28

per of the surface. So all we want to do basically is learn a scalar field and then it's can be a neuronet network

Átirat · 6:34

which we query with a point and it's going to say okay is it on the positive or a negative side of the surface. Um so

Átirat · 6:41

the next one is 3D gaussians. So these are similar to point clouds. The their advantage is

Átirat · 6:47

that uh in the same way like point clouds we have a bunch of points in the space but

Átirat · 6:54

instead of g instead of points we have proper gaussians which have color values shades and so on and so we can represent

Átirat · 7:01

a scene by a bunch of little gaussians spread over the space and it's super easy to uh render them because we

Átirat · 7:08

basically create a plane we project all these gaussians on it and there we go. So we can have a scene and render it

Átirat · 7:15

super fast. And so in my talk I'm going to focus on the other ones as well but

Átirat · 7:20

in my talk I'm going to focus on these do these two meshes and the Gaussians.

Átirat · 7:26

Okay. So why would we want to use this in industry? It would be nice to use it for example to turn

Átirat · 7:33

u sketches into design. So in the design we would also like to use them to create

Átirat · 7:38

objects in a simulation. Right? So you have a simulation running. I need something. I look at an image, click on

Átirat · 7:44

it, I create a 3D object and put it in my simulation. Um, now it depends what I want to use

Átirat · 7:50

for like sometimes I want to use for only for rendering so that in the simulation I have a rendering and I have an image and I can use it for something

Átirat · 7:56

else and some other cases I want actually want to use it the object to interact with something. So in for

Átirat · 8:03

example finite element methods, crash test simulations and um like those type

Átirat · 8:08

of simulations and so on. So the current problems with these models are that

Átirat · 8:14

whatever we generate is not accurate enough. The geometry they generate especially the meshes come with a lot of

Átirat · 8:20

dis disconnected components some triangles that are quite degenerate. So

Átirat · 8:26

they don't have nice uniform faces and the 3D gaussian useful for rendering. We

Átirat · 8:34

cannot actually do any anything with them. So good. So the question is like uh how do

Átirat · 8:42

we generate data and the two things that we are looking at is how do you generate data from images and how do we generate

Átirat · 8:48

data from text and possibly how can we generate

Átirat · 8:54

something and then reactively manipulate say I want to change something on this

Átirat · 8:59

model and so on. Um so let's jump into this. What's the goal is what are

Átirat · 9:05

generative models. Generative models are basically um some sort of mathematical functions

Átirat · 9:13

that takes a code or condition and it gen and add some stoastic source and it

Átirat · 9:19

generates something like for example in this case the digits like we have a for example a Gaussian noise we put it to

Átirat · 9:25

this uh and the digit that we want to generate we put it to this generator which is just a function or a neural

Átirat · 9:30

network or whatever it is a universal approximator and then basically we want to generate sample that are similar to

Átirat · 9:38

the data we have seen. And there are a lot of ways to define similarity.

Átirat · 9:44

And in most cases, if we have a property distribution that we can actually compute, then we can do max maximum

Átirat · 9:51

likelihood. We just tune the parameters such that we plug in the data in the model we generate. If that's not

Átirat · 9:58

possible then we can need you we can if we can only sample for example then we can you uh use uh discrepancy measures

Átirat · 10:06

all kinds of empirical estimates of differences between property distributions um and based on samples. So for example

Átirat · 10:13

we sample a huge batch then we take a huge batch from the data then we have two distributions given by samples and

Átirat · 10:20

then we try to compare them and then based on that we define some function that allows us to minimize the

Átirat · 10:26

parameters of the generator. So that's the story. Um and then also it's

Átirat · 10:32

important to point out that the condition right is and what condition is

Átirat · 10:38

and why. So if there's no stoastic noise right then we cannot generate different objects. So we have to put in some

Átirat · 10:46

stoastic u source to generate different images and um then we have a code and then we

Átirat · 10:54

can we have the freedom to choose what goes where. So if you notice that we have a bunch of slanted

Átirat · 11:01

u um digits then we have u border lines. So we can also put this one in the code

Átirat · 11:07

if we want to and it goes along with whatever Stefan said that we want to disentangle important um

Átirat · 11:16

um properties of variation and so on. Good. So what we want to do this with this one now we want to generate SDF or

Átirat · 11:24

some field that describes our object and ultimately we want to go through a bunch of boxes query this um field and then

Átirat · 11:33

create a mesh. There's an algorithm called margin cubes that can do that for us. All right. So um how exactly happens

Átirat · 11:41

this in current models? Typically as a generator we have a neural network and then a lot of people so that um very

Átirat · 11:51

deep neural networks have been used in the recent times that take a lot of uh space and a lot of time to train and

Átirat · 11:59

um learning happens through some sort of gradient method. And it often happens that for large networks these kind of

Átirat · 12:06

gradients either vanish or explode and learning can be very hard. So for that

Átirat · 12:11

reason people try to come up with some methods that don't have to propagate through the whole network. They just

Átirat · 12:17

have to train locally a bit of the network one single block one single layer and so on. And they ended up

Átirat · 12:25

funnally with u ordinary differential equations. And

Átirat · 12:30

basically what they do the whole algorithm is like four lines of code and

Átirat · 12:36

um these are the best model now models nowadays and then rely on four lines of

Átirat · 12:42

code. So what they do is basically say that I want to take any data point see

Átirat · 12:47

how it transforms to u so so he see how uh standard normal noise or a standard

Átirat · 12:54

normal distribution transforms to this data point draw this line and then try to basically every time I draw a new

Átirat · 13:00

line sample a noise draw the line and then try to learn the field at an

Átirat · 13:06

associated point right so this doesn't require by propagating the whole generator but it only requires to

Átirat · 13:13

evaluate it at a certain point. So what we learn is a field that can basically

Átirat · 13:19

at any time and any special location can point us towards something that is similar to the data. Nope. So it's that

Átirat · 13:27

simple the whole thing and on top of that there's also something that Stefan

Átirat · 13:32

mentioned. So the out encoders so it's really hard to generate

Átirat · 13:38

u HD quality images uh without choosing some sort of

Átirat · 13:44

relevant smaller representation. So what happens is that typically we employ an

Átirat · 13:49

out encoder. First we compress the data into some smaller more manageable size

Átirat · 13:55

and then we learn a flow model from it. Right? So we learn how to generate um

Átirat · 14:02

data effectively and so this is exactly what happens

Átirat · 14:08

nowadays. Um so we have billions of parameters only for one single bit of the network and then we reuse that

Átirat · 14:16

basically all the time. Good. So one of the basic components of these ones are

Átirat · 14:22

uh potential layers um which are like um

Átirat · 14:30

composed of three things. The value a key and a query. So usually value and key key go together. They characterize a

Átirat · 14:38

set a data point and the query it can be also depend either on the same data or some other data. And what we do is we

Átirat · 14:46

basically want to generate some um weighted average of the values given how

Átirat · 14:55

the keys fit to the queries. Right? So if you have um query that is similar produces a high

Átirat · 15:03

similarity with a certain key then we pick that part of the data. So this is how we go through for example

Átirat · 15:10

uh a sentence right and then picks the next word. So we look into what's the

Átirat · 15:16

best next to generate we look into the history and what is gives a high score we are going to output that word for

Átirat · 15:22

example in case of images uh this holds for features and so these are like built into blocks

Átirat · 15:29

and then huge models are created but essentially um these are the three components

Átirat · 15:35

something a flow that generates the data an autoenccoder that generates the latence an autoenccoder that

Átirat · 15:42

autoenccodes the data and ton of attention blocks that basically define the architecture of these models. Okay,

Átirat · 15:50

so um currently there are two models out there that are really nice and first one is trellis that's from Microsoft and

Átirat · 15:58

there's another one. So trellis is has available code you can find download it

Átirat · 16:03

fine tune it play with it and tunan is only available for um inference so you

Átirat · 16:10

can use it but you cannot do anything other than that with it you cannot train it because the training code is not available um so we kind of use the first

Átirat · 16:20

one and then try to figure out what the second one does um so what is trellis about so that came

Átirat · 16:29

out in December uh last year and it's pretty recent. Um so what it does it

Átirat · 16:35

basically uh takes any type of a lot of types of data most of the times it's a mesh and a

Átirat · 16:40

bunch of renderings of the mesh. So what it does around 50 150 directions and

Átirat · 16:46

what it does it takes the mesh it voxalizes it. So it creates basically a voxil space and then based on the images

Átirat · 16:54

the multiv- views it creates it uses Facebook's dynino model to create features. So for each vau you have a set

Átirat · 17:01

of features and then these are then encoded into some sparse tensors

Átirat · 17:08

and then decoded. Right? So what we end up with is almost the same as the

Átirat · 17:14

original object but instead we have features attached to each boxer in the

Átirat · 17:19

space right and then now so once we are done with this is the encoding the out

Átirat · 17:25

encoding part you can see that you can take meshes and images and it produces

Átirat · 17:31

SDFs like as meshes it produces um the gaussian I've mentioned that are useful

Átirat · 17:37

for rendering so it produces different types of data that you can use for different applications. ations and so

Átirat · 17:42

how does um prediction happen? So the generation the actual generation it has

Átirat · 17:48

two stages. It also has an autoenccoder for the actual structure of the mesh right. So it has like a compact

Átirat · 17:55

non-sparse representation in terms of tensors and um it has an autoenccoder

Átirat · 18:00

for that as well. So first it basically takes some stoastic source then it

Átirat · 18:05

transforms it into some latent data that can be then that

Átirat · 18:12

from which we can generate the vauo structure first right and as a next step

Átirat · 18:17

then it goes and populates the structure with some features then from that

Átirat · 18:23

feature finally goes to the decoder and it creates the actual data that you can we can use and then so these are all

Átirat · 18:31

trained with conditioned images on conditioned on text and images. So you

Átirat · 18:37

can choose [snorts] whatever you want to whatever conditioning data you want to

Átirat · 18:42

use. Um so I'm going to show you examples that are like uh mostly on images

Átirat · 18:48

um because that's what okay architecture is a bit complicated but it's like very

Átirat · 18:53

standard. So you can see there are a bunch of attention layers and the model was trained with not that much data. So

Átirat · 19:00

you can see like up to 12 million um 3D objects

Átirat · 19:07

and then the boxer resolution is like 64 cube but then the act of the latent

Átirat · 19:15

space but then the actual resolution you get for this SDF function that creates the mesh is like uh 256 cubes.

Átirat · 19:24

Okay. So, you can generate a lot of fun stuff. You can put in images and then

Átirat · 19:32

generate 3D models. You can put in text and generate 3D models. Uh so, for example,

Átirat · 19:39

we can take three images of a car and then you can create a 3D model of a car.

Átirat · 19:46

Then we can put sketches in and then we generate and same thing you can download

Átirat · 19:53

some shoes from the internet create 3D models and then you can also download

Átirat · 19:59

something like that. See what it does with it. This is actually really funny because um this similar data surely is

Átirat · 20:07

not in the data set but the the obje has a lot of 3D models for games. So I

Átirat · 20:12

suppose this image turns into something like that. Okay. So the other one is

Átirat · 20:19

this one. I'm not going to have much time for that. But this is a different representation. This uses a point

Átirat · 20:25

cloud-based representation um tokens like Stefan mentioned before.

Átirat · 20:30

So instead of walkers we have um a set of tokens and it does the similar thing

Átirat · 20:36

and it ends up um with a latent that you can query for

Átirat · 20:43

different uh box values and it gives you the STF

Átirat · 20:49

value then you can turn it into a nice surface. So same architecture. It can

Átirat · 20:55

generate some really nice 3D models from images and then so this is basically a

Átirat · 21:01

comparison of the trellis and this better model. Um so you can see for

Átirat · 21:07

example that here it kind of knows that he has to generate someone with a leg

Átirat · 21:13

but the and the shoe but the shoes go on the horse for example in this case. Right? So it sees that there's a person

Átirat · 21:22

and but it generates the shoes for the horse. So funny stuff. But it knows how

Átirat · 21:27

what the back of the horse looks like. It kind of knows that the person has to be somewhat symmetric, right? So it

Átirat · 21:33

generates those parts as well. So this is u but yeah still the head is like

Átirat · 21:40

quite close to the body here. Here it kind of differentiates and it creates a better horse. You can see the back is

Átirat · 21:47

also much nicer. Uh, so it's a definitely a better quality model.

Átirat · 21:53

Um, this one knows what's in the head and turns it into a crown because it's

Átirat · 21:58

probably a king or something. And this one is a bit a bit better in

Átirat · 22:04

that respect. Okay. So, the second model can also generate a texture on top of

Átirat · 22:09

the mesh and then it can produce some quite nice stuff.

Átirat · 22:16

Yep. Something like that. Good.

Átirat · 22:23

So this is where these models are.

Átirat · 22:29

So at the moment they are really cool to play with. Uh it can be used to help you out. Um it can help with the design to

Átirat · 22:37

some degree if you have sketches and so on. But then um it really lacks the

Átirat · 22:42

quality to be used in engineering applications. And what would be great if we could have

Átirat · 22:48

CAD models to do these things but we can only have quad models within the company

Átirat · 22:54

and then but then we don't have the software for it yet. Uh and then we

Átirat · 23:00

cannot train a model like this within the company unfortunately. So these are like I think we would have the money

Átirat · 23:07

because you can train one of these like for half a million or so on but no one

Átirat · 23:13

wants to give it to us so we can't do it. [snorts] So we are uh basically fine-tuning stuff

Átirat · 23:20

and then creating digging into the latent space building applications around and so on. Thank

Átirat · 23:27

you. That was

Átirat · 23:35

Thank you very much. Uh a little bit of uh

Átirat · 23:40

technicalities I'm going on and uh Andre uh in the meantime please come and help me.

Átirat · 24:00

This one is

Átirat · 24:17

the original. So our next speaker is Andre Mihi uh from this university uh

Átirat · 24:25

University of Babes Bon Bet. He is a a researcher lecturer here at the M

Átirat · 24:32

machine learning research group. So the floor is yours. Okay. Thank you.

Átirat · 24:38

Just Yeah.

Témák

MI-támogatott

Összefoglaló

MI-támogatott

Cseke Botond a 3D generatív modellek és foundation modellek jelenlegi állapotát mutatja be, különös tekintettel ipari és autótervezési alkalmazásaikra. A 3D adatok szűkösségét és reprezentációit – többnézetű képek, pontfelhők, mesh-ek, CAD-modellek, implicit mezők és 3D Gaussianok – áttekintve elmagyarázza a generatív modellek, autoencoderek, flow-alapú módszerek és attention szerepét. Két korszerű 3D modellt hasonlít össze, köztük a Trellist, és rámutat arra, hogy bár képből vagy szövegből látványos 3D geometriák generálhatók, a jelenlegi modellek pontossága és topológiai minősége még nem elegendő mérnöki CAD-felhasználásra.

Videó navigáció

Fejezetek

MI-támogatott