Előadás

Explainable AI with prototypes - methods and challenges

Csató Lehel

BBTE

Beyond Artificial Intelligence Research: a New Innovation and Industrial Domain

2025. november 7. 13:00Babeș-Bolyai University, Central Building, Aula Magna · Kolozsvár · Románia

Átirat · 44:28

Um, hello. Wow. So things work fine.

Átirat · 44:34

Uh yeah, thanks for being here. Um I'm from the from the local university babes boy

Átirat · 44:42

and u I speak on behalf of a team which you see on the right hand side.

Átirat · 44:49

Hopefully you can see them all. Uh some of them are here, some of them are possibly online. Uh some of them are

Átirat · 44:56

were here. uh like subwo uh they have former team team members and subwo

Átirat · 45:01

please consider the three dots uh on the line line behind that's uh that is yours

Átirat · 45:08

um uh the talk that I'm trying to speak uh trying to give today is about

Átirat · 45:14

something we started a few years ago it's called prototype based decision

Átirat · 45:20

making systems and um yeah let's let's see just the

Átirat · 45:25

just the content. So I will uh I will structure it as as something like a

Átirat · 45:32

motivation why why is a a decision has to be justified what are the benefits

Átirat · 45:39

what are the drawbacks then I will I will have a one or two two slides about

Átirat · 45:46

why is important from a from a legal perspective if a little bit zoom zoom out from our technical fields and then I

Átirat · 45:53

will try to speak a little bit about the the technical it is uh hope I won't be

Átirat · 45:59

too long anyhow I was told that at six uh no at five sharp we should stop I

Átirat · 46:06

think that this was well okay if not then then I'm a little

Átirat · 46:11

bit little bit happier hope I won't drag you too too long on it so anyhow and I

Átirat · 46:17

will draw some conclusions which are mine and does not represent the group or

Átirat · 46:23

anyone of the team members So first I would I would like to I would

Átirat · 46:28

like to start with an with the with the issue of the explanability like every one of you knows the number

Átirat · 46:35

42 which is which is the greatest number of all. Um it says that um that's the

Átirat · 46:41

answer of to the question what is the meaning of life the universe and everything and what he said it's the

Átirat · 46:50

computer was saying 42 and he said that after 10 millions of years of computing and I find it really metaphorical the

Átirat · 46:57

fact that uh today as of today we're building um computer b compute bases

Átirat · 47:04

that are can do literally this 10 millions of years of computing in a matter of months

Átirat · 47:11

and the promise is that they will be even faster. So then then when the computer gives us

Átirat · 47:18

the 42 and just remember in the in the matter of months the question is so what

Átirat · 47:24

why 42? What does 42 represents? And even in the in the in the hitchhiker

Átirat · 47:30

guide to the galaxy by Douglas Adams u this question it says that it would require another 10 millions of years of

Átirat · 47:38

computing but the fact is that the people did not know what they want to ask in the first time f the first

Átirat · 47:44

instance. So uh just to put it back into into much smaller scale um we all know

Átirat · 47:52

that neuronet networks are are black boxes. So uh what do we do when when we

Átirat · 47:58

see a decision? What what how do we interpret it? Are we able to say if

Átirat · 48:04

things go wrong did they go go wrong? So so the these are the questions that we were trying to uh trying to to to

Átirat · 48:12

answer. And if we focus on on the image classific on the image and decisions

Átirat · 48:17

related to the images for example we might say well why is that system saying

Átirat · 48:24

that the road is muddy because I see there are some patches which which have a certain uh specifics

Átirat · 48:30

or let's see that and this is a just I I um this the images are little bit old

Átirat · 48:37

and and for for most most of the of the people that that work with the this is looks familiar. here. So for example,

Átirat · 48:43

what I what I want to say is that you see there are three three images and the three images the top is represents the

Átirat · 48:50

motorway and the second is that the rural type and below is something which

Átirat · 48:57

is which we call urban type and the well the the heat maps they saying that

Átirat · 49:05

probably the the reason why the classification is such because the heat maps indicating it. So we want to

Átirat · 49:13

achieve something like that or uh this is this is work done by um um Hana and

Átirat · 49:19

Tomas. What what do we want? What we do want is that for each class of m monkeys along

Átirat · 49:27

the lines of saying uh of saying that is the first type of monkey. The second

Átirat · 49:32

type of monkey [snorts] I'm not a biologist and I think the first line is an orangutan. Uh the rest you no uh

Átirat · 49:41

thanks. So no it's not even even that I don't know but essentially it's the the

Átirat · 49:48

the boxes which of which I highlighted only the the ones that on the first

Átirat · 49:53

column they saying that we found that as a justifi just justifying uh codes for

Átirat · 49:59

that image being of a certain class. So just remember next next to the fact that

Átirat · 50:06

we have we have images belonging to a class what we want to to achieve is to to see why are those images belong to

Átirat · 50:13

the class and and and we want to achieve that adaptively. So

Átirat · 50:18

um this uh justifiability also comes comes from from from a more general

Átirat · 50:25

point of view and if when I'm speaking about decisions I'm speaking about about

Átirat · 50:30

decisions in the banking world for example you might you might have a loan

Átirat · 50:36

uh approved or you might have a loan approved with a with a much higher interest and it's completely justified

Átirat · 50:42

that you're asking why do I get get this these these these numbers And uh as of

Átirat · 50:48

today um essentially this uh act of uh EU EU AI act European Union artificial

Átirat · 50:57

intelligence act what it says is that unless you have systems that have that

Átirat · 51:03

that have no risk on your financial and health and other status. uh you have to

Átirat · 51:11

you have to have a reasoning and you have to show every details that that led to the to the conclusion that you are

Átirat · 51:18

subjected to. So so this this kind of this kind of of an of an explan

Átirat · 51:24

explanation or explanatory mechanisms they are quite quite important for also

Átirat · 51:30

legal legal perspective. Um and in the in the second part of the talk I will try to speak about about

Átirat · 51:39

some uh specifics a little bit more tech technical technical issues that we we um

Átirat · 51:45

we tried out and we we had in the in the [clears throat] explaining of decisions

Átirat · 51:52

when they were related to the to the images. So what we what we did was kind of an an

Átirat · 52:01

like whether we can do we uh we tested whether we can do an an architectural

Átirat · 52:08

modification to a to a given simple system and this architectural

Átirat · 52:13

modification should be should have the explanation and those of you have who

Átirat · 52:19

are familiar with it's h something in contrast with something which we call the post hoc methods this is like ad hoc

Átirat · 52:25

methods and a a priority method. Sorry. So a first first uh of the the methods

Átirat · 52:32

if that you see is something which appeared in the talk of Stefan and then

Átirat · 52:37

later in in Botton Botton speech was that we should consider something some

Átirat · 52:43

some feature vectors and what what you see there is on the on the red on the on the right side of the image and these

Átirat · 52:49

features vectors there feature vectors they somehow encode the image and this is called on it's a you can you can

Átirat · 52:56

imagine like a loop on the on the left hand side of the image And it says please uh can I can I have a features

Átirat · 53:03

extracted added to these features there's something which we call uh which we the

Átirat · 53:09

the system takes away from some some images and takes away something which is which are called prototypes essentially

Átirat · 53:16

a representation of the image and it says that this image is is so important is so prototypical for the for the

Átirat · 53:23

specific class that I better keep it and Um, [clears throat] also Stefan was was

Átirat · 53:30

saying some some something about the history of the of these uh line of research already in the 1980s that

Átirat · 53:38

appeared as an something we we call the radial basis function networks RBFs

Átirat · 53:43

which had a the notion of a center and everything that was the so the decision

Átirat · 53:49

was based on the on the set of centers and some combination of it usually a an

Átirat · 53:56

exponentiated a Gauss bear curve and and then a a linear summation weighted sum

Átirat · 54:02

summation. So just to just to just to to to get how these systems work, it's

Átirat · 54:10

specifically instructive and in this case if the the further away you are

Átirat · 54:15

from the image, the the the the better you can glimpse that on the left hand side of of of the bottom image is a I

Átirat · 54:23

think is a puppy is a small dog. Uh and um what you see is that the uh

Átirat · 54:33

decision whether there's a puppy or not there's a there's a dog there or not is based on the reconstruction on the

Átirat · 54:40

feature of of which is reconstructed as that blurry image that what is seen. So

Átirat · 54:47

the conclusion is that these systems these prototypical systems even even if

Átirat · 54:54

it if you try to do it well or or or very very well um they essentially they

Átirat · 55:00

have a kind of a color and much less a texture but but a shape

Átirat · 55:05

based um uh features as a decision. So

Átirat · 55:11

the improvement that was made is to somehow to consider regions of a of a

Átirat · 55:18

bigger image and there you see you see there um I think is a finch or

Átirat · 55:23

something. uh re so to consider regions from a from a from an image and uh

Átirat · 55:29

essentially this these features ve feature vectors of which you saw the in this uh autoenccoder type um um networks

Átirat · 55:36

of the latent representation these feature vectors somehow should relate

Átirat · 55:41

also to the topologies to the location of the of um of those features in the in

Átirat · 55:47

the input image. So you can you can you can also see that that in this um in this uh um the

Átirat · 55:57

prototypes they they are not only encode the image but they also encode the region of the image that is that is

Átirat · 56:04

interesting to us. So, uh what you had to see is is that uh

Átirat · 56:12

also they they work very very close to the to the um to the um autoenccoders

Átirat · 56:18

and also the uh prototype layers added to that this kind of locations

Átirat · 56:25

um to the in in these images and u again

Átirat · 56:30

what I what I want to show is that um here we didn't need the reconstruction

Átirat · 56:35

construction. But what we had what we could do is that um we had the kind of a

Átirat · 56:41

classification and uh you see the three classes there on the uh vertical axis

Átirat · 56:47

and the correct was the rural class which is shown in in um green and what

Átirat · 56:56

we see is that we could we could also see the justification. Um

Átirat · 57:03

so what what was what was wrong is that very often we had we we did this uh this

Átirat · 57:09

uh classifications they were they were not not correct but nevertheless they

Átirat · 57:15

were pretty certain that that that class uh is urban and and it should should

Átirat · 57:21

have been rural in uh in this case. So what we've done so is that was the um

Átirat · 57:30

the conclusions that that we drew is that while this

Átirat · 57:35

uh prototype networks they in spite of fact them them being good it's uh that

Átirat · 57:41

we might we might do some some some other uh we call it improvement uh and

Átirat · 57:47

[snorts] one specific type one specifics of the system that we we we attacked uh

Átirat · 57:53

was to see some to see the the non nonlinear um type of the of the backbone

Átirat · 58:01

and we said that why not having it uh uh something which we call locally linear

Átirat · 58:07

and uh uh do the the composition along those those lines. So it's a it's a

Átirat · 58:14

protoet system that we that we u uh

Átirat · 58:19

developed. It's done with Tamas Costa and Halib and she she she's present here

Átirat · 58:26

and then acknowledges to thanks to her. Uh and what we've done is that we we

Átirat · 58:32

used kind of a rectangular regions that you see on the on the on the image. It's

Átirat · 58:38

a monkey. I think it's a bonobo. Sorry.

Átirat · 58:44

Mandril. Thank you. So it's a it's a mandrel. And uh uh this uh every every

Átirat · 58:50

uh rectangular area is decomposed so so that it it has a feature uh assigned to

Átirat · 58:56

it. Uh the good part is that the decomposition is such that it's it is

Átirat · 59:03

invertible. So if you if if you speak mathematically

Átirat · 59:09

mathematically that invertability is good because we can immediately

Átirat · 59:14

identify. So we have no problems with explaining of why why is there we can always show it and uh uh we've done it

Átirat · 59:23

with with a with a wave that de composition that was supposedly working well for uh for images and this is this

Átirat · 59:32

is a kind line of research that is done by mathematicians like doashes or or um

Átirat · 59:38

Stefan Mala um just to just to name a few and

Átirat · 59:45

What was that? We could we could identify next to the correct classification, we could identify the

Átirat · 59:51

the the markings. And hopefully you see it's a is a is a little patchy, but

Átirat · 59:57

there's a there's an orange or a purple a purple uh rectangle around the around

Átirat · 1:00:04

the u the landmarks or the reasonings of why why it was the uh the uh decision

Átirat · 1:00:11

was was was made. Of course, you also had incorrect incorrect cases.

Átirat · 1:00:20

But uh overall, what uh what we saw what we said that these all these methods

Átirat · 1:00:26

that we that we employed I mean the protoet and the and the and the wave wave protoet they were achieving results

Átirat · 1:00:34

that are that were very similar uh on a test case of this data set. I'm just I'm

Átirat · 1:00:40

just foreshadowing that it's uh it was it was from the from the Bosch

Átirat · 1:00:46

company. Uh so the results from this data set were were um the losses that as

Átirat · 1:00:54

Mihi was saying they were a little bit better than than those of the protoet.

Átirat · 1:01:00

Uh so just just a few conclusions and once again these conclusions are mine.

Átirat · 1:01:06

Um, of course that is uh what that what what we had that the prototype based

Átirat · 1:01:12

architectures are a possible direction to explainability

Átirat · 1:01:17

and again it's better that the prototypes to be simple. Um and that is a that is a that is a

Átirat · 1:01:24

good sign and is is a is a bad bad um sign also. For example, it is not able

Átirat · 1:01:31

to recognize rotations the way it was u uh implemented.

Átirat · 1:01:37

And um why is that important? uh not for the for the drivers but if you consider

Átirat · 1:01:44

medical images uh for example what we're doing is uh salivation images

Átirat · 1:01:50

um and the crystallization then that's the rotation invariance

Átirat · 1:01:56

it feels like it it it it is a must um also um from a from a teaching

Átirat · 1:02:06

perspective I I I have also some some some conclusions. Uh we coded in a in

Átirat · 1:02:14

something which is which which is not Python. So we did not have the enormously large um code base and

Átirat · 1:02:21

library for that. Uh we used we use something which is called Julia. Um it's

Átirat · 1:02:29

very close to to uh to my heart because it's a kind of a function functional

Átirat · 1:02:35

like programming and it's very close to mat lab which most of you know or most of you know with a with an epos from the

Átirat · 1:02:44

statistics or uh numerical computings. Um also it was it is it is interesting

Átirat · 1:02:50

that in the in the last one or two years that's a kind of a in this GPT

Átirat · 1:02:57

and AI for everything uh um the conclusion was that u um it was very

Átirat · 1:03:04

difficult to get the programmers to to to do the job or to try to code and and

Átirat · 1:03:11

interpret uh the results. And um just to say this to you that I'm

Átirat · 1:03:18

I'm speaking about students in the third in the third years of of of study who

Átirat · 1:03:25

worked with me in a in a group of uh what we call collective projects and uh

Átirat · 1:03:31

there was a there was quite a lot of quite a bit of u issues with that.

Átirat · 1:03:38

Why why should we do that? Why is that the results? and they say and they they just did not know it. So with that

Átirat · 1:03:46

thanks and I just want to acknowledge once more the uh the u the work of the

Átirat · 1:03:52

group. I also want to acknowledge the the help from uh from the Bosch and I

Átirat · 1:03:58

and I acknowledge the uh Ka the Romanian hub for artificial intelligence which is

Átirat · 1:04:04

which is now the group is is part of. So with that thanks

Átirat · 1:04:12

[applause] Thank you very much. Very nice. Uh so we

Témák

MI-támogatott

Magyarázható MI Gépi tanulás

Összefoglaló

MI-támogatott

Csató Lehel a magyarázható MI prototípus-alapú megközelítéseit vizsgálja, abból kiindulva, hogy a neurális hálózatok döntéseinek indoklása technikai szempontból fontos; az előadás az EU AI Actet is jogi motivációként említi. A képosztályozás példáin bemutatja a prototípusokkal és lokalizált képrészletekkel működő architektúrákat, majd egy lokálisan lineáris, invertálható wavelet-dekompozícióra épülő ProtoNet-változatot ismertet. Az eredmények mellett a módszerek korlátaira – például a rotációs invariancia hiányára – és az oktatási tapasztalatokra is kitér.

Videó navigáció

Fejezetek

MI-támogatott