Előadás

The Technology Powering Personal Digital Assistants

Németh Géza

BME TMIT

Beyond Artificial Intelligence Research: a New Innovation and Industrial Domain

2025. november 7. 13:00Babeș-Bolyai University, Central Building, Aula Magna · Kolozsvár · Románia

Átirat · 26:08

I thought about uh [snorts] trying to uh go back uh to some a little

Átirat · 26:15

bit in time about 10 years and actually most of the slides here come from Rui

Átirat · 26:21

Sarikaya who was a Microsoft Corporation uh researcher at that time and he gave a

Átirat · 26:27

keynote talk at inter speech in Dresden Germany and then uh it was so good that I asked

Átirat · 26:34

him to give the talk to me uh the slides and then I'm still using it and uh I

Átirat · 26:41

think that it can be useful right now for you as well. So I will try to explain uh why personal digital

Átirat · 26:48

assistants u came to life. what is uh their role and uh why it has not changed

Átirat · 26:57

that much uh in the last 10 years even if there was uh there were a lot of

Átirat · 27:02

events but please remember that the breakthrough in deep learning happened in speech recognition in about 2011 2012

Átirat · 27:12

and uh already uh three years later uh there were some uh well widely known

Átirat · 27:21

techniques uh that are still there. So it's a good outline what you see here

Átirat · 27:27

that why are they why are they important and um what are the lessons which are

Átirat · 27:33

still there and it's very interesting for me to see um well what happens so here we have some very nice ideas and

Átirat · 27:41

very nice people so I would like to ask you uh please raise your hand if you use

Átirat · 27:47

uh digital assistant let's say once a week from here who are those people who

Átirat · 27:54

use a digital assistant once a week. Okay, so it's like minority and uh we'll

Átirat · 28:02

see why. Um so already 10 years ago we had Siri,

Átirat · 28:08

we had Google Now, we had Cortana, Echo, Amazon Alexa and whatever. So they were

Átirat · 28:15

all there and even now when I ask just very few people use them. So the

Átirat · 28:22

question is why and that's why it's uh good to see um well what it is actually

Átirat · 28:30

um and the main point is that it is a sort of meta layer of intelligence. So

Átirat · 28:35

they want to do something which we usually do like one by one and it would

Átirat · 28:42

provide um integrated environment and the natural language interface where the

Átirat · 28:47

natural language is sometimes typing which is like typing I think is not so natural. So um very few people are born

Átirat · 28:55

by typing and reading and writing. Um actually we are speaking and um

Átirat · 29:03

already like a two or three year old child is very good in controlling the environment by speech or even not even

Átirat · 29:10

speech but just by sounds. So I have a a granddaughter at the three month and

Átirat · 29:16

she's very good in providing information about her mental state just by providing

Átirat · 29:22

some sounds. So then uh what are the techniques which are used here? machine

Átirat · 29:28

learning, speech recognition, natural language processing, dialogue management, ranking, inference,

Átirat · 29:34

personalization and so on. So all these things have been there um or were there

Átirat · 29:39

10 years ago and they are still with us. We are still working on them. It's a good question actually why do we need uh

Átirat · 29:47

this kind of technology and uh the most important of course is varial

Átirat · 29:54

things. So first take certain routine things uh which I don't want to do so

Átirat · 29:59

much and have some more efficient way of uh getting information.

Átirat · 30:06

So the main point would be to make our life more productive

Átirat · 30:12

uh both in work. So a good example could be which is still there. You booked a

Átirat · 30:18

flight a couple of weeks ago and then what could happen? your assistant re

Átirat · 30:23

recognizes uh that you did that booking uh through email then it can u compute

Átirat · 30:32

your current uh location and find out what are the traffic conditions and then you tell you when you have to start yes

Átirat · 30:39

it's a problem I think the same problem inclusion Budapest that we have traffic jams and it's not so easy to get to the

Átirat · 30:46

airport so sometimes um probably is better to drive uh from Budapest clues, then take the plane

Átirat · 30:53

because probably uh it's more deterministic and then we could have this kind of uh

Átirat · 31:00

screen on our uh mobile phone and giving us uh instructions and theoretically

Átirat · 31:07

already 10 years ago it was claimed that this can provide a breakthrough and we

Átirat · 31:14

are here 10 years later and we are still waiting for the breakthrough.

Átirat · 31:20

Okay. And the good question is that if we are still waiting then why are they investing into the thing. The first

Átirat · 31:29

thing is that um already yes we have more slow we have u

Átirat · 31:34

better and better uh techniques uh but it means that there are new and

Átirat · 31:42

new devices uh and after a while if you look at uh global shipments already like

Átirat · 31:51

10 years ago it turned out that everybody buys basically smartphones and

Átirat · 31:56

then all the rest laptops and tablets and whatever they are uh not so much.

Átirat · 32:03

And if you look at uh the number of shipments which I added a little bit later, it turns out uh that actually

Átirat · 32:11

um we are buying less and less of those devices because already everybody has something. So then we are just replacing

Átirat · 32:19

the old ones. So mobile there is a mobile for domination but

Átirat · 32:26

um already um 10 years ago about 23 of the like

Átirat · 32:31

real population like above 12 years already owned a smartphone. So here uh

Átirat · 32:38

we can see um the general statistics of 10 years ago and then we if we go a

Átirat · 32:45

little bit further uh to more uh recent data basically we see the same there is

Átirat · 32:51

a sm small increase but not uh really big.

Átirat · 32:57

So it means uh that from a business point of view where companies should

Átirat · 33:02

grow and uh have more and more sales and more and more profit. It turns out that from the smart devices they cannot

Átirat · 33:11

um make more money. So then the next idea was if you cannot sell more software then let's try to sell more

Átirat · 33:19

software. So then they started to introduce software stores

Átirat · 33:25

and uh already 10 years ago we had like 1.6 million applications in a standard

Átirat · 33:33

software phone uh store. And now um if we look uh at more recent things we see

Átirat · 33:42

about u the same. So maybe for Google Play we have more more than 3 million

Átirat · 33:48

for the Apple store about 2 million but basically it does not make much

Átirat · 33:54

difference. So who will look at 2 million applications? how much time that

Átirat · 34:01

would take and how can you find something if you if you need.

Átirat · 34:07

Um so then there was a an increasing number of downloads. It is uh still

Átirat · 34:13

there but the problem is that usually uh you download maybe around 30

Átirat · 34:21

applications in a year on your phone. So it could be a good homework for you uh

Átirat · 34:27

colleagues to look at your phone and count the number of applications that you have on your phone. And what is even

Átirat · 34:34

worse from the business point of view that uh you download maybe 30 40 applications in a year but you will only

Átirat · 34:42

use 12 of them in a month. So again it could be a good exercise for you try to

Átirat · 34:49

count the number of applications that you use uh within a month. So it means that uh about 80% of the applications on

Átirat · 34:58

the software sports are dead like practically nobody uses them. So then

Átirat · 35:04

people start uh uh to find like strange ideas already 10 years ago there was an

Átirat · 35:10

application which would count your next pe time when you are supposed to go to the toilet and these kind of uh very

Átirat · 35:18

intelligent applications appear. uh so the next uh challenge is uh about

Átirat · 35:25

how you spend your time with these devices and the applications. So already

Átirat · 35:31

10 years ago it happened that uh people started to spend um about the same

Átirat · 35:38

amount of time uh on mobile phones than on the television.

Átirat · 35:44

And then if you look at more recent statistics, we can see uh that now you

Átirat · 35:51

people in the US at least but I guess also here spend more time on the mobile

Átirat · 35:56

phones than on the television. But and there is a small decrease on the

Átirat · 36:03

television time. Uh but if you add up uh actually it is more and more. So you

Átirat · 36:08

spend more and more time um both on the phone and the TV. sometimes maybe in

Átirat · 36:14

parallel so you can check in your home for example how much you have the TV

Átirat · 36:20

turned on while you are watching your mobile phone and u and then of course there is a

Átirat · 36:27

competition who can gain your time so that you would be there so it's again a

Átirat · 36:35

um bottleneck uh cognitive bandwidths um and also the

Átirat · 36:40

amount of time you are willing to spend on uh learning an app

Átirat · 36:46

whether usually when I ask my students in UX and

Átirat · 36:51

AI course um most people say that they never spend more than one minute on

Átirat · 36:57

learning an application. So if you are developing software this is a really big challenge and um if you look at the

Átirat · 37:04

statistics um how people uh spent uh their time 2009 uh 2013

Átirat · 37:11

you can see uh that somehow it's strange because

Átirat · 37:16

um if you add up all the time that people spend here and there then basically you have no time to sleep

Átirat · 37:22

because uh something had to uh happen and then of course we can question.

Átirat · 37:28

Okay, where do we use the phones? Already 10 years ago, the situation was something like that. Uh so even if you

Átirat · 37:37

want um something you need to do some messaging uh over the phone otherwise

Átirat · 37:44

it's hopeless to reach people. So now we have a a very special uh way of life

Átirat · 37:50

because then we don't have uh professional life and uh personal life

Átirat · 37:57

because uh the two get absolutely confused. So sometimes when people ask

Átirat · 38:02

me how many are my working hours at the university, I would say 24 hours. And uh

Átirat · 38:10

because uh anytime any device uh can ring me and send information even in

Átirat · 38:15

very special situations like this and um so it means somehow we need some

Átirat · 38:23

help uh to handle uh this thing. And then the idea is that uh these personal

Átirat · 38:29

assistants could help us in this. So the question is why companies invest in this

Átirat · 38:36

because here there seem to be uh motivation there seem to be bottlenecks

Átirat · 38:42

and then somehow uh these requirements should be filled.

Átirat · 38:48

So 10 years investment and uh nothing happened. So it seems uh that this is a

Átirat · 38:56

big growth area and they supposed already 10 years ago that it the business will be more than 100 billion

Átirat · 39:02

US dollars per year and it has not come yet but uh why it's interesting because

Átirat · 39:12

uh it is something like um we had in department stores. So there are people

Átirat · 39:19

who want to have some products that are sold in the department store. They have to pay to the department store to put it

Átirat · 39:25

in the good position. And now that may happen in software that those people who

Átirat · 39:30

those companies who win the assistant competition assistant business they will

Átirat · 39:37

uh get money from those who are uh doing uh the messaging systems who are doing

Átirat · 39:43

the navigation systems and uh booking.com and so on. and we shall see it a little bit later.

Átirat · 39:50

So because the main idea is that they will integrate the potentially all the

Átirat · 39:57

important applications in a scalable way into one uh user uh scenario. So for

Átirat · 40:04

example, if you want uh to travel from somewhere, then the various uh

Átirat · 40:09

applications will be called here. And why are they personal? because uh they

Átirat · 40:16

know what we do probably they know our interest they should see uh what we

Átirat · 40:24

actually need and and also know the location where it happens but the problem is uh for that they need a lot

Átirat · 40:31

of data a lot of personal data so I think like security would be very happy

Átirat · 40:36

to know know all that data that is required for this uh application we need

Átirat · 40:42

the computing uh capacity There has to be an interest from users and then there

Átirat · 40:48

has to be some um things. So how is it being performed? So one of the most

Átirat · 40:55

important thing is that we have more and more sensors in the phones. So it is

Átirat · 41:00

just a list of 10 years ago. Since then we have much more. So then from using

Átirat · 41:06

the sensors we can find a lot find out a lot about people. So they can uh measure

Átirat · 41:14

where we are, how we move. For example, we made a student project on identifying

Átirat · 41:19

the person just by walking. So the way you walk in 10 seconds, it's possible to

Átirat · 41:25

give who it is. So that would be a good application to prevent stealing our mobile phone. So the next question is uh

Átirat · 41:33

okay, how to make it personal? So you need a user profile uh see the activity,

Átirat · 41:39

location and time. And of course already a lot of people

Átirat · 41:44

try to do it but uh interestingly we are just trying to save our privacy. So it's

Átirat · 41:50

a controversy. We want to have this sort of intelligent assistant but uh we don't

Átirat · 41:56

want to give uh all this information and how uh can we imagine it like two

Átirat · 42:02

basic approaches. uh the first is proactive when the system knows what we need and then

Átirat · 42:09

offers some application or even does some activity for us or the reactive

Átirat · 42:14

part. So here we need a very interesting uh

Átirat · 42:20

background psychological background of theory of proactivity and uh see how things can be uh

Átirat · 42:28

automated and then the question is of course whether it really it would really do what we want or do something else and

Átirat · 42:35

uh sometimes we have uh that uh situation. So what would be the architecture? It tries to collect a lot

Átirat · 42:43

of information uh build a model on us and then in the real situation do the

Átirat · 42:49

inference uh activity and that's a big challenge uh to put the

Átirat · 42:54

inference on edge devices and uh do it well uh for us. The more traditional one

Átirat · 43:02

is the reactive classical uh dialogue system where we do some instructions and

Átirat · 43:08

uh also here you have a a lot of interactions but here we still have a

Átirat · 43:14

lot of design gaps. So young people here you have a good chance that probably in the next 10 years uh you will move these

Átirat · 43:21

things ahead and uh I cannot give the same talk 10 years from now hopefully.

Átirat · 43:28

So here you can see uh some examples. So the the big challenge is that the

Átirat · 43:34

personal assistant should be real something like a human assistant or at

Átirat · 43:39

least comparable. And uh here the key issue is how to

Átirat · 43:45

integrate third party applications into it because uh until now all these

Átirat · 43:51

assistants were monolithic uh applications from a single vendor and

Átirat · 43:56

then u the users of course have these uh challenges to handle the errors also

Átirat · 44:04

sometimes uh lack of competence both from the device and from the users. For

Átirat · 44:09

example, myself, I'm lacking the competence of driving a recent car. It

Átirat · 44:14

would take maybe one week uh to teach me how to use all the possible uh

Átirat · 44:19

functionalities. And then uh these uh things are even more complex. So it's a

Átirat · 44:25

good question from us. And the last key question is trust. whether are we willing to give all that information

Átirat · 44:32

that is very personal um very important to these systems and how can we gener uh

Átirat · 44:38

guarantee that the system use it in u in a in a proper way. Then next question is

Átirat · 44:45

how to measure these things. Um good question is how often we are

Átirat · 44:51

willing to uh to use it and then how successfully uh we do it and in the end

Átirat · 44:57

of course it should pro produce some revenue profit for the companies. It's not by accident that now we have all

Átirat · 45:02

this hype around AI but very little information about AI companies whether they generate profit or or generate loss

Átirat · 45:11

and then for the component systems uh we all have also have solutions uh to

Átirat · 45:17

measure it. So there are some traditional ways. So what are the big challenges? Third party speech

Átirat · 45:24

recognition, language understanding, just uh very few people know that in the

Átirat · 45:29

world we have about 7,000 languages and uh out of that maybe 3,00 500 400

Átirat · 45:37

4,000 are being used by a larger number of people and then most uh companies can

Átirat · 45:42

cover in the best case maybe 100 120 languages. So you have enough room for the remaining um maybe 3 4 thousand

Átirat · 45:52

and then here in the future uh we see a lot of challenges uh which will go on

Átirat · 45:58

and also where these uh techniques will appear. So not only in the smartphones but maybe in the cars maybe in the

Átirat · 46:05

watches or or anything and maybe uh it's also a question of how to take the

Átirat · 46:10

information from one device to the others. So you already see this uh challenge in uh for example watches,

Átirat · 46:18

rings, uh phones, PCs and so on. And the recent news with which I like to finish

Átirat · 46:24

my talk is that uh just a month ago um

Átirat · 46:30

they announced OpenAI that uh CH GPT will open uh up interfaces for third

Átirat · 46:37

party applications. Uh so this is like a a good question. uh

Átirat · 46:43

also for young people if you are able to create some very specific application that could integrate well with CH GPT

Átirat · 46:50

maybe it would be a good interesting uh research and business opportunity for you but it's also a a sort of risk for

Átirat · 46:58

everyone because uh if uh there will be some like big boss over everybody and

Átirat · 47:05

they decide whether which navigation is good for him and not necessarily for the

Átirat · 47:11

users then we can lose use in this and last but not least I'd like to recommend for you the Mfield project which is

Átirat · 47:19

about green and trustworthy AI with a number of uh partners where we are

Átirat · 47:24

partners there and you are all welcome uh to join uh this activity and put it

Átirat · 47:30

your calendar that uh at the end of July next year so last week of July in 26 we

Átirat · 47:40

shall have a special event on artificial intelligence in Budapest and you are all welcome there and if you have an

Átirat · 47:46

interest please let me know. Thank you very much.

Átirat · 47:51

[applause]

Átirat · 47:58

Thank you very much Gaza. And uh our next speaker is Roland Vida also from

Átirat · 48:03

the Budapest University of Technology and Economics.

Átirat · 48:22

Okay. And uh Roland uh arrived also from uh our department and he has uh several

Átirat · 48:30

courses on uh smart city and he is part of uh or actually one of the board

Átirat · 48:37

members of the sensors council of IT. So the floor is yours.

Témák

MI-támogatott

Összefoglaló

MI-támogatott

Németh Géza a személyes digitális asszisztensek tízéves fejlődését vizsgálja, a Siri, Google Now, Cortana és Alexa korszakától a mai MI-rendszerekig. Bemutatja az asszisztensek mögötti beszédfelismerési, természetesnyelv-feldolgozási, párbeszédkezelési, rangsorolási és személyre szabási technológiákat, valamint a hardverpiac telítődéséből, az alkalmazásgazdaságból és a felhasználói figyelemért folyó versenyből adódó üzleti motivációkat. Külön foglalkozik a proaktív és reaktív asszisztens-architektúrákkal, a harmadik fél alkalmazások integrációjával, az adatvédelemmel és a bizalommal, majd a ChatGPT harmadik fél alkalmazások felé történő nyitását a terület lehetséges új fordulópontjaként említi.

Videó navigáció

Fejezetek

MI-támogatott