Előadás

LLMOps 101: Patterns for Taking GenAI to Production

24. workshopResearch and Development in a World Changed by GenAI

2026. szeptember 29. 14:00BlackRock Hungary Kft. – GTC White House · Budapest

Átirat · 1:17:07

working on a on an intersection of AI cloud and media transmission and this time I will talk about my AI works. So

Átirat · 1:17:15

anonymops is a is a is a big thing. Uh it involves many many sub subjects as the previous

Átirat · 1:17:23

presentation has shown uh such as governance and monitoring and fine-tuning questions. Uh in my talk which I wanted to be a 101 but it turns

Átirat · 1:17:31

out this field is too big. So I will focus on the serving and deployment questions. Uh the goal with the talk is to s to show some patterns how we can

Átirat · 1:17:41

run these LM services. We have seen how to do guard rails. Now we can we'll see how to make them run in production

Átirat · 1:17:48

reliably and scale well with increasing load. So some some basics first. So in in case of a large language model, the

Átirat · 1:17:57

model life has two phases. Uh and the model consists of layers and and and and weights that are interconnects between

Átirat · 1:18:06

them. I don't really want to go to the to the details here. What is important that first the model is trained. This is

Átirat · 1:18:12

done usually offline. Uh this is used to learn the weights between the layers and the neighbors. And there is inference when we actually run this model. This is

Átirat · 1:18:21

an online thingy. It it runs continuously. Uh user [clears throat] request comes in and the model sends

Átirat · 1:18:28

some data back. So this is the this is the basic idea. It's it's somewhat analogous to programming. When we build

Átirat · 1:18:35

a program and we run it, we build it once and we run it multiple times. So this is a this is a rough analogy here.

Átirat · 1:18:42

Uh financial wise training burns the money but inference brings the money. So it's it's it's an important thing here.

Átirat · 1:18:48

Uh inference is not just simple serving like an HTTP server or or or or a classical service. Um many mainly

Átirat · 1:18:56

because the outputs are not not deterministic. An LLM can give different outputs to the very same questions depending on its internal state or or

Átirat · 1:19:05

any other reasons. Um this makes testing some and and deployment and scheduling somewhat harder. Uh the other important

Átirat · 1:19:13

thing is it's it's cost. The cost is not in a request basis. It's it's it scales with the tokens. Tokens are little units that are smaller than the words. This is

Átirat · 1:19:22

this is what the large language model speaks. And last but not least, latency here has two axes. One is the times to

Átirat · 1:19:30

first token. How long does it take to get the first first pieces of response back from a from a request? Uh and and

Átirat · 1:19:39

time per output token. How long does it take to get the follow-up tokens? Uh under the under the hood uh an inference

Átirat · 1:19:48

engine. That's the prefill phase to get the first token and then the decode phase runs. Uh this this will be

Átirat · 1:19:55

important later. So uh this is where our story begins. uh we have a little chatbot application uh it it can it can answer finance or question or whatnot.

Átirat · 1:20:04

Uh it runs on a single server with a single GPU and and we will see with the story of this this small chatbot how we

Átirat · 1:20:13

can how LMOS builds up from bottom to the top to to make it run like a full service with with capabilities to change

Átirat · 1:20:22

models on the fly and and uh monitoring and authorization etc etc. So our road map for this little stories is starting

Átirat · 1:20:30

from the bottom. We will see the runtime the the software and hardware components that are needed to run a model. Then we will see the infrastructure questions

Átirat · 1:20:39

how when we have multiple such little components how to orchestrate them together and we will see the tooling how to make this uh uh work for higher

Átirat · 1:20:47

operational level questions. So uh the runtime is it focuses on the question what runs on the GPU. So if if we come

Átirat · 1:20:56

from the user perspective, user request comes into our server. The inference engine runs on the server and this inference engine receives this requests

Átirat · 1:21:05

and and it also instruments the model on the GPU. On the GPU there are two important things. One is the compute

Átirat · 1:21:13

part uh that runs the kernels and and do the math stuff uh matrix multiplication and stuff like that. And the other

Átirat · 1:21:20

important part is memory. uh memory is responsible to store the model weights and catch some intermediate results. Um

Átirat · 1:21:29

in current yeah this can run on a GPU or any specific hardware. CPUs are not that fast for that or TPUs but in this

Átirat · 1:21:36

example we use GPUs as these are the mostly used hardware elements. G the the the one bottleneck here is how how how

Átirat · 1:21:45

fast we can we access the memory here and uh and uh we will see multiple techniques how to squeeze uh the maximum

Átirat · 1:21:54

from this setup. First we will focus on considering a single GPU and a single host. Uh the first technique is is to

Átirat · 1:22:03

improve the decoding is called a continuous batching. uh batching on its own is is a technique that requires that

Átirat · 1:22:12

the main idea is that to wait for a couple of tasks and do them together. It works really well when we do the do the

Átirat · 1:22:19

do the do the bashing at home but it also works very well in computing. In this case, uh we see that when we have

Átirat · 1:22:28

some longunning tasks, it can it can prevent the system to load up its full potential. But we can use the the the

Átirat · 1:22:35

auto reggressive nature of a large language model. When the decode phase runs and and output tokens come one by one, we can rearrange the works after

Átirat · 1:22:43

each token and so we can do it in a way that we can fill the batch up. So the decode part will run at full perform can

Átirat · 1:22:51

utilize the full performance. uh this is one trick that a runtime can use to inc to to to to

Átirat · 1:22:58

provide high performance and to minimize the TPU. The other thing technique is is is the prefix caching. We have said I we

Átirat · 1:23:07

have seen in the previous slide that the key there is a key value cache that stores intermediate results and and it pays off when we when we save the pref

Átirat · 1:23:17

results. As you can see the first query comes in uh we pro we do the prefix processing uh and then we do unique part

Átirat · 1:23:25

uh a prefix is something that is common in in in multiple queries like if everyone says hi computer and then says the question the high computer can be a

Átirat · 1:23:33

prefix that we can store and we don't need to generate the output tokens for that. Uh so so the idea here is that if another user comes with the very same

Átirat · 1:23:41

query we can reuse this share prefix and as we can see the time that takes to get the first token from the system is can

Átirat · 1:23:49

be much smaller. So this is a good optimization for the time to first token. Um and these are all patterns that are used by the runtime engine

Átirat · 1:23:58

these days. Um the other thing another angle to to optimize performance is is is

Átirat · 1:24:08

one is the quantization uh when we have when we want to free up some memory and and uh use bigger models

Átirat · 1:24:15

in our system. Um and and quantization is basically just the idea here is that we can do we can represent the model

Átirat · 1:24:24

weights with smaller uh smaller resolution. Um in this case the the

Átirat · 1:24:31

memory that we need uh is uh is smaller so we can run bigger models or have more space to run the caches. Um also in some

Átirat · 1:24:40

hardwares this can also lead to better better performance due to the smaller memory usage. Um we have to be careful

Átirat · 1:24:47

though uh when we are yeah this is this is not lossless compression in most cases. So check the output quality when we do some quantization. The other

Átirat · 1:24:56

technique is not that universal. Some models support it is the mixture of expert architecture that came to life

Átirat · 1:25:02

recently. Um standard language models uh proc use the whole model weight goes

Átirat · 1:25:10

through the whole memory area of a model to get to the results. And in case of a mixture of experts model there is a there is a router which decides which

Átirat · 1:25:19

the the logic is implemented in this thing expert and there is a router which highlights the this and this expert can resolve this this query efficiently. So

Átirat · 1:25:28

this time we are not shrinking the memory but we are shrinking the active memory. So this can make our system faster. So these are the techniques we

Átirat · 1:25:36

can use with a single GPU without adding any extra hardware to our system. uh the

Átirat · 1:25:43

widely used uh widely used LLM runtime engine VLM uh supports modes of this the mixture of expert architecture comes

Átirat · 1:25:51

from the model itself. Um sooner or later our little chatbot will be so exciting that we will see that oh

Átirat · 1:26:00

és 26 percwhy not use a bigger model or why not have better performance because we're so users and and one tactic is the tensor parallelism when we split the model and

Átirat · 1:26:09

and We split layers and some and and distribute this layer data evenly between the GPUs. So each DPU can run a

Átirat · 1:26:19

small small set of a layer data. So this way we can run bigger models. So this

Átirat · 1:26:26

way we can run bigger models on multiple GPUs. So we can use their memory efficiently here and and and

Átirat · 1:26:33

also we can we win some parallel execution here that that both GPUs run the same uh uh uh can run the same

Átirat · 1:26:41

credit and and work on smaller memory size. So that's that's that can lead to some speed up. The only drawback here is that in some cases the two GPUs have to

Átirat · 1:26:50

communicate with each other and that's that can be a bottleneck unfortunately.

Átirat · 1:26:54

There's another parallelism technique we can use with multiple GPUs and this is expert parallelism. Uh the idea here is that the inference engine has an

Átirat · 1:27:02

internal router uh uh and and the experts run on different GPUs. Um yeah

Átirat · 1:27:09

there are one subset of experts on one GPU and one on the other. And when the query comes in, this router decides which experts to activate. And this way

Átirat · 1:27:18

we can use all the memory and and and only that GPU is used where the expert runs and this can improve performance

Átirat · 1:27:25

but this can also be a bottleneck if all if uh if uh there is a hot expert in the system. So these are techniques uh that

Átirat · 1:27:32

we can still do in a runtime level techniques of so-called patterns that we can run in a runtime level with a single

Átirat · 1:27:39

host. Um but we are still running a single replica. So when user requests

Átirat · 1:27:46

comes in they might be queued up because of the single replica that is processing them. So here's when data parallelism come in the picture. The idea is when

Átirat · 1:27:54

our model fits into a single GPU why not run it on multiple GPUs. We can do it in a simple single huster. We can do it in a multihost manner as we can see on the

Átirat · 1:28:03

figure. Uh this is good because we have multiple replicas and we can distribute the load between them. This will enable

Átirat · 1:28:10

us to autoscale like add new hosts and we can distribute the load better or we can remove them if there aren't many uh things and and and here comes the

Átirat · 1:28:18

interesting thing. This is the first time when it it comes in that we can improve the availability that our system starts to become fault tolerant when any

Átirat · 1:28:27

of the of of the runners uh on a GPU goes wrong we can just yeah use the other one. Um so this is this is another

Átirat · 1:28:35

parallelism technique. Of course we can do hybrid parallelism when we combine the tensor parallelism and expert parallelism with the data parallelism.

Átirat · 1:28:44

Um we see this is the level we can reach with the runtime. So this this calls for an orchestration to to that that can

Átirat · 1:28:52

that can place these workloads that can scale the capacity and this can route our traffic and keep our services

Átirat · 1:28:59

running. Um and uh and this is the next level. So the community come up with different solutions for it. Uh mostly

Átirat · 1:29:08

reusing existing tools. Uh one of them is the ray and ray surf stack. Um someone might be familiar with it. It

Átirat · 1:29:15

was originally used to write distributed applications in Python and it got its renaissance with AI. It provides a raer

Átirat · 1:29:23

LLM serving route to serve LLMs. The other thing is this slurp thing that come from the high performance computing

Átirat · 1:29:32

is a cluster and job allocation system that can also be hacked to do the same.

Átirat · 1:29:36

Uh but most commonly and most widely used is is Kubernetes which is the def facto platform for clouds and and it

Átirat · 1:29:43

yeah it started to run general purpose services but over the time it got some extensions like the gateway API

Átirat · 1:29:50

inference extension that enabled it to to be aware of how to run these workloads on on it. Uh yeah, fun fact,

Átirat · 1:29:59

it can also run this ray surerve via its cube ray operator. Um and there is LMD a big project that combines all these bits

Átirat · 1:30:08

and pieces for Kubernetes and seems to be a a big tool here. So next I I would like to show what we what what the

Átirat · 1:30:15

orchestra what h how this or the how the or how hard this orchestration

Átirat · 1:30:22

gives us a platform and I will use some Kubernetes examples here. So we have seen that there are three decision decisions here uh from the orchestration

Átirat · 1:30:31

we have to place those workloads which means to schedule the the model workers on the available GPUs and for this purpose we have we have in Kubernetes

Átirat · 1:30:40

device plugins that enables us to access these GPU resources um namely to and and

Átirat · 1:30:46

and these things scaling uh which is to add replicas or remove replicas when the request comes in or goes out and for

Átirat · 1:30:54

this purpose Kubernetes has a nice abstraction is the horizontal pod autocaler with some custom metrics. It's important to

Átirat · 1:31:01

use some LM spec LM inference specific metrics namely the GPU usage and such things. Um and for routting uh which is

Átirat · 1:31:11

which is the task to pick the the replica for each request. Uh for this purpose it introduces this gateway API inference extension and there is an

Átirat · 1:31:19

endpoint picker that enables to do a selection between the possible replicas to run. We'll see them in a an example

Átirat · 1:31:27

in the following slides. So as I mentioned LMD is a good example that combines all of these and and gets gets traction nowadays. So one thing is is is the is the is the placing the workload.

Átirat · 1:31:38

Uh here we see the pattern of the prefield disagregation. The idea here is not just we not just split the traffic of the users but rathers we've split the

Átirat · 1:31:47

work. So we have some dedicated prefield pool and we have a dedicated decode pool. We have seen these are the two options that an two two functions that

Átirat · 1:31:55

an LM inference engine runs and and and the idea here is that we can scale them independently and and and this works

Átirat · 1:32:04

well if if if the input is is is is roughly 10 times bigger than the output and and for this purpose our chatbot uh

Átirat · 1:32:13

came with some so so for for the sake of the example it will not work with a single chatbot that short answers and

Átirat · 1:32:20

responses are requested But in a financial case we will use the retrieved augmented generation as we have seen in previous slides. So presentation. So

Átirat · 1:32:29

this leads us to have a longer context and and it makes worse to use this. Why this is not a universal thingy is because we have to transfer the key

Átirat · 1:32:37

value cache from the pref pool to the decode pool. And this is this is a pricey operation nowadays and it's it's

Átirat · 1:32:45

it's it's it takes times and it yeah it's it's a pricey operation. So it it might be a good optimization to deploy

Átirat · 1:32:52

this pattern but it it worth to be cautious and do some measurements here uh whether it's better than the

Átirat · 1:33:00

és 33 perccolloccated serving. So uh the other option is is the routting when we decide which replica to choose. uh here there is a there is a balancing act that needs

Átirat · 1:33:08

to be done where to select for example in this prefield example one prefill pod might have the prefix ready but it might

Átirat · 1:33:16

serve too many requests u if you send to them we have to wait the other uh to wait to be

Átirat · 1:33:25

to let it be free but we we spare the time to yeah keep the prefix cache warm or we can send it to another worker when

Átirat · 1:33:34

we have to populate the graphics cache but it's available. So this is a balancing act here. Um a good rule of

Átirat · 1:33:42

thumb here is that to to focus on the good putut. Good putut is is is the is the is the metric uh that that measures

Átirat · 1:33:50

the requ the requests that completed under the the requested SLO targets namely it's under the TTF or the TPOT

Átirat · 1:33:58

and it's a rate. So these are the two patterns that we can use in infrastructure layer. uh tooling can be used for higher level uh uh operational

Átirat · 1:34:07

things such as scannery deployment and and uh of of new model versions and and and uh monitoring etc. So these high

Átirat · 1:34:15

level operation things um yeah for example in this example we see yeah and for this purpose the pattern we

Átirat · 1:34:24

call is the LLM gateway that provides authorization monitoring data and decisions on where

Átirat · 1:34:31

to route the traffic. Uh so when this all goes together is this LM gateway is

Átirat · 1:34:37

the userf facing front that that provides us to choose different models and and model versions. The inference

Átirat · 1:34:45

router picks the worker itself and the workers can be put uh the workers are running the runtime and in each runtime there are different techniques to make

Átirat · 1:34:53

them work fast. uh yeah under the hood we have observability and ongoing evaluations that help us to to bring sol

Átirat · 1:35:02

to bring yeah control over where to run each each request. So this is the whole picture. Uh I have a couple of open

Átirat · 1:35:11

problems in this tech for the curious minds. Do we have time? Okay. So I just put three questions here. As I mentioned earlier moving the key value cache is is

Átirat · 1:35:18

is is a is a problem because it it the network is a bottleneck here. H how can we solve this? This is one question that there are many researchers these days.

Átirat · 1:35:27

The other thing is we have seen rooting and scaling. Uh can we make them work together to optimize the goodut uh and

Átirat · 1:35:34

the and this is not not trivial here because they are working in different angles and last but not least I was talking about goodput but what if there

Átirat · 1:35:43

are different constraints such as energy or carbon footprint or any other things to optimize. So this is uh this is uh

Átirat · 1:35:51

these are interesting questions here and I yeah I just jumped to the to the conclusion. So yeah this is the end of my talk. Uh I put some references and

Átirat · 1:35:59

reading list worth checking and was a big help for me to get all this together. Um so thank you for your attention.

Átirat · 1:36:08

[applause]

Átirat · 1:36:13

Thank you very much so uh please uh prepare.

Átirat · 1:36:18

So our next speaker is Nora Salotsi from the semiverse university and uh the

Átirat · 1:36:26

title of her talk is everyone has AI which is obvious. Uh but as class still decides who benefits which is not obvious. So the floor is yours now.

Átirat · 1:36:38

Okay. Sorry just give me a second. It work before Oh yeah that's the issue. So, I'm I'm going to be really honest with you guys.

Témák

MI-támogatott

Nagy nyelvi modellek AI infrastruktúra Generatív MI Vállalati mesterséges intelligencia

Összefoglaló

MI-támogatott

Lévai Tamás az LLMOps éles üzemeltetési mintáit az inferenciakiszolgálás szemszögéből tekinti át. Egy egyszerű, egy GPU-n futó chatbotból kiindulva mutatja be, miként növelhető a nagy nyelvi modellek kiszolgálásának teljesítménye és megbízhatósága. Kitér az első tokenig eltelt időre, a tokenenkénti késleltetésre, a prefill és decode fázisokra, a continuous batchingre, a kvantálásra és a mixture-of-experts modellekre. Tárgyalja a több GPU-s és több példányos skálázást, a Ray és Kubernetes szerepét, a prefill–decode szétválasztást, a KV-cache mozgatásának költségét, valamint a gateway, routing, monitorozás és értékelés összefüggéseit.

Videó navigáció

Fejezetek

MI-támogatott