How to Train Custom LLMs: From Prompting to Pre-Training

0:00Happ classic is a nice description actually it was rough it was like a three-hour hair appointment um it was brutal one of the biggest acquisition around ai9 so it has been a generation since we had an another great neural network architecture so literally get the paper try to implement it and see and so we now have Rex

0:23nice the official masc of DB Rex thanks everybody for um joining so today I'm super excited to have a special guest Jonathan and with me is Maria Maria isan uh senior specialist architect sorry on M Andi hey Mar hello guys I'm glad to be here actually and we have with us our special guest so Jonathan Frankl who um

0:47is Chief scientist at Databricks within Mosaic Chief AI scientist oh sorry Chief be careful I don't don't make me responsible for all of spark I can't tell you anything about how Spark works but I know a lot about AI so thank you so much for joining um Janan we will spend a few uh minutes to discuss about dat bricks Mosaic ml uh and what you are

1:11doing into this world um I was surprised because I was expecting you to see with green hairs I understood but now I see like something very classic so can you can you tell us what happened classic is a nice description actually classic is very generous I've heard I've heard it described as Justin timberlay from the 1990s um no this is what remains of the

1:33green hair that I had a couple months ago when we released dbrx um or rather as it's now pronounced DB Rex um you know like the dinosaur and I have I have a little prop to go with that in a moment um but you know I promised the team that if they trained the best open source llm in the world at coding um I would dye my hair green and they were

1:56quite far off a couple of weeks out but you know apparently they are motivated much more by embarrassing their boss than by money or fame or power or Fortune um and so now um I had to dye my hair grain and I had to bleach it first so I the dye is gone but the blonde is still there I've gotten it cut a couple times so now the blonde is just left on

2:18the top and one of these days I'll have my normal hair back maybe by did anyi Summit now you feel the pain that we women go through right it was rough it was like a three-hour hair appointment um um it was brutal before working with their bricks you worked um at mosaic ml so you were training llms for customers and then you

2:41released I think the first was it the first open source llm MPT um available to the to the public and then Mosaic ml was acquired by D bricks last year it was like one of the biggest acquisition around Ai and llm can you tell us you know maybe what you did at mosaic ml what you doing now how was the acquisition what what changed yes so I

3:03think you know Mosaic ml we were always dedicated to the idea that it's possible to make it cheap and cost effective and efficient for anyone to train their own neural networks now what neural networks people wanted to change wanting to train have changed a lot over the years um you know it started off with we were training reset 50 on imet back when Mosaic ml started and by the end we were

3:27training you know 30 billion parameter llms reset 50 for the record is I think 25 million parameters um and trains in a day on eight gpus 8 A1 100s maybe you know a lot less than that by the time Mosaic was done it trained in a I believe less than an hour um whereas now we're training models that train on thousands of gpus and run for days or weeks or months but the mission was

3:49always the same can we make this cheap enough and easy enough and efficient enough that anyone can do this any company any person anybody who wants to customize their own model on their own data um we think data is kind of something unique that every individual and every company has and they should be able to leverage it to build you know build that into AI now our strategy was

4:13always around kind of trying to find every way to make things more efficient we cared about you know fancy compilers and kernels and making the most out of the GPU I mean you can see some dead gpus right above me if you really want to enjoy that um but we were also obsessed with kind of algorithmic speedups can we change the architecture the data the optimizer all the details change the math behind how

4:37you train neural networks to get speed up on your average h100 or a100 today your llm is getting about 50% of the maximum theoretical usage of the chip what that means is that there's not really much further to go even if you had perfect use of the chip you'd only get a 2X speed up that's not enough we need 4X or 10x and so we have to change how we train the models and that was

5:00always the Approach at mosaic ml that was the approach even during my PhD which was kind of you know where a lot of those ideas personally for me began and we've continued that now that we've joined Databricks except now we have 12,000 customers that we can get that out to um but you can even look at this today dbre Rex which which which was our latest llm was about 4ex more efficient

5:23than the MPT models that we trained last summer um and that was forx in less than a year through a combination of better data and data practices which it turns out spark and Databricks help a lot with that and better architectures and ways of training and just a lot of learning and maturity so 4X in less than a year that's something that wouldn't have physically been possible just from

5:45you know getting better use of the GPU we had to change the way we trained and Mosaic was committed to that now it's at Databricks and it's here for all of our customers that's amazing so all of these optimization and all of this new knowledge that we have now so how can I customers leverage it so what are the different options that we can provide them with yes so whenever a customer

6:07comes and they want to do anything with Gen and today that's pretty much everybody um I always tell them there are a couple important principles number one is you know you really have to start small and work your way up nobody should jump in today and say I'm going to train my own gp4 from scratch um and there are a lot of reasons for that one is that you know

6:30gen we're still trying to understand it and there's certain cases where it works really really well and certain cases where it's not working amazingly well right now and it's hard to know before the fact so I always recommend to people like let's try things that are really easy and really cheap and really fast it's all about Roi and you should kind

6:52of climb that ladder you should start off with things that require small investment and we'll get you very small returns if any but you know let's try that first let's try prompting GPT 4 it's an excellent model let's prompt it and see what happens let's do rag and see what happens maybe something will happen maybe something won't but let's see and then maybe let's go to F tuning and datab Bricks has a fine tuning product you know for prompting gp4 or

7:15llama 3 or DB Rex I don't really care which model we have an inference product in our model Gateway where you can go ahead and just do this today we have something we're going to announce it you know I guess we will have announced at this point given when we're recording at data Summit to make it really easy to do rag and then from there you can find tune your own model you can actually customize it on your data requires a little more investment in terms of time

7:37and energy and cost but you can get a lot more out of it and you can make it a lot more efficient to use because you can have your own little custom model you don't have to rely on a big model you could customize something small like llama 38b excellent model and see if that helps then you can and you know we have a fine-tuning product for that which is you know a great way to do that and then we can move on to what I call continued pre-training where you're fine

7:59tuning on lots of data and then you can move on to curriculum learning where you're literally replacing part of how we pre-trained a model like DB Rex with pre-training on your data we can kind of give you a model that's 90% trained and you can redo the last 10% on your data it's kind of an in between a fine tuning and pre-rain we actually have an API for that the pre-training API that's now in preview and then you can go all the way

8:21and pre-train your own model from scratch now again you should start small and work your way up if you try to do that last step right right away it's kind of dangerous and again a couple reasons for that first of all we need to make sure there's Roi we need to make sure that it's worth it too to do this I don't want anybody wasting their time and energy and money unnecessarily and those smaller steps are the right way to start building that maturity and working

8:43your way up the second is a maturity question I don't think many folks are ready to dive in and pre-train a gp4 model and what I mean by that is you've got to have your data in a really good State you've got to know your data incredibly well and probably bring in some additional data to kind of fill out gaps that you have and you need to know where those gaps are you need to know how to measure what success looks like it would terrify me this is my biggest

9:06fear someone comes to me and says hey Jonathan I just trained this gp4 class model it took months and months and tons of money and tons of work is it good that's my biggest fear and honestly I think it's a legitimate fear right now we need to have evaluation so that we can measure whether that model is good before we've trained it in fact you need to have trained enough smaller models that you can predict how good that big

9:28model is going be before you ever train it and know whether you succeeded that requires some maturity that requires some time and energy it's a hill that we have to climb together and so for me it's it's working your way up that ladder step by step different people are in different places we're here to help you to do that um and you know at the end of it I think a lot of

9:50people should have their own custom models but some people may be very satisfied with rack and that may be the right Roi that's great some people may find that gen just isn't ready for what they're trying to do it can't reach the level of reliability or quality that they need it can't work with the kind of data they have that's okay too we're making a lot of progress and we're doing new research every day if it doesn't work today it may work next year or it may work tomorrow for that matter um but

10:13we have to start small and work our way up and be patient can you give us a few example like a few customer example what they did using fine tuning or training from like a new model from scratch yeah definitely so I you know I don't want to speak out of turn about any of my customers but I can give you just a few examp examples off the top of my head for training from scratch this tends to be the most popular in cases where there simply doesn't exist any model that's

10:36good at the thing that people care about so one example is repet which trained these really small really efficient code models because they wanted to offer code completion to every user even their free users code completion is pretty expensive to do if you're doing it for everybody so they had to train small efficient models that were cheap and cost effective enough that they could do this um another example we have a company doing you know proteomic there's

10:58no model for proteomics in the world you can't use llama 3 for that so they had to train their own model from scratch one company Ola which is training the world's best model for India the model is built to be good at I think it's a couple dozen languages spoken in India there are a lot of different languages it's low resource data Ola has done really hard work to collect data in those languages and has

11:21provided a model that's serving you know a huge and important community in the world and is doing so better than any model in the world so for custom pre-training those are the scenarios that really speak to me when it comes to fine-tuning and pro prompting this is where you see dozens and dozens and dozens of scenarios everybody from the BBC which is using this to you know help prepare articles um so that journalists

11:43can spend more time in the field and less time you know doing mundane work um you know there are tons and tons of scenarios like this so I'm seeing it across the board and lots of folks are working their way up that maturity curve and yesterday we doing prompting and today are doing fine-tuning and a lot of people who yesterday were doing fine tuning and today are doing pre-training so it's really exciting to just watch everyone climb that maturity curve so

12:05Jonathan specifically on the fine tuning could you explain a little bit more like what what will what will fine tuning give us specifically right so okay pre-training give us a completely new model fine tuning what is the benefit of fine tuning per se so fine tuning will give you a model that knows about your data it's not going to be a model that you know only knows about your data or

12:27is really truly specialized to your domain that kind of only speaks your language and speaks it really well but it's model that at least has seen your data before you prompt it prompting is stateless the model will see the data you know in the context and you have to hope that the model is able to kind of Reason about that data live in a way that where good things will happen and fine tuning the model is actually or the data is actually built into the model

12:49and so it's a little more powerful now there are trade-offs there by training on that data you're specializing the model for just that data if your data gets updated frequently that can be tricky because have to fine tune frequently to do that if you have access control restrictions where certain users should only interact with certain data fine tuning can be tricky because it's all baked into one model these are scenarios where rag can be really helpful it can keep your data up to date

13:11it can give you access control my best recommendation is actually do fine tuning and rag gives you the best of both worlds the model will kind of know about your data fundamentally it won't have to like gather information about your data live in the context where it can do less with the data it's less powerful but you also get the benefits of constantly having new data that you can pull in Via rack so keep in mind

13:34these aren't mutually exclusive you can do many of these things you could even pre-train your own model and then fine-tune it on something more specialized to really make it Focus now fine-tuning is a little bit you know their trade-off sometimes the model will get so specialized to the data it will forget other things and so there are a lot of tricks and techniques we have for kind of managing this trade-off between learning and forgetting in fact we just put out a paper on Laura which is a very

13:57popular technique for fine tuning and Laura is interesting you know I don't actually recommend it in all scenarios but one thing we found it's really good at is keeping your model from Forgetting the trade-off is that it restricts how much your model can learn so pros and cons you have to make this trade-off for yourself but it gives you options it gives you a design space to work in to try to find the best trade-off for you and I think that's the most important thing my team provides it's just good

14:20choices that have science to back up when you should use them and when you shouldn't maybe we should Deep dive a little bit on DB a well DB Rex yeah actually on DV Rex if you'll give me one moment I'll be right back I need to show you something okay now I'm I'm expecting to see like a full T-Rex so every you know every good model needs um a good mascot um you know there was

14:43llama there was Dolly and so we now have Rex nice the official mascot of DB Rex um we have a mascot some graphic design that will go with DB Rex and some swag that will be given out at Dani Summit including you know a bunch of these little rexes this is the cutest dinosaur I think I've ever seen um

15:06and um you know the next version of dbrx and I think I can say there's another version coming um the code name is Velociraptor um so you know we're we're leaning in very hard to this branding every model needs a mascot and I think Rex is a fantastic mascot um for the kinds of models we're creating and Velociraptor is a great you know mascot for um you know for what's to come and

15:32you know that may tell you it's going to be small very fast and you know a little bit bloodthirsty like this from EXA exactly you know I did have to you know got to share my little friend can you tell us about the like what did you do to make DB Rock like DB Rex Rock did you

15:59like what does it take to train a full model like that how is how is it working so I think the you know the first thing we should always start with is why like why do you train a model like this and I I mean that because you don't train a model just for fun um you know it is very fun to train a model but it's really hard it's really expensive it takes a lot of time and energy so why do

16:23that and there were a couple of reasons the first is you know part of my job here is to make sure that data rcks is the best place in the world to create your own custom gen at every step of this process we talked about I need to figure out whether we are the best place in the world to do that and so I had to build a model I had

16:45to build something that was bigger I hoped than anything our customers would do anytime soon that would push the envelope on all of our tooling everything from spark you know we were working on massive data sets and use spark to pre-process all of our data to m flow for experiment tracking to our inference service for Speed to you know our actual Mosaic training we used the same Mosaic training infrastructure we used for MPT both a lot of upgrades and

17:09you know a lot of Databricks integration um and I wanted to see what is the end to-end experience for training models using all of our datab bricks gen Tools in fact there was actually one thing that we needed to use that wasn't native to datab Bricks it was this tool called lilac for data exploration um but you may recall that Databricks recently acquired lilac yes kind of interesting um you know I we liked it a lot we liked it so much we

17:31decided to buy it um so dbrx or you know DV Rex um was really about testing this endtoend pipeline stress testing it figuring out where we had work to do figuring out how we could make things better and how we could make things amazing for our customers so I think mission

17:53accomplished there we used the full Suite of datab bricks tools we used Unity catalog for governance we you know basically everything and had a great experience and there were a bunch of places where we came out with new recommendations and those recommendations are now going into production um ml flow has been greatly upgraded for geni use cases and we're now switching our team over to completely using ml flow because it's

18:16met that standard um lilac is here and we're revamping lilac um to integrate with Databricks spark has a bunch of you know a bunch of new use cases and a bunch of notebooks that we've created and we're now sharing with customers that we used for DB Rex so that was goal number one and to do that you know the model doesn't have to be perfect the model doesn't even have to be good we just have to have tried now it it was

18:38also a pretty good model and I'm really proud of that um the other piece was that we wanted to show our customers that if they came and wanted to pre-train a model with us they could train worldclass models this is the best model in the world that someone could come and train today with with a company that you could go to a company and say Hey I want to train a model just like that but with my data or my

19:02customizations this is the best model in the world where that's ever been done where that's a real product not just something internal but a legit product that somebody can go and use and I hope we prove to all of our customers that you know Databricks with Mosaic AI is still the best place in the world to do that and you know you look at this model I said a 4X efficiency improvement over MPT this is our next iteration of our

19:26pre-training product and our continued pre-training product product and our fine-tuning product this is now the latest stuff and you get all those benefits if you come and train with us I promise our customers efficiency if I lower costs Roi is higher for anything that a customer wants to do and it's good for business because customers will want to do more stuff and they'll get more out of it that's just a complete win across the board for everybody and

19:48the third was we wanted to provide something great for the open source Community um I wanted to do something that was unique mixture of expert models were pretty unique at that time I wanted to do something that was really powerful it was the most powerful open source model in the world at the time of release I wanted it to be big it's 130 billion parameters which looked huge at the time um and so you know for all three of those reasons we serve the

20:11community because the community helps us um you know we stand on the shoulders of giants in the community and we I we've tried through DB Rex to recognize all the amazing work and the amazing people who helped us um we hope by giving this back um the community will have another amazing artifact and they'll be able to do great things as well I come from that community and I want to never forget where I came from wow thanks so this

20:35explains why it was built and why you open sourced mainly everything right so not only the model but all the different components you use to train it so yeah because we have no business need not to yeah um you know it's always a tricky thing doing research in Industry where you have to balance business and business is what pays for the research and allows you to keep doing it and sharing that research and serving the community um I feel really lucky that

20:58we're at Databricks where there's a commitment from the very first day the company was founded to share with the community and to make the world a better place a belief that we can make Databricks the best place to use this technology but we should have to win on our own merits not because we locked you in and so DB Rex is open source I think data brick is the best place to work with that model and the technology underlying it but if somebody else

21:22builds something better that's competition and we have to do an even better job we have to win our customers business every single day and I like that a lot more than locking people into an ecosystem we're going to build better products and it's just going to be good for the team um but again the other piece we have nothing to hide we don't make our money off of having some big secret we make our money off of having an amazing product where you can come and customize this so what do we have to

21:45hide why shouldn't we share our checkpoints and share how we built this and share you know our you know our code for training this model and our code for serving this model what do we have to lose if we're the best product people are still going to work with us so I take this as a sign of confidence but also a sign of investing in the community because that's what will power the next version of DB Rex is great Innovation from the community that we're able to put together into something you

22:07know really great and then we share that back and the community does more great work with it and it's that cycle that I think you know that's what makes the scientific world go around we can't just take we have to give back as well so you you mentioned a new version of dbrx um how is it working like do you keep the same architecture and just swap the version do you just

22:29um I don't know update the weights do you retrain on whatever you have what's the secret Source um I can't say too much I don't really like to tease things so that's as much teasing as I'll give um you know I I say this in all seriousness first of all you never know what you're going to get whenever you set out to train a new model um so you know I don't want to promise too much or promise a timeline right now because who

22:51knows what will happen the other is I really like to speak through my work less so through words so I hope that dbx speaks for itself in terms of what we did how we built it what our values are um how we care about our customers in the community and I like to let the models and let the papers and let the science speak for itself as well so I

23:15hope the next model will speak for itself um and you know we'll wait for the model to do that amazing so Jonathan let's take you back to the data so how any any best practic that you can share on how to structure Data before training a big model like dbrx yeah we learned a ton of lessons along the way um data is really

23:39hard I you know one of the things I learned along the way was thank goodness for Databricks um we did the MPT data on our own using some weird collection of tools that we threw together it was really hard um with the help of some Databricks folks um especially you know Shan Owen who I'll give a shout out to um we were able to get all of our

24:01pre-training data and all of our after training data into Databricks within two weeks of joining and then we've started moving so much faster on the data side because we can just you know as an example we can I think I think it's we can embed a trillion tokens of data for a thousand dollars which is insane and we can tokenize our entire you know many trillions of data that we use for

24:24pre-training in like minutes these are things that took us days manual work previously so we're much more focused on the science not on the process and that's been huge from a data practices perspective I'm going to sound like a broken record but the very first thing to think about is how you measure whether your model's good um until you have evaluations you

24:46have nothing and there's no such thing as a free lunch um when it comes to data you're choosing your data for a purpose you're training your model for a purpose and if I told you this is an exercise I love to run whenever I give a talk about dbx I'll run this at data and AI Summit um I'll put up a list of kind of data sources and then ask people to take a minute and

25:09tell me how to choose that data should you like repeat Wikipedia a lot of times because it's a very high quality data set or should you use data from the web which is you know probably lower quality but more quantity if you need to build a one trillion token data set how do you kind of weigh these things and I love giving this question number one because it's interesting to see people's answers intuition is really hard we have best practices here but the other is that

25:33eventually someone will ask me after a couple minutes of doing this wait but what is this model supposed to do and that's exactly what I really want people to get out of this is to realize wait I can't choose the data independent of the context of what I'm trying to do with the model and how I'm measuring success if it's a code model you'll make very different choices than if it's a model designed for Japanese if it's model designed to be in an ultra safe kind of factual setting versus a model for

25:56creative and storytelling purposes you'll make very different choices and so that's the first step we made the choice that we wanted a model that was kind of good at general purpose work um especially good at code but you know really good as a general purpose chatot and that affected some of the data that we chose we then added a few other interesting best practices um we you

26:19know it's impossible at large scales to try out all the possible different data mixes so we tried to make very specific choices of what we explored like code versus non-code is the right percentage there because we couldn't explore every possible combination we looked into like should you upsample high quality data sets like Wikipedia or should you not um the lesson was actually high quality data sets the model tends to learn pretty quickly anyway so treat all

26:42tokens equally for the most part um so we did that um we learned that actually you should change your data mix lat in training to focus on data sets that you think are higher quality and the nice thing is that although it's too expensive to try out all possible data mixes at the beginning of training for the whole thing it is possible to just try out a bunch of different data mixes for the very end of training especially on smaller models and so we're able to find a really amazing data mix for the

27:05end of training and use that as a way to you know kind of finish strong um we learned that if you're trying out small models sometimes they aren't actually good enough to give you signal on large models to give you one example there's a popular coding Benchmark called human eval it has nothing to do with actual human evaluation that's just what they call it I don't know why um you need a

27:27pretty big model model trained on that data set before you get any signal before your model isn't just randomly guessing and so if you train too small of a model while exploring the amount of code you'll end up just randomly guessing and you won't actually get any signal you'll just be choosing based on noise you need a sufficiently big model to get signal there so it's important to pay attention to that as well so there a lot of pitfalls and a lot of gotas but

27:50at the end of the day I'm actually really proud of the data effort and token for token we actually ran this experiment we retrained MPT 7B but using our new data set instead otherwise identical architecture identical hyperparameters identical everything it was way better with the new data set and I did this cool thing where I told the team let's try training a version of MPT 7B but on half the data with the new data set and it did better than the

28:12original MPT 7B which had twice the data that is to say our data was two times better token for token than our last data set and I think that's pretty amazing and then when you throw in some of these other tricks that we didn't even use this is just the raw data set I would say we're probably Clos to 4X X better um if you do this kind of data set switching at the end of training and a few other things I've recommended so just the data alone was 2x better with

28:35additional data practices forx better that's incredible in nine months actually less than that because it was closer to six months when we got the data set before we started training so then is there a rule of thumb as in how many tokens you should have depending the how big the model you want to train would be so sort of for pre-training the rule of thumb is kind of the best way to make use of your is

28:5820x number of parameters as number of tokens that's kind of the optimal ratio there's this paper called chinchilla that showed this and we've seen it still in our experiments to this day the paper's about three or four years old with that said though there are a lot of tradeoffs maybe you want to spend a little extra money for training train a model that's too small because it'll make inference faster this is what the

29:20Llama paper did they trained models that were far smaller than optimal so they were being a little less efficient on training but it paid off because they these amazingly small models for you know fine tuning and inference that became very popular because they were so small and we did the same thing for dbrx um the optimal ratio is 20 we actually did adx for DB Rex and the new llama 3 Model I believe did like you know closer

29:43to a THX for llama 38p um you know that's very inefficient from a training compute perspective but it's an investment so the details matter um and the situation matters it's always a question of how are you going to use this in Roi this is honestly why I love getting to work with customers and I I recommend they come work with us we understand these tradeoffs really

30:05well we've studied them really carefully because we do this every day for ourselves and we're always happy to have that conversation about what the right tradeoff is for your setting when it comes to fine tuning these are much less of a challenge because the model's already mostly trained and you're just really doing a little bit on top of it but it still comes down to a question of you know if you need the model to be sufficiently fast in practice when you're doing inference then you're going

30:27to have to train a small model and maybe train it for longer so again it's all about tradeoffs and Roi but I'm here to help with that and my team is here to help with that is it the same using a mixture of experts or you know just going with a Danse model like is using a mixture of experts just a trade-off between the cost of training and the cost of insurance it is we'll talk about mixture of experts I think in a moment because I do it's always good to talk a

30:50little bit about what that means and what that doesn't mean I think people kind of the name sounds a little bit sexier than what it is um but mixture of experts it's just a different you know it's a different way of structuring the network kind of a an A variation of a transformer but really we've seen kind of the same behaviors mixture of experts do behave a little bit differently in some respects but nothing that's meaningful enough that I'd really tell

31:13people to change what they're doing um I want to say a bit about what a mixture of expert model is because I think this is important people think of this and they say oh I'll have a specialized model for finance and for Japanese and for retail and what have you and I'll somehow put them all together and make that into one super model that you know can combine the best of that that's not really what it is um I don't want to

31:35deal with the technical details because I think they're actually a little bit confusing and the name is confusing the most important part of a mixture of expert model is this generally bigger models are better they're smarter um again I mentioned that that that llama llama 38b used a 1,000x ratio of parameters versus amount of training data and that 20 is the

31:57right thing so they were way off from that they were 50x off from that um ideally you'd train a bigger model and have a little bit less data and that would be a better way to spend your budget it would be more efficient um the challenge is bigger models are more expensive to serve they're just bigger and slower and so how do you break this tradeoff what if you have a big model but you don't use all of the model every

32:21time that you ever you know use the model for serving or even train the model you only use a small part of it and the model dynamically chooses which pieces are relevant that's a mixture of expert model so you're decoupling the overall size and in that sense the smarts of the model from how fast it is and how much it costs to use it that's the power of mixture of expert model we love this um

32:43because it gives new tradeoffs it it improves the trade-off that you have between you know between your training compute and your inference compute between the smarts of the model and how efficient it is that's the big deal of mixture of experts the rest is just detail so don't don't get too confused by the name don't get misled by any of that the important part is you can have your cake and eat it too you can have a big smart model

33:06and you can have the cost from a much smaller model so Jonathan can you expand a little on this like so if I have dbx which is a mixture of experts model and I want to serve it I will serve it as a whole model no and then on inference different parts of the architecture are being let's say kicked kicked off and to create the output for me

33:29exactly so dbrx has 132 billion parameters if you were to actually use that 32 billion parameters it'll be about twice as slow as a 70 billion parameter model like llama 3 70b or you know but it only ever uses 36 billion parameters on any given input it's actually twice as fast as llama

33:52370b so it has double the parameters but it's twice as fast that's kind of you know you would assume it would you know be the other way around double the parameters twice as slow but thanks to the mixture of experts we're only using about half as many parameters as llama 370b does so you know it's a it gives us a really nice kind of a really nice Advantage um it's a tricky architecture

34:15to train but I've already done the work of figuring out how to do that and now we can just plug and play if you come to me and you say you want a train run cool figured that out already um now we can help you do it on your data so then in terms of memory when you load the model it's still a big model but on INF time it's much faster and in terms of quality I guess it's comparable or even better yeah it gives you like quality is always

34:38tricky because any model size can get to any quality for the most part if you try hard enough kind of as the first approximation llama 38b is about as good as llama 270b but it was a lot more expensive to train yeah um so it's really about you can cram anything into any package dbrx was in some sense me saying well I've got a limited budget not all of us you know can go and spend you know billions

35:01and billions of dollars just for a fun project some of us you know we have a business to focus on here um so I have to make the best tradeoff for what I can afford and get a really good model and you know what my customers will be able to take advantage of in a meaningful way and having mixture of experts gave me better tradeoffs to make than just training a normal model and so I could

35:23you know I could spend my budget wisely still train a really big model and still make it really really fast that's really exciting so and and how do you so how do you Benchmark that like do you compute the cost of training and you and you compare that with the actual performance like you have a flop like a training flop per performance outputs yeah yeah

35:45we track very closely the actual flop utilization on the GPU because you know just because you're you're hypothetically using a certain number of flops for the model doesn't mean you're actually getting good performance out of the GPU so so we always look at what percentage of the real GPU performance are we getting um and we try to maximize that and then we separately look at okay what is the you know the same thing for

36:07inference basically what flop utilization are we going to get out of the GPU how many tokens per second are we going to get what is the dollars per token all those sorts of metrics and we think about that really carefully and we try to make the best choices there um and you know it's it's a tradeoff for me if I had 10 times the budget maybe I would have trained a model much smaller for much longer and tried to get something even more efficient for inference or maybe if I had 10 times the

36:30budget I'd actually just train something a lot better um and maintain the same tradeoff or even pick a bigger model so it's always this question of you know how good does it need to be versus how fast and how cheap does it need to be versus how much budget do we have and how should we spend that optimally it's really tricky um It's Tricky for anybody who comes along and wants to train or fine tune their own model that's what my team is here for we've done our homework

36:52we've already trained the model we've had to do we've had to think this through and we're here to help you think it through let's maybe talk about evaluation when you like when you when you train a general model like dbrx how do you know it's walking well at the end um it's really hard to know evaluation is the single hardest thing in all of gen right now we have a huge Suite of

37:14benchmarks we call it the you know the Databricks model Gauntlet It's a combination of 36 open source benchmarks at least it was 36 last I checked um all from the community that we've combined into various categories like reading comprehension or logical reasoning or programming where we've also curated the benchmarks a little bit we've kind of we've said here are the benchmarks that we think are just noise here are the

37:37benchmarks where we've saturated like every model is going to do amazingly well those kinds of benchmarks don't contribute any meaningful signal so we leave them out um then we go through and we look at which benchmarks are actually good quality some of these benchmarks are just kind of nonsense have a lot of wrong answers or things like that and we measure the model on that now I'm not satisfied with that I don't think this is good enough nobody thinks the evaluations we have are good enough

38:00right now um so internally we're constantly building new evaluations and we have a bunch of In-House benchmarks that we've created for our scenarios both from product teams that are trying to deploy these models and from my team trying to measure um for example there are a lot of public benchmarks on text to seq we're not very satisfied with them we don't think they're hard enough we don't think they reflect real world use cases where there are typically multiple turns of interaction with the model to try to get the right SQL and

38:22get the right output so we're building benchmarks for that um we think function calling benchmarks that exist in practice are pretty weak they're not hard enough and they're not kind of reflective of real scenario so we're building an Enterprise function calling Benchmark right now internally um and everybody should be doing this every customer we work with should be building a benchmark to measure the thing they care about again we're doing this the hard way so that our customers can do

38:44this the easy way are you planning to open source that too um so the benchmarks it's always a tricky thing with open sourcing benchmarks a lot of people are starting to do a thing where they open source part of the Benchmark but not the whole thing thing because once you've open source The Benchmark people can just train on it they can game it um if you keep a little bit of it closed that gives you a way to maintain kind of a clean leader board yeah it was always

39:07like my intuition would be that as soon as you open source any Benchmark and they will somehow get leaked and and you know appear on the training data sets um so I'm always surprised to see all these you know open Benchmark because for for me like even if you don't want it somehow it's like it's must be super easy to get them into the training it can get into the training data just because it's in the training data doesn't mean the model memorizes it or will focus on it it's still a small part

39:30of the training set all things considered and the other piece is that I think most people are acting in good faith um but yeah it's hard um and once you have a benchmark what is it good heart's law like you know any metric ceases to be useful as soon as you know you start you know evaluating people on it um so we constantly need new benchmarks and I think maintaining a little bit in-house and a little bit externally is a really good way to

39:53strike this balance because we do want to give this to the community having to submit your model to a leaderboard is really slow yeah I want people to move fast and are we tuning our bench smarts depending the type of model we're trying to optimize for oh yeah like we're you know we're constantly thinking about like you know should we build an independent you know model for function

40:15calling or a model specific for chat or what have you um you know those sorts of things we've already you know data rck has you know built independent models for text to SQL um should these models be Specialized or should they not be and you know which Benchmark should we focus on this is constantly a discussion so it's really important to me that we you know we always have our eyes on what's

40:37important to our customers so what about evaluation uh but for customers like I'm fine tuning my llm I'm funing maybe dbr RS on their bricks can you share like the best practices around evaluation what you what should you check so I think the evaluation is still really tricky and I don't know I have like a clear best practice that everybody should

41:00follow it is somewhat bespoke but the you know the way that I think about this is to the extent that you already have an existing pipeline um you know like a human driven pipeline where this model is in practice and being checked or even just a process where things are being checked trying to leverage that Pipeline and having humans review the outputs of the model is the very best way to do this that is the gold standard automated evals are nice

41:24so that you can iterate but automated evals are very like very finicky if you want to have a perfect automated EV that will always give you the right answer you typically have to go with a multiple choice question and nothing is multiple choice in the real world if you want to have you know unstructured outputs that some that check automatically you have to use another model to check them and that's very finicky and unreliable it

41:46works it's not perfect and it's good enough for a lot of scenarios but it's still finicky and unreliable and it's not as good as the gold standard of having humans review so it's kind of worth thinking about that hierarchy you need something that's fast with using LM as a judge is a great way to do that it shouldn't be the end all be all if you can possibly help it so it it does get tricky in these scenarios um and my team is grappling with this every day as well we read our eval sets the entirety of

42:09our eval sets very carefully um because they're often surprises we look at the outputs of the model in addition to having an llm as a judge look at them because sometimes the llm as a judge is not doing what you think it's doing or behaving in a way you like sometimes you know your model is writing correct SQL but the SQL implementation you've chosen use like sqlite doesn't support the commands and so you're erroneously getting error messages we've seen that

42:32before so there's a lot of just it is still a very Hands-On process the way that I think of deep learning right now is it used to be that we had to spend a ton of our time building software and then a little bit of our time writing unit tests with deep learning in some sense the software just comes into being on its own you just push a button give it some data hand it some gpus and the software build itself so most of the

42:55work is going to be an evaluation measurement and test and that's kind of that's the price we pay for having this amazing technology that builds our software for us so then it's use as ADV Dan and always human in the loop then as best practice at least for now yeah to the best extent that you can and I know it's easier said than done for me to say like oh just use a bunch of humans if only we could use a bunch of humans this is why I say when

43:17you're an Enterprise you're probably building a model to augment an existing process to the extent that you can instrument that process and add you know exploit that process process as evaluation that can be really powerful that's a great way to do this without incurring a lot of time and energy um you know it's much more difficult if you don't have that existing process that's where llm as a judge can come in handy

43:39but again there are trade-offs to both of these so easy for me to say just to have humans do it um much harder in practice and I totally respect that so you mean like if you are whatever the easiest use case like a a bol that replies to you with I don't know a rug use case for example when you saw the the answer to the customer then or to whoever is viewing that then they can

44:02judge it as a positive or negative or good answer bad answer like crowdsource the evaluation to anyone who could use this application mainly yeah the way that I would think about it is that you know you let's suppose you have a process where your model is trying to trying to do some kind of classification let's say it's trying to put something into one of a thousand categories and right now a human would do all of this

44:24but you want your model to take care of the easy scenarios and you know have humans focus in the hard scenarios that's that's better for everybody you should still like you know once you deploy that model initially maybe you don't even deploy the model you kind of use the model keep track of what the model is predicting and test it against what the humans are predicting maybe once you've deployed the model you still have the humans check you know one

44:46out of every 10 model predictions and you know see what the human says that's a great way to find out whether your model is really behaving the way you want it to you shouldn't just you know release it into practice and hope for the best sh do the humans then exactly because you're you know you're trying to make humans life humans lives easier with these models but you know the flip side is that the models actually have to be

45:09good and reliable amazing so in terms of future of gen and what is coming in my view in my mind what I see coming is a lot of agents and a lot of multimodal stuff so what is your view on that Jonathan so if you see that coming you're smarter than me um I I I always say during these podcasts that I try not to predict the future okay um the

45:32present is complicated enough and I think if anyone were trying to predict the future over the past several years um they would have been completely wrong so I think it's better to accept that we're not certain except that you know we're you know we don't know what's coming prepare for anything um and preparing for anything means getting our data in order

45:55figuring out how to measure those are things that will always stand the test of time the rest the technologies will change what we want to do with them will change the ideas will change but data good evaluation and good processes to integrate gen into those will always be valuable and so I you know that's really kind of my my prediction of the future

46:19the rest we'll have to find out nicely said with me the data part is really sticking because because as you start tuning and training models you start seeing that if your data is not in the right place if your quality is not good then again R like trust in trust out right the usual ml problem so it's really interesting to see it coming back

46:41and all the time yeah on the data front I kind of I often get asked like in the age of gen do you even need to clean your data or structure it can't you just throw it all into the model and the answer is yes you need to the more structure you can give the better if your data is structured in a SQL database you have a model right SQL it works much better than just handing all the data to the model in the context and having the model flatten

47:03it so you know the structure Quality Cleaning all of that stuff is more important than ever in a gen world and I just you know who knows what the future will bring the best we can do is to be prepared all right anything else oh I have a last question I so you said that but I I saw you have a bet on is

47:26attention o on need can you can you comment on that what happened yeah so this is my dear friend Sasha Rush um at Corell Tech and hugging face um he you know asked on Twitter a couple years ago anybody willing to bet with me that you know Transformers and and networks with self attention will still be the you know the best way to train for natural

47:48language processing in a few years and I was dumb enough to say yes and Sasha being the wonderful person that he is literally created a website for this bet I believe we bet you know a couple shares of mosaic against a couple shares of hugging face both of which become much more valuable since then so you know hopefully one of us will be making a good donation to charity at some point in a few years um but you know the I bet that attention will still be

48:11the deao way of doing this or the default way and my reasoning is pretty simple um I love to ask this question I'll ask both of you um what were we using for NLP prior to the Transformer and self attention bug of GS good guess what's the kind of neural network model architecture be be behind bag of GS or just kind of in general

48:35prior to Transformers what was our pre- Transformer well it was um just embeddings and things like gloves I would say oh we oh you're you're going back to 2015 I'm going back to the 90s 90s so so it was it was a recurrent neural networker in lstm oh yes okay okay those are the default architecture prior to a Transformer and now I like to ask which is older me or an lstm I would

48:59never guess an age of a person tell me which me or an lstm I'm not asking you to guess the age just you know which one's older you then um yeah are you gonna guess me or the lstm I'm gonna say you so I'm barely older barely it has been a generation since we had an another great neural

49:23network architecture the convolutional neural network for computer vision is still in my view the best way to do computer vision networks the vision Transformer has still not surpassed it and the convolutional neural network is older than me we get new neural network architectures very rarely it's really hard to come up with them it takes a lot of work and a lot of people failing and someone gets lucky

49:47so I you know I'm willing to take this side of the bet because my bet is took us a generation to get a new architecture for NLP it's been more than a generation and we still haven't gotten a new architecture for computer vision I'm willing to bet the Transformers are going to be here to stay for a long time very interesting and then since we are

50:09talking about conv convolutional neural networks so uh since we have dbrx is it are we going to plan to release any other model like text to video text World deal again I don't like to talk about things before they're out um if this is going to go out after data and AI Summit I can say that I'm very excited that datab bricks has announced our first text image model that is

50:32available for our customers trained in cooperation with Shutterstock um which is you know the leading stock photo company in the world they have a data set where you know all the data is you know licensed in you know a careful way from content creators content creators actually get compensated every time this model is used on Databricks or on Shutterstock um we've been working on this for about a year it's been a huge

50:54collaboration with our friends at Shutterstock I can't be more excited that this is you know the first of many things to come out of that collaboration um we hope this gives Enterprise customers a lot of confidence in a text image model I know there's been a lot of kind of concern and nervousness around working with these models given you know what data they might have been trained on and where it came from um this is something that I feel very comfortable

51:16standing behind as you know something I trust something that I hope Enterprises can trust so I'm really excited that we've now announced this and it's available to our customers beautiful and last question from my side so how is the day-to-day life in Mosaic ml team look like like what is happening behind the scenes we're doing science every day we're

51:40there a few topics I set the agenda and there a few topics that we really care about um and we're doing science every day to try to figure out how to improve that we only work on things where there's no known answer otherwise it's engineering and so we're constantly trying new things pushing the envelope testing things we've read about in the literature to see if they actually work and whether they actually work in scenarios our customers care about

52:04that's every day for us um we're in the lab we're trying out new ideas we're trying to push the envelope on what's available for us and our customers so literally get the paper try to implement it and see uh or try out a new idea nice um completely new ideas things that we think are relevant for our customers but you know we do have to test a lot of papers because papers come out um but

52:26but we don't know whether they really work in the settings we care about just because it's a paper just because it's published doesn't mean it works doesn't mean it works for a customer so you know we spend a lot of time thinking about that and sometimes papers work phenomenally well often they don't because we're testing them more rigorously and at larger scales than people in Academia can because we have more resources so you know sometimes we

52:50succeed sometimes we don't I try not to talk about the things that don't work in public because you know a L of them are written by students and I don't have any reason to want to make a student's life difficult I was a PhD student not too long ago um but in general we need to test everything and have data to back up what works you can see this in our new paper on Laura everybody's talking about Laura

53:12everybody on Twitter seems to love Laura um you know is Laura really good just because people are talking about it and hyping it up doesn't mean it's good and so we had to go and figure it out all right don't trust TR Twitter yeah don't trust Twitter don't trust your friend of a friend at open AI don't trust the paper trust real data and real science thank you so muchan uh thanks

53:36for all the insights thank you Maria you know for being here here with us I hope it was a good sis and you you know had a good time and I hope we'll see you soon this was awesome and I can't wait to do this again soon when we have more to talk about guys thank you awesome thank you so much