Executive Interview
The Secret Sauce Behind DBRX with Jonathan Frankle
Jonathan Frankle · Chief AI Scientist · LinkedIn
Happ classic is a nice description actually it was rough it was like a three-hour hair appointment um it was brutal one of the biggest acquisition around ai9 so it has been a generation since we
had an another great neural network architecture so literally get the paper try to implement it and see and so we now have Rex nice the official masc of DB Rex thanks everybody for um joining so today I'm
super excited to have a special guest Jonathan and with me is Maria Maria isan uh senior specialist architect sorry on M Andi hey Mar hello guys I'm glad to be here actually and we have with us our
special guest so Jonathan Frankl who um is Chief scientist at data bricks within Mosaic Chief AI scientist oh sorry Chief be careful I don't don't make me responsible for all of spark I can't
tell you anything about how Spark works but I know a lot about AI so thank you so much for joining um Janan we will spend a few uh minutes to discuss about dat bricks Mosaic ml uh and what you are
doing into this world um I was surprised because I was expecting you to see with green hairs I understood but now I see like something very classic so can you can you tell us what happened classic is
a nice description actually classic is very generous I've heard I've heard it described as Justin timberlay from the 1990s um no this is what remains of the green hair that I had a couple months
ago when we released dbrx um or rather as it's now pronounced DB Rex um you know like the dinosaur and I have I have a little prop to go with that in a moment um but you know I promised the
team that if they trained the best open source llm in the world at coding um I would dye my hair green and they were quite far off a couple of weeks out but you know apparently they are motivated
much more by embarrassing their boss than by money or fame or power or Fortune um and so now um I had to dye my hair grain and I had to bleach it first so I the dye is gone but the blonde is
still there I've gotten it cut a couple times so now the blonde is just left on the top and one of these days I'll have my normal hair back maybe by did anyi Summit now you feel the pain that we
women go through right it was rough it was like a three-hour hair appointment um um it was brutal before working with their bricks you worked um at mosaic ml so you were training llms for customers and then you
released I think the first was it the first open source llm MPT um available to the to the public and then Mosaic ml was acquired by D bricks last year it was like one of the biggest acquisition
around Ai and llm can you tell us you know maybe what you did at mosaic ml what you doing now how was the acquisition what what changed yes so I think you know Mosaic ml we were always
dedicated to the idea that it's possible to make it cheap and cost effective and efficient for anyone to train their own neural networks now what neural networks people wanted to change wanting to train
have changed a lot over the years um you know it started off with we were training reset 50 on imet back when Mosaic ml started and by the end we were training you know 30 billion parameter
llms reset 50 for the record is I think 25 million parameters um and trains in a day on eight gpus 8 A1 100s maybe you know a lot less than that by the time Mosaic was done it trained in a I
believe less than an hour um whereas now we're training models that train on thousands of gpus and run for days or weeks or months but the mission was always the same can we make this cheap
enough and easy enough and efficient enough that anyone can do this any company any person anybody who wants to customize their own model on their own data um we think data is kind of
something unique that every individual and every company has and they should be able to leverage it to build you know build that into AI now our strategy was always around kind of trying to find
every way to make things more efficient we cared about you know fancy compilers and kernels and making the most out of the GPU I mean you can see some dead gpus right above me if you really want
to enjoy that um but we were also obsessed with kind of algorithmic speedups can we change the architecture the data the optimizer all the details change the math behind how you train neural networks to get speed
up on your average h100 or a100 today your llm is getting about 50% of the maximum theoretical usage of the chip what that means is that there's not really much further to go even if you
had perfect use of the chip you'd only get a 2X speed up that's not enough we need 4X or 10x and so we have to change how we train the models and that was always the Approach at mosaic ml that
was the approach even during my PhD which was kind of you know where a lot of those ideas personally for me began and we've continued that now that we've joined data bricks except now we have
12,000 customers that we can get that out to um but you can even look at this today dbre Rex which which which was our latest llm was about 4ex more efficient than the MPT models that we trained last
summer um and that was forx in less than a year through a combination of better data and data practices which it turns out spark and data bricks help a lot with that and better architectures and
ways of training and just a lot of learning and maturity so 4X in less than a year that's something that wouldn't have physically been possible just from you know getting better use of the GPU
we had to change the way we trained and Mosaic was committed to that now it's at data bricks and it's here for all of our customers that's amazing so all of these optimization and all of this new
knowledge that we have now so how can I customers leverage it so what are the different options that we can provide them with yes so whenever a customer comes and they want to do anything with
Gen and today that's pretty much everybody um I always tell them there are a couple important principles number one is you know you really have to start small and work your way up nobody should jump in today and
say I'm going to train my own gp4 from scratch um and there are a lot of reasons for that one is that you know gen we're still trying to understand it and there's certain cases where it works
really really well and certain cases where it's not working amazingly well right now and it's hard to know before the fact so I always recommend to people like let's try things that are really
easy and really cheap and really fast it's all about Roi and you should kind of climb that ladder you should start off with things that require small investment and we'll get you very small
returns if any but you know let's try that first let's try prompting GPT 4 it's an excellent model let's prompt it and see what happens let's do rag and see what happens maybe something will
happen maybe something won't but let's see and then maybe let's go to F tuning and datab Bricks has a fine tuning product you know for prompting gp4 or llama 3 or DB Rex I don't really care
which model we have an inference product in our model Gateway where you can go ahead and just do this today we have something we're going to announce it you know I guess we will have announced at
this point given when we're recording at data Summit to make it really easy to do rag and then from there you can find tune your own model you can actually customize it on your data requires a
little more investment in terms of time and energy and cost but you can get a lot more out of it and you can make it a lot more efficient to use because you can have your own little custom model
you don't have to rely on a big model you could customize something small like llama 38b excellent model and see if that helps then you can and you know we have a fine-tuning product for that
which is you know a great way to do that and then we can move on to what I call continued pre-training where you're fine tuning on lots of data and then you can move on to curriculum learning where
you're literally replacing part of how we pre-trained a model like DB Rex with pre-training on your data we can kind of give you a model that's 90% trained and you can redo the last 10% on your data
it's kind of an in between a fine tuning and pre-rain we actually have an API for that the pre-training API that's now in preview and then you can go all the way and pre-train your own model from
scratch now again you should start small and work your way up if you try to do that last step right right away it's kind of dangerous and again a couple reasons for that first of all we need to
make sure there's Roi we need to make sure that it's worth it too to do this I don't want anybody wasting their time and energy and money unnecessarily and those smaller steps are the right way to
start building that maturity and working your way up the second is a maturity question I don't think many folks are ready to dive in and pre-train a gp4 model and what I mean by that is you've
got to have your data in a really good State you've got to know your data incredibly well and probably bring in some additional data to kind of fill out gaps that you have and you need to know
where those gaps are you need to know how to measure what success looks like it would terrify me this is my biggest fear someone comes to me and says hey Jonathan I just trained this gp4 class
model it took months and months and tons of money and tons of work is it good that's my biggest fear and honestly I think it's a legitimate fear right now we need to have evaluation so that we
can measure whether that model is good before we've trained it in fact you need to have trained enough smaller models that you can predict how good that big model is going be before you ever train
it and know whether you succeeded that requires some maturity that requires some time and energy it's a hill that we have to climb together and so for me it's it's working your way
up that ladder step by step different people are in different places we're here to help you to do that um and you know at the end of it I think a lot of people should have their own custom
models but some people may be very satisfied with rack and that may be the right Roi that's great some people may find that gen just isn't ready for what they're trying to do it can't reach the
level of reliability or quality that they need it can't work with the kind of data they have that's okay too we're making a lot of progress and we're doing new research every day if it doesn't
work today it may work next year or it may work tomorrow for that matter um but we have to start small and work our way up and be patient can you give us a few example like a few customer example what
they did using fine tuning or training from like a new model from scratch yeah definitely so I you know I don't want to speak out of turn about any of my customers but I can give you just a few
examp examples off the top of my head for training from scratch this tends to be the most popular in cases where there simply doesn't exist any model that's good at the thing that people care about
so one example is repet which trained these really small really efficient code models because they wanted to offer code completion to every user even their free users code completion is pretty
expensive to do if you're doing it for everybody so they had to train small efficient models that were cheap and cost effective enough that they could do this um another example we have a
company doing you know proteomic there's no model for proteomics in the world you can't use llama 3 for that so they had to train their own model from scratch one company Ola which is training the
world's best model for India the model is built to be good at I think it's a couple dozen languages spoken in India there are a lot of different languages it's low resource data Ola has done really hard work to
collect data in those languages and has provided a model that's serving you know a huge and important community in the world and is doing so better than any model in the world so for custom
pre-training those are the scenarios that really speak to me when it comes to fine-tuning and pro prompting this is where you see dozens and dozens and dozens of scenarios everybody from the
BBC which is using this to you know help prepare articles um so that journalists can spend more time in the field and less time you know doing mundane work um you know there are tons and tons of
scenarios like this so I'm seeing it across the board and lots of folks are working their way up that maturity curve and yesterday we doing prompting and today are doing fine-tuning and a lot of
people who yesterday were doing fine tuning and today are doing pre-training so it's really exciting to just watch everyone climb that maturity curve so Jonathan specifically on the fine tuning
could you explain a little bit more like what what will what will fine tuning give us specifically right so okay pre-training give us a completely new model fine tuning what is the benefit of
fine tuning per se so fine tuning will give you a model that knows about your data it's not going to be a model that you know only knows about your data or is really truly specialized to your
domain that kind of only speaks your language and speaks it really well but it's model that at least has seen your data before you prompt it prompting is stateless the model will see the data
you know in the context and you have to hope that the model is able to kind of Reason about that data live in a way that where good things will happen and fine tuning the model is actually or the
data is actually built into the model and so it's a little more powerful now there are trade-offs there by training on that data you're specializing the model for just that data if your data
gets updated frequently that can be tricky because have to fine tune frequently to do that if you have access control restrictions where certain users should only interact with certain data
fine tuning can be tricky because it's all baked into one model these are scenarios where rag can be really helpful it can keep your data up to date it can give you access control my best
recommendation is actually do fine tuning and rag gives you the best of both worlds the model will kind of know about your data fundamentally it won't have to like gather information about
your data live in the context where it can do less with the data it's less powerful but you also get the benefits of constantly having new data that you can pull in Via rack so keep in mind
these aren't mutually exclusive you can do many of these things you could even pre-train your own model and then fine-tune it on something more specialized to really make it Focus now
fine-tuning is a little bit you know their trade-off sometimes the model will get so specialized to the data it will forget other things and so there are a lot of tricks and techniques we have for
kind of managing this trade-off between learning and forgetting in fact we just put out a paper on Laura which is a very popular technique for fine tuning and Laura is interesting you know I don't
actually recommend it in all scenarios but one thing we found it's really good at is keeping your model from Forgetting the trade-off is that it restricts how much your model can learn so pros and
cons you have to make this trade-off for yourself but it gives you options it gives you a design space to work in to try to find the best trade-off for you and I think that's the most important
thing my team provides it's just good choices that have science to back up when you should use them and when you shouldn't maybe we should Deep dive a little bit on DB a well DB Rex
yeah actually on DV Rex if you'll give me one moment I'll be right back I need to show you something okay now I'm I'm expecting to see like a full T-Rex so every you know every good model needs um
a good mascot um you know there was llama there was Dolly and so we now have Rex nice the official mascot of DB Rex um we have a mascot some graphic design that will go with DB Rex and some swag
that will be given out at Dani Summit including you know a bunch of these little rexes this is the cutest dinosaur I think I've ever seen um and um you know the next version of dbrx
and I think I can say there's another version coming um the code name is Velociraptor um so you know we're we're leaning in very hard to this branding every model needs a mascot and I think
Rex is a fantastic mascot um for the kinds of models we're creating and Velociraptor is a great you know mascot for um you know for what's to come and you know that may tell you it's going to
be small very fast and you know a little bit bloodthirsty like this from EXA exactly you know I did have to you know got to share my little friend can you tell us about the like what did you do
to make DB Rock like DB Rex Rock did you like what does it take to train a full model like that how is how is it working so I think the you know the first thing we should always start with
is why like why do you train a model like this and I I mean that because you don't train a model just for fun um you know it is very fun to train a model but it's really hard it's really expensive it
takes a lot of time and energy so why do that and there were a couple of reasons the first is you know part of my job here is to make sure that data rcks is the best place in the world to create
your own custom gen at every step of this process we talked about I need to figure out whether we are the best place in the world to do that and so I had to build a model I had to build something that was bigger I
hoped than anything our customers would do anytime soon that would push the envelope on all of our tooling everything from spark you know we were working on massive data sets and use
spark to pre-process all of our data to m flow for experiment tracking to our inference service for Speed to you know our actual Mosaic training we used the same Mosaic training infrastructure we
used for MPT both a lot of upgrades and you know a lot of data bricks integration um and I wanted to see what is the end to-end experience for training models using all of our datab
bricks gen Tools in fact there was actually one thing that we needed to use that wasn't native to datab Bricks it was this tool called lilac for data exploration um but you may recall that
data bricks recently acquired lilac yes kind of interesting um you know I we liked it a lot we liked it so much we decided to buy it um so dbrx or you know DV Rex um was really about testing this endtoend
pipeline stress testing it figuring out where we had work to do figuring out how we could make things better and how we could make things amazing for our customers so I think mission
accomplished there we used the full Suite of datab bricks tools we used Unity catalog for governance we you know basically everything and had a great experience and there were a bunch of
places where we came out with new recommendations and those recommendations are now going into production um ml flow has been greatly upgraded for geni use cases and we're now switching our team over to
completely using ml flow because it's met that standard um lilac is here and we're revamping lilac um to integrate with data bricks spark has a bunch of you know a bunch of new use cases and a
bunch of notebooks that we've created and we're now sharing with customers that we used for DB Rex so that was goal number one and to do that you know the model doesn't have to be perfect the
model doesn't even have to be good we just have to have tried now it it was also a pretty good model and I'm really proud of that um the other piece was that we wanted to show our customers
that if they came and wanted to pre-train a model with us they could train worldclass models this is the best model in the world that someone could come and train today with with a company
that you could go to a company and say Hey I want to train a model just like that but with my data or my customizations this is the best model in the world where that's ever been done
where that's a real product not just something internal but a legit product that somebody can go and use and I hope we prove to all of our customers that you know data bricks with Mosaic AI is
still the best place in the world to do that and you know you look at this model I said a 4X efficiency improvement over MPT this is our next iteration of our pre-training product and our continued
pre-training product product and our fine-tuning product this is now the latest stuff and you get all those benefits if you come and train with us I promise our customers efficiency if I
lower costs Roi is higher for anything that a customer wants to do and it's good for business because customers will want to do more stuff and they'll get more out of it that's just a complete
win across the board for everybody and the third was we wanted to provide something great for the open source Community um I wanted to do something that was unique mixture of expert models
were pretty unique at that time I wanted to do something that was really powerful it was the most powerful open source model in the world at the time of release I wanted it to be big it's 130
billion parameters which looked huge at the time um and so you know for all three of those reasons we serve the community because the community helps us um you know we stand on the shoulders of
giants in the community and we I we've tried through DB Rex to recognize all the amazing work and the amazing people who helped us um we hope by giving this back um the community will have another
amazing artifact and they'll be able to do great things as well I come from that community and I want to never forget where I came from wow thanks so this explains why it was built and why you
open sourced mainly everything right so not only the model but all the different components you use to train it so yeah because we have no business need not to yeah um you know it's always a tricky
thing doing research in Industry where you have to balance business and business is what pays for the research and allows you to keep doing it and sharing that research and serving the
community um I feel really lucky that we're at data bricks where there's a commitment from the very first day the company was founded to share with the community and to make the world a better
place a belief that we can make data bricks the best place to use this technology but we should have to win on our own merits not because we locked you in and so DB Rex is open source I think
data brick is the best place to work with that model and the technology underlying it but if somebody else builds something better that's competition and we have to do an even better job we have to win our customers
business every single day and I like that a lot more than locking people into an ecosystem we're going to build better products and it's just going to be good for the team um but again the other
piece we have nothing to hide we don't make our money off of having some big secret we make our money off of having an amazing product where you can come and customize this so what do we have to
hide why shouldn't we share our checkpoints and share how we built this and share you know our you know our code for training this model and our code for serving this model what do we have to
lose if we're the best product people are still going to work with us so I take this as a sign of confidence but also a sign of investing in the community because that's what will power
the next version of DB Rex is great Innovation from the community that we're able to put together into something you know really great and then we share that back and the community does more great
work with it and it's that cycle that I think you know that's what makes the scientific world go around we can't just take we have to give back as well so you you mentioned a new version of dbrx um
how is it working like do you keep the same architecture and just swap the version do you just um I don't know update the weights do you retrain on whatever you have what's the secret Source um I can't say too
much I don't really like to tease things so that's as much teasing as I'll give um you know I I say this in all seriousness first of all you never know what you're going to get whenever you
set out to train a new model um so you know I don't want to promise too much or promise a timeline right now because who knows what will happen the other is I really like to speak through my work
less so through words so I hope that dbx speaks for itself in terms of what we did how we built it what our values are um how we care about our customers in the community and I like to let the
models and let the papers and let the science speak for itself as well so I hope the next model will speak for itself um and you know we'll wait for the model to do that amazing so Jonathan
let's take you back to the data so how any any best practic that you can share on how to structure Data before training a big model like dbrx yeah we learned a ton of lessons along the way um data is really
hard I you know one of the things I learned along the way was thank goodness for data bricks um we did the MPT data on our own using some weird collection of tools that we threw together it was
really hard um with the help of some data bricks folks um especially you know Shan Owen who I'll give a shout out to um we were able to get all of our pre-training data and all of our after
training data into Data bricks within two weeks of joining and then we've started moving so much faster on the data side because we can just you know as an example we can I think I think
it's we can embed a trillion tokens of data for a thousand dollars which is insane and we can tokenize our entire you know many trillions of data that we use for pre-training in like minutes these are
things that took us days manual work previously so we're much more focused on the science not on the process and that's been huge from a data practices perspective I'm going to sound like a
broken record but the very first thing to think about is how you measure whether your model's good um until you have evaluations you have nothing and there's no such thing as a free lunch um when it comes to data
you're choosing your data for a purpose you're training your model for a purpose and if I told you this is an exercise I love to run whenever I give a talk about dbx I'll run this at data and AI Summit um I'll
put up a list of kind of data sources and then ask people to take a minute and tell me how to choose that data should you like repeat Wikipedia a lot of times because it's a very high quality data
set or should you use data from the web which is you know probably lower quality but more quantity if you need to build a one trillion token data set how do you kind of weigh these things and I love
giving this question number one because it's interesting to see people's answers intuition is really hard we have best practices here but the other is that eventually someone will ask me after a
couple minutes of doing this wait but what is this model supposed to do and that's exactly what I really want people to get out of this is to realize wait I can't choose the data independent of the
context of what I'm trying to do with the model and how I'm measuring success if it's a code model you'll make very different choices than if it's a model designed for Japanese if it's model
designed to be in an ultra safe kind of factual setting versus a model for creative and storytelling purposes you'll make very different choices and so that's the first step we made the
choice that we wanted a model that was kind of good at general purpose work um especially good at code but you know really good as a general purpose chatot and that affected some of the data that
we chose we then added a few other interesting best practices um we you know it's impossible at large scales to try out all the possible different data mixes so we tried to make very specific
choices of what we explored like code versus non-code is the right percentage there because we couldn't explore every possible combination we looked into like should you upsample high quality data
sets like Wikipedia or should you not um the lesson was actually high quality data sets the model tends to learn pretty quickly anyway so treat all tokens equally for the most part um so
we did that um we learned that actually you should change your data mix lat in training to focus on data sets that you think are higher quality and the nice thing is that although it's too
expensive to try out all possible data mixes at the beginning of training for the whole thing it is possible to just try out a bunch of different data mixes for the very end of training especially
on smaller models and so we're able to find a really amazing data mix for the end of training and use that as a way to you know kind of finish strong um we learned that if you're trying out small
models sometimes they aren't actually good enough to give you signal on large models to give you one example there's a popular coding Benchmark called human eval it has nothing to do with actual
human evaluation that's just what they call it I don't know why um you need a pretty big model model trained on that data set before you get any signal before your model isn't just randomly
guessing and so if you train too small of a model while exploring the amount of code you'll end up just randomly guessing and you won't actually get any signal you'll just be choosing based on
noise you need a sufficiently big model to get signal there so it's important to pay attention to that as well so there a lot of pitfalls and a lot of gotas but at the end of the day I'm actually
really proud of the data effort and token for token we actually ran this experiment we retrained MPT 7B but using our new data set instead otherwise identical architecture identical
hyperparameters identical everything it was way better with the new data set and I did this cool thing where I told the team let's try training a version of MPT 7B but on half the data with the new
data set and it did better than the original MPT 7B which had twice the data that is to say our data was two times better token for token than our last data set and I think that's pretty
amazing and then when you throw in some of these other tricks that we didn't even use this is just the raw data set I would say we're probably Clos to 4X X better um if you do this kind of data
set switching at the end of training and a few other things I've recommended so just the data alone was 2x better with additional data practices forx better that's incredible in nine months
actually less than that because it was closer to six months when we got the data set before we started training so then is there a rule of of thumb as in how many tokens you should have
depending the how big the model you want to train would be so sort of for pre-training the rule of thumb is kind of the best way to make use of your is 20x number of parameters as number of
tokens that's kind of the optimal ratio there's this paper called chinchilla that showed this and we've seen it still in our experiments to this day the paper's about three or four years old
with that said though there are a lot of tradeoffs maybe you want to spend a little extra money for training train a model that's too small because it'll make inference faster this is what the
Llama paper did they trained models that were far smaller than optimal so they were being a little less efficient on training but it paid off because they these amazingly small models for you
know fine tuning and inference that became very popular because they were so small and we did the same thing for dbrx um the optimal ratio is 20 we actually did adx for DB Rex and the new llama 3
Model I believe did like you know closer to a THX for llama 38p um you know that's very inefficient from a training compute perspective but it's an investment so the details matter um and the situation matters it's
always a question of how are you going to use this in Roi this is honestly why I love getting to work with customers and I I recommend they come work with us we understand these tradeoffs really
well we've studied them really carefully because we do this every day for ourselves and we're always happy to have that conversation about what the right tradeoff is for your setting when it
comes to fine tuning these are much less of a challenge because the model's already mostly trained and you're just really doing a little bit on top of it but it still comes down to a question of
you know if you need the model to be sufficiently fast in practice when you're doing inference then you're going to have to train a small model and maybe train it for longer so again it's all
about tradeoffs and Roi but I'm here to help with that and my team is here to help with that is it the same using a mixture of experts or you know just going with a Danse model like is using a
mixture of experts just a trade-off between the cost of training and the cost of insurance it is we'll talk about mixture of experts I think in a moment because I do it's always good to talk a
little bit about what that means and what that doesn't mean I think people kind of the name sounds a little bit sexier than what it is um but mixture of experts it's just a different you know
it's a different way of structuring the network kind of a an A variation of a transformer but really we've seen kind of the same behaviors mixture of experts do behave a little bit differently in
some respects but nothing that's meaningful enough that I'd really tell people to change what they're doing um I want to say a bit about what a mixture of expert model is because I think this
is important people think of this and they say oh I'll have a specialized model for finance and for Japanese and for retail and what have you and I'll somehow put them all together and make
that into one super model that you know can combine the best of that that's not really what it is um I don't want to deal with the technical details because I think they're actually a little bit
confusing and the name is confusing the most important part of a mixture of expert model is this generally bigger models are better they're smarter um again I mentioned that that that llama llama 38b used a
1,000x ratio of parameters versus amount of training data and that 20 is the right thing so they were way off from that they were 50x off from that um ideally you'd train a bigger model and
have a little bit less data and that would be a better way to spend your budget it would be more efficient um the challenge is bigger models are more expensive to serve they're just bigger
and slower and so how do you break this tradeoff what if you have a big model but you don't use all of the model every time that you ever you know use the model for serving or even train the
model you only use a small part of it and the model dynamically chooses which pieces are relevant that's a mixture of expert model so you're decoupling the overall size and in that sense the smarts of the
model from how fast it is and how much it costs to use it that's the power of mixture of expert model we love this um because it gives new tradeoffs it it improves the trade-off that you have
between you know between your training compute and your inference compute between the smarts of the model and how efficient it is that's the big deal of mixture of experts the rest is just detail so don't
don't get too confused by the name don't get misled by any of that the important part is you can have your cake and eat it too you can have a big smart model and you can have the cost from a much
smaller model so Jonathan can you expand a little on this like so if I have dbx which is a mixture of experts model and I want to serve it I will serve it as a whole model no and then on inference
different parts of the the architecture are being let's say kicked kicked off and to to create the output for me exactly so dbrx has 132 billion parameters if you were to actually use
that 32 billion parameters it'll be about twice as slow as a 70 billion parameter model like llama 3 70b or you know but it only ever uses 36 billion parameters on any given input it's actually twice as fast as llama
370b so it has double the parameters but it's twice as fast that's kind of you know you would assume it would you know be the other way around double the parameters twice as slow but thanks to
the mixture of experts we're only using about half as many parameters as llama 370b does so you know it's a it gives us a really nice kind of a really nice Advantage um it's a tricky architecture
to train but I've already done the work of figuring out how to do that and now we can just plug and play if you come to me and you say you want a train run cool figured that out already um now we can
help you do it on your data so then in terms of memory when you load the model it's still a big model but on INF time it's much faster and in terms of quality I guess it's comparable or even better
yeah it gives you like quality is always tricky because any model size can get to any quality for the most part if you try hard enough kind of as the first approximation llama 38b is about as good
as llama 270b but it was a lot more expensive to train yeah um so it's really about you can cram anything into any package dbrx was in some sense me saying well I've got a limited budget not all of us you
know can go and spend you know billions and billions of dollars just for a fun project some of us you know we have a business to focus on here um so I have to make the best tradeoff for what I can
afford and get a really good model and you know what my customers will be able to take advantage of in a meaningful way and having mixture of experts gave me better tradeoffs to make than just
training a normal model and so I could you know I could spend my budget wisely still train a really big model and still make it really really fast that's really exciting so and and how do you so how do
you Benchmark that like do you compute the the cost of training and you and you compare that with the actual performance like you have a flop like a training flop per performance outputs yeah yeah
we we track very closely the actual flop utilization on the GPU because you know just because you're you're hypothetically using a certain number of flops for the model doesn't mean you're
actually getting good performance out of the GPU so so we always look at what percentage of the real GPU performance are we getting um and we try to maximize that and then we separately look at okay
what is the you know the same thing for inference basically what flop utilization are we going to get out of the GPU how many tokens per second are we going to get what is the dollars per
token all those sorts of metrics and we think about that really carefully and we try to make the best choices there um and you know it's it's a tradeoff for me if I had 10 times the budget maybe I
would have trained a model much smaller for much longer and tried to get something even more efficient for inference or maybe if I had 10 times the budget I'd actually just train something
a lot better um and maintain the same tradeoff or even pick a bigger model so it's always this question of you know how good does it need to be versus how fast and how cheap does it need to be
versus how much budget do we have and how should we spend that optimally it's really tricky um It's Tricky for anybody who comes along and wants to train or fine tune their own model that's what my
team is here for we've done our homework we've already trained the model we've had to do we've had to think this through and we're here to help you think it through let's maybe talk about
evaluation when you like when you when you train a general model like dbrx how do you know it's walking well at the end um it's really hard to know evaluation is the single hardest thing in all of
gen right now we have a huge Suite of benchmarks we call it the you know the data bricks model Gauntlet It's a combination of 36 open source benchmarks at least it was 36 last I checked um all
from the community that we've combined into various categories like reading comprehension or logical reasoning or programming where we've also curated the benchmarks a little bit we've kind of
we've said here are the benchmarks that we think are just noise here are the benchmarks where we've saturated like every model is going to do amazingly well those kinds of benchmarks don't
contribute any meaningful signal so we leave them out um then we go through and we look at which benchmarks are actually good quality some of these benchmarks are just kind of nonsense have a lot of
wrong answers or things like that and we measure the model on that now I'm not satisfied with that I don't think this is good enough nobody thinks the evaluations we have are good enough
right now um so internally we're constantly building new evaluations and we have a bunch of In-House benchmarks that we've created for our scenarios both from product teams that are trying
to deploy these models and from my team trying to measure um for example there are a lot of public benchmarks on text to seq we're not very satisfied with them we don't think they're hard enough
we don't think they reflect real world use cases where there are typically multiple turns of interaction with the model to try to get the right SQL and get the right output so we're building
benchmarks for that um we think function calling benchmarks that exist in practice are pretty weak they're not hard enough and they're not kind of reflective of real scenario so we're
building an Enterprise function calling Benchmark right now internally um and everybody should be doing this every customer we work with should be building a benchmark to measure the thing they
care about again we're doing this the hard way so that our customers can do this the easy way are you planning to open source that too um so the benchmarks it's always a tricky thing with open sourcing
benchmarks a lot of people are starting to do a thing where they open source part of the Benchmark but not the whole thing thing because once you've open source The Benchmark people can just
train on it they can game it um if you keep a little bit of it closed that gives you a way to maintain kind of a clean leader board yeah it was always like my intuition would be that as soon
as you open source any Benchmark and they will somehow get leaked and and you know appear on the training data sets um so I'm always surprised to see all these you know open Benchmark because for for
me like even if you don't want it somehow it's like it's must be super easy to get them into the training it can get into the training data just because it's in the training data doesn't mean the model memorizes it or
will focus on it it's still a small part of the training set all things considered and the other piece is that I think most people are acting in good faith um but yeah it's hard um and once
you have a benchmark what is it good heart's law like you know any metric ceases to be useful as soon as you know you start you know evaluating people on it um so we constantly need new
benchmarks and I think maintaining a little bit in-house and a little bit externally is a really good way to strike this balance because we do want to give this to the community having to
submit your model to a leaderboard is really slow yeah I want people to move fast and are we tuning our bench smarts depending the type of model we're trying to optimize for oh yeah like we're you
know we're constantly thinking about like you know should we build an independent you know model for function calling or a model specific for chat or what have you um you know those sorts of
things we've already you know data rck has you know built independent models for text to SQL um should these models be Specialized or should they not be and you know which Benchmark should we focus
on this is constantly a discussion so it's really important to me that we you know we always have our eyes on what's important to our customers so what about evaluation uh but for customers like I'm
fine tuning my llm I'm funing maybe dbr RS on their bricks can you share like the best practices around evaluation what you what should you check so I think the evaluation is still really tricky
and I don't know I have like a clear best practice that everybody should follow it is somewhat bespoke but the you know the way that I think about this is to the extent that you already have
an existing pipeline um you know like a human driven pipeline where this model is in practice and being checked or even just a process where things are being checked trying to leverage that Pipeline and having humans
review the outputs of the model is the very best way to do this that is the gold standard automated evals are nice so that you can iterate but automated evals are very like very finicky if you
want to have a perfect automated EV that will always give you the right answer you typically have to go with a multiple choice question and nothing is multiple choice in the real world if you want to
have you know unstructured outputs that some that check automatically you have to use another model to check them and that's very finicky and unreliable it works it's not perfect and it's good
enough for a lot of scenarios but it's still finicky and unreliable and it's not as good as the gold standard of having humans review so it's kind of worth thinking about that hierarchy you
need something that's fast with using LM as a judge is a great way to do that it shouldn't be the end all be all if you can possibly help it so it it does get tricky in these scenarios um and my team
is grappling with this every day as well we read our eval sets the entirety of our eval sets very carefully um because they're often surprises we look at the outputs of the model in addition to
having an llm as a judge look at them because sometimes the llm as a judge is not doing what you think it's doing or behaving in a way you like sometimes you know your model is writing correct SQL
but the SQL implementation you've chosen use like sqlite doesn't support the commands and so you're erroneously getting error messages we've seen that before so there's a lot of just it is
still a very Hands-On process the way that I think of deep learning right now is it used to be that we had to spend a ton of our time building software and then a little bit of our time writing
unit tests with deep learning in some sense the software just comes into being on its own you just push a button give it some data hand it some gpus and the software build itself so most of the
work is going to be an evaluation measurement and test and that's kind of that's the price we pay for having this amazing technology that builds our software for us so then it's use as ADV Dan and always human in
the loop then as best practice at least for now yeah to the best extent that you can and I know it's easier said than done for me to say like oh just use a bunch of humans if only we could use a
bunch of humans this is why I say when you're an Enterprise you're probably building a model to augment an existing process to the extent that you can instrument that process and add you know
exploit that process process as evaluation that can be really powerful that's a great way to do this without incurring a lot of time and energy um you know it's much more difficult if you
don't have that existing process that's where llm as a judge can come in handy but again there are trade-offs to both of these so easy for me to say just to have humans do it um much harder in
practice and I totally respect that so you mean like if you are whatever the easiest use case like a a bol that replies to you with I don't know a rug use case for example when you saw the
the the answer to the customer then or to whoever is viewing that then they can judge it as a positive or negative or good answer bad answer like crowdsource the evaluation to anyone who could use
this application mainly yeah the way that I would think about it is that you know you let's suppose you have a process where your model is trying to trying to do some kind of classification
let's say it's trying to put something into one of a thousand categories and right now a human would do all of this but you want your model to take care of the easy scenarios and you know have humans focus in the
hard scenarios that's that's better for everybody you should still like you know once you deploy that model initially maybe you don't even deploy the model you kind of use the model keep track of
what the model is predicting and test it against what the humans are predicting maybe once you've deployed the model you still have the humans check you know one out of every 10 model predictions and
you know see what the human says that's a great way to find out whether your model is really behaving the way you want it to you shouldn't just you know release it into practice and hope for
the best sh do the humans then exactly because you're you know you're trying to make humans life humans lives easier with these models but you know the flip side is that the models actually have to be
good and reliable amazing so in terms of future of gen and what is coming in my view in my mind what I see coming is a lot of agents and a lot of multimodal stuff so what is your view on that
Jonathan so if you see that coming you're smarter than me um I I I always say during these podcasts that I try not to predict the future okay um the present is complicated enough and I
think if anyone were trying to predict the future over the past several years um they would have been completely wrong so I think it's better to accept that we're not certain except that you know we're you
know we don't know what's coming prepare for anything um and preparing for anything means getting our data in order figuring out how to measure those are things that will always stand the test
of time the rest the technologies will change what we want to do with them will change the ideas will change but data good evaluation and good processes to integrate gen into those will always be
valuable and so I you know that's really kind of my my prediction of the future the rest we'll have to find out nicely said with me the data part is really sticking because because as you
start tuning and training models you start seeing that if your data is not in the right place if your quality is not good then again R like trust in trust out right the usual ml problem so it's
really interesting to see it coming back and all the time yeah on the data front I kind of I often get asked like in the age of gen do you even need to clean your data or structure it can't you just
throw it all into the model and the answer is yes you need to the more structure you can give the better if your data is structured in a SQL database you have a model right SQL it works much better than just handing all
the data to the model in the context and having the model flatten it so you know the structure Quality Cleaning all of that stuff is more important than ever in a gen world and I just you know who knows what the future
will bring the best we can do is to be prepared all right anything else oh I have a last question I so you said that but I I saw you have a bet on is attention o on need can you can you
comment on that what happened yeah so this is my dear friend Sasha Rush um at Corell Tech and hugging face um he you know asked on Twitter a couple years ago anybody willing to bet with me that you
know Transformers and and networks with self attention will still be the you know the best way to train for natural language processing in a few years and I was dumb enough to say yes and Sasha
being the wonderful person that he is literally created a website for this bet I believe we bet you know a couple shares of mosaic against a couple shares of hugging face both of which become
much more valuable since then so you know hopefully one of us will be making a good donation to charity at some point in a few years um but you know the I bet that attention will still be
the deao way of doing this or the default way and my reasoning is pretty simple um I love to ask this question I'll ask both of you um what were we we using for NLP prior to the Transformer
and self attention bug of GS good guess what's the kind of neural network model architecture be be behind bag of GS or just kind of in general prior to Transformers what was our pre-
Transformer well it was um just embeddings and things like gloves I would say oh we oh you're you're going back to 2015 I'm going back to the 90s 90s so so it was it was a recurrent
neural networker in lstm oh yes okay okay those are the default architecture prior to a Transformer and now I like to ask which is older me or an lstm I would never guess an age of a person tell me
which me or an lstm I'm not asking you to guess the age just you know which one's older you then um yeah are you gonna guess me or the lstm I'm gonna say you so I'm barely older barely it has been a generation
since we had an another great neural network architecture the convolutional neural network for computer vision is still in my view the best way to do computer vision networks the vision
Transformer has still not surpassed it and the convolutional neural network is older than me we get new neural network architectures very rarely it's really hard to come up with them it takes a lot
of work and a lot of people failing and someone gets lucky so I you know I'm willing to take this side of the bet because my bet is took us a generation to get a new architecture for NLP it's been more than
a generation and we still haven't gotten a new architecture for computer vision I'm willing to bet the Transformers are going to be here to stay for a long time very interesting and then since we are
talking about conv convolutional neural networks so uh since we have dbrx is it are we going to plan to release any other model like text to video text World deal again I don't like to talk
about things before they're out um if this is going to go out after data and AI Summit I can say that I'm very excited that datab bricks has announced our first text image model that is
available for our customers trained in cooperation with Shutterstock um which is you know the leading stock photo company in the world they have a data set where you know all the data is you
know licensed in you know a careful way from content creators content creators actually get compensated every time this model is used on data bricks or on Shutterstock um we've been working on
this for about a year it's been a huge collaboration with our friends at Shutterstock I can't be more excited that this is you know the first of many things to come out of that collaboration
um we hope this gives Enterprise customers a lot of confidence in a text image model I know there's been a lot of kind of concern and nervousness around working with these models given you know
what data they might have been trained on and where it came from um this is something that I feel very comfortable standing behind as you know something I trust something that I hope Enterprises
can trust so I'm really excited that we've now announced this and it's available to our customers beautiful and last question from my side so how is the day-to-day life in Mosaic ml team look like like
what is happening behind the scenes we're doing science every day we're there a few topics I set the agenda and there a few topics that we really care about um and we're doing science every
day to try to figure out how to improve that we only work on things where there's no known answer otherwise it's engineering and so we're constantly trying new things pushing the envelope
testing things we've read about in the literature to see if they actually work and whether they actually work in scenarios our customers care about that's every day for us um we're in the
lab we're trying out new ideas we're trying to push the envelope on what's available for us and our customers so literally get the paper try to implement it and see uh or try out a new idea nice
um completely new ideas things that we think are relevant for our customers but you know we do have to test a lot of papers because papers come out um but but we don't know whether they really
work in the settings we care about just because it's a paper just because it's published doesn't mean it works doesn't mean it works for a customer so you know we spend a lot of time thinking about that and sometimes
papers work phenomenally well often they don't because we're testing them more rigorously and at larger scales than people in Academia can because we have more resources so you know sometimes we
succeed sometimes we don't I try not to talk about the things that don't work in public because you know a L of them are written by students and I don't have any reason to want to make a student's life
difficult I was a PhD student not too long ago um but in general we need to test everything and have data to back up what works you can see this in our new paper on Laura everybody's talking about Laura
everybody on Twitter seems to love Laura um you know is Laura really good just because people are talking about it and hyping it up doesn't mean it's good and so we had to go and figure
it out all right don't trust TR Twitter yeah don't trust Twitter don't trust your friend of a friend at open AI don't trust the paper trust real data and real science thank you so muchan uh thanks
for all the insights thank you Maria you know for being here here with us I hope it was a good sis and you you know had a good time and I hope we'll see you soon this was awesome and I can't wait to do
this again soon when we have more to talk about guys thank you awesome thank you so much