Cleanlab
Synthetic Data SDK
Hey, today I have the pleasure to host John. Hi John. Hey Yousef, good to be here. I'm super super happy to have you.
And can you introduce yourself to the community? Yeah, sure thing. Pleasure to be here. My name is John Sullivan. I lead the go to market team at a company called Mostly AI and we're an Austrian
founded Sirib startup working on a a pretty big problem albeit a small team which is that the majority of the world's valuable data remains underutilized. It's difficult to access and share. Often it's proprietary. It's
it's personal information about us, financial data, healthcare data, behavioral data. And even if you do get access to that data, it's difficult to um extract insights and value for a lot
of us. So what we do at most AI is enable data access and data insights for everyone. And we do it through a combination of our open source SDK which I'm going to demo today and works really
really nicely in data and also our data intelligence platform uh patent pending.
Um excited about the conversation.
Cheers. So I think you talked also about mos.ai which is an Australian uh company which is awesome and um and is there like how does your company operates?
like is it you're just like building open-source technologies or you still have like other things to handle on your side? Yeah, it's a combination of open source and also commercial options. So
on the open source side, our synthetic data SDK allows folks to take a real world tabular data set and train a generative model from scratch. It's based on our model architecture. It's
not based on anyone else's. And then you take that we call a generator which is a combination of the model weights and the metadata and from that generator you can flexibly create privacy preserving
synthetic data. Um so that's the the core open- source option that we have.
We also have a mock data option and then on the commercial side the enterprise offering is the mostly AI data intelligence platform. We obviously got inspiration from you folks on on the
naming and um that's a full enterprise option. It has a excellent UI so that basically anyone can work with synthetic data train models and extract insights.
We also have a a Python native assistant which makes it easy in natural language chatdesque conversations with your data.
So a mix of open source and enterprise options. Yes, that's amazing. like data bricks they have their they have like the open all the open source technologies like spark mlflow delta
lake apache iceberg unity catalog and that's are that's the core of everywhere like a big company you need to have this open source uh yeah it's huge it's important for transparency uh with the
community as well so all of our code then is is available in our GitHub repo um it's not a blackbox folks can understand the model architecture and understand why our model makes certain
decisions So yeah, took inspiration from from data bricks and others and in various aspects. That's amazing. And you can can you tell us more about your uh your data bricks your sorry your SDK
regarding the synthetic uh data like what are like the requirements and how can I start using this SDK? Yeah, cool.
Of course. Um so it's available under a fully permissive Apache v2 license. Um so you can do whatever you like for as long as you like with it. um fully feature complete from a model training
and a data generation perspective. So everything we make available in the enterprise platform you can also do through our Python syntax and we have a few core primitives um mostly train
mostly generate mostly probe. Um so it's kind of a in a nutshell two steps. You train a generative model from scratch and from that model which we call a generator you flexibly create synthetic
data. So that's in essence what we do.
The synthetic data looks and feels and means the same as your real world data in terms of requirements. All you need is your your source data. So you take a tabular data set CSV park time series uh
relational database and feed it into the model. It learns the joint probability distributions. So the patterns distributions of the real world data deletes the real world data because
we're a privacy first company and then you're left with a model that can create statistically representative synthetic samples that are 100% private. So easier to access, easier to share, and easier
to consume across uh your teams, but also externally with partners. Yeah, that's um that's amazing. And can you maybe show us a demo of how we can get started quickly? Would love to. Yeah,
absolutely. So um let me share screen here. I'll do the entire screen. So Yousef, is that coming through okay for you? Yes. Excellent.
So this is who we are. Data access and data insights for for everyone. And as I mentioned, a combination of our mostly data intelligence platform and also our open- source synthetic data SDK. So I'll
focus on the SDK for the demo. This is our GitHub repo and you can find all the information needed to get started here.
As I mentioned, there is a few key primitives the spine of the workflow. Um, and to get folks help or help folks get started, I should say, um, we have a guide here and some usage examples. um
and also a cheat sheet and more advanced tutorials. So the cool thing is you can um by default create representative privacy safe synthetic data but you can also create better than real data
depending on your use cases. So you can mold and manipulate um the data in synthetic form things like conditionally generate a subset of your uh customers.
You can 10x 100x the synthetic data. You can rebalance certain attributes. You can change the temperature of the synthetic data to bias towards outliers.
So all of that's available for folks.
I'll give a quick shout out at this point to uh the mostly AI prize. I mentioned it to you before Yousef. We're running a competition uh to see who can generate the best synthetic data across
both a single table setup um and a sequential data setup giving uh $100,000 in cash away uh to promote open data and open source. Uh the competition isn't exclusive to our SDK. You're welcome to
use any open source tool, but I think it's pretty cool for a small series B startup to be um yeah run the competition like that for for the community's benefit. That said, I'll move into data bricks. It's a Python
library, so you can pip install it in any Python native environment. The scenario I'm going to run through in about 5 minutes is we have a sensitive production data set as you can see on screen now. It's the US census
income. It's uh customer data, demographic data, and typically um it's usually quite locked down. So when we work with customers, they'll have a production area and that data is proprietary, it's sensitive, it's
difficult to access, it's difficult to share.
So in the demo, I'm going to show the simple endto-end workflow of how you can use our SDK directly in data bricks.
Again, it's Apache V2 license, so people can run with it from today and it's fully feature complete. a few simple simple steps. I've pip installed um MOSI already and restarted the Python kernel.
So the first thing I'm going to do for the demo is um initialize the SDK in local mode. So you can run it in local mode to consume the resources you have available um on your laptop or in data
bricks wherever it is that you're pip installing our Python library or you can run it in client mode and will talk to our platform which is typically deployed within your own secure environment. So
I've initialized the client in local mode and I'm going to read in that table I showed before. And um next I'm going to do uh uh training. So I'm going to mostly train and train a gener generator
uh from scratch. Um this is going to print out um our endto-end pipe but also a training log. And while that's running, let me just explain a little bit about what's going on. So the model
is going to um pull, analyze, and encode the training data. So the income data set that I showed before and it's going to learn the statistical patterns and distributions of that real world data
and then create a model which we call a generator.
You'll see in the configuration I've kept it quite simple for the purpose of a demo. The only thing I've done is cap the training time to 1 minute. So here with the model training already started
that should finish after 60 seconds.
There's lots of other things you can do.
I mentioned before it's fully feature complete. So from a training and generation perspective, everything you can do in our enterprise platform, you can do with our open source SDK. That
means you can uh invoke differential privacy. So we have built-in privacy mechanisms and by default all of the synthetic data is private. But if you need an additional mathematical
guarantee on top, you can invoke differential privacy. You can also create multi-table and time series synthetic data. So you can configure your primary and foreign key. And you've
got flexible flexible compute as well.
So CPU, GPU, whatever your preference is. We also support text data. Um so tabular text, time series is all possible via the SDK.
Um so what's happening here? You'll see uh we are almost finished with training the model and you can see that the training log the validation losses begin to flat flatten out. So even though I've
capped the training time at 1 minute, it looks like the model has become saturated on this training data set which is about 50,000 rows and 15 columns. So a ra rather small data set
and now we are complete. So we've taken the real world data, we trained a model to understand the statistical patterns and distributions and it's left us with an artifact which we call a generator.
Again that's a combination of the metadata and the model weights. And um what's pretty cool is that you can generate a generate a a full HTML report directly in your your notebook. Um and that's
really crucial because synthetic data is still while it's very very powerful technology, it's still rather niche. Um so the two questions we always get asked are how do you know it's accurate and
how do you know it's private? Because ultimately it's a drop in replacement for your sensitive production data. So you need to understand and have trust that it's capable of of doing the job.
uh and fulfilling your use case. So in this model report here, you'll see there's an overall accuracy score of 95.4%.
Um but there's also a bunch of statistical analysis to demonstrate that the generator has effectively learned the training data. So you can see there's a correlation matrix here.
You've got your original data in gray, your synthetic in green. You have a breakdown of the uni variant distributions. So each of the columns you'll see the synthetic data tracks the
um original data very closely. Same should go for the bary distributions. So the pair-wise relationships again the synthetic data in green mirrors the gray original data. You have an accuracy uh
per column and also a matrix.
We provide even cosine similarity for highdimensional spaces and we have our privacy metrics as well. So our distance metrics are plotted in the report. So that's a really important piece of the
workflow because you're taking the training data um you have access to it which ultimately you want to give this generator to a colleague, a partner, another data consumer so that they can
flexibly create synthetic data and they're going to ask is it accurate? Is it private? Basically can I trust it?
And that's where the model report comes in. Yeah. And you don't you have all the statistics generated for you. So you don't need to do this on your behalf which can be time consuming. Yeah.
Exactly. And with that being said when we talk to customers we always do encourage them to first off um evaluate the quality of synthetic data outside the platform. So we do have in the
tutorials I showed earlier we have um train synthetic test reel. So we tell folks look create some synthetic data take it out of the platform or away from the SDK workflow and um train a model
using the synthetic data and the input data the training data the real data have a hold out uh 20% from the original data and then compare the results. So we definitely encourage folks to try it out
themselves. You don't have to rely on our model report. And then what happens is that's kind of the penny drop moment because people say this is this is my model. Um this is my experiment. I
understand the parameters and I can see the quality carries over into synthetic data. And from there once it gets kind of productionized people relied in on the model report because they have
understood from their own experiment that uh they can trust the accuracy and the privacy. Uh so once you have that report there's a few things you can do.
Another one of our primitives is mostly probe. you can without actually having to create a synthetic data set, you can just probe the model for statistically representative of see safe synthetic
samples. Um, and what's cool about these is so there's a 100 for example, let's try 100,000. In a matter of seconds, you can probe that model and get statistical uh statistically representative
synthetic samples. And they're fully disposable. So I can keep generating and over and over again and probing I should say over and over again and I'm going to get new synthetic samples which is
really really cool. Um and it gives you the ability to inspect and analyze the data on the fly. I mentioned earlier by default everything is highly representative and privacy safe. But the
beauty of the generator and the beauty of synthetic data is that you can augment it. So you can mold and manipulate your data set. For example, one of the things you can do is conditionally generate a synthetic data
set. So in this example, I'm going to feed in a seed file to that generator which understands my real world data and I'm going to fix two columns. I'm going to fix the uh sex and the age. So I want
only 35year-old males in my data set.
And what that gives you back then is a mixed real synthetic data set. So I can single in on a specific cohort of my customer base, which are 35-year-old males. um fix those attributes as real
and then I can synthetically generate all of the rest of the attributes which may be deemed to be too sensitive in in production form.
Have a really cool use case which we're promoting at the moment with Swiss Post and they use conditional generation via our open source SDK to fix the longitudinal and latitudinal columns in
their data set at an area code level. So that was deemed to be sufficiently aggregated. But what they couldn't do was access the granular customer information. So they fixed the longitudinal, latitudinal, real geo
coordinates of their customer database.
And then they synthetically generated all of the customer attributes. Um, which is really cool because they went from the ability to um access 11% of their customers who had given consent to
100% of their customers in synthetic form. So that's a a nice little use case for conditional generation. And then we can move on and use our third primitive which is mostly generate. So I can then
um actually create an artifact a synthetic data set that we can push back into um our catalog. And um for this example I'm just going to use a quick 10,00 examples or samples I should say.
And uh as you can see we're generating that data and it'll ultimately be published here super quick. You can see from a an eye level inspection that it is highly representative of the original
data. Um and you have all of the structure retained but also deeper the correlations and the statistical quality is represented as well and retained in synthetic form. Um so I have my data set
um which I've generated from my trained model and now I'm just you can save it to the catalog and then you can start doing your basic stuff transforming cleaning or whatever you need to you
need. Exactly. That's it in a nutshell.
Um the thing I'd mention is uh obviously once you get into your um your catalog here um I have an example already. This is the synthetic version and I guess the end point is because it's privacy safe
because there's no real world customer information available there. You can enable faster and safer data sharing and data access and that's the value that we bring uh as partners and I should have
mentioned at the start obviously formal ISV partners to data bricks uh faster safer better access to sensitive data using a synthetic replica. So in this example here, I've given it access to
BI, data scientists, etc. So that's it in a nutshell. That's the uh synthetic data SDK and that's how it uh operates inside of your data bricks environment.
I've noticed it's very easy to use. You just need pip install and and you're and you're good to go and just learn some few APIs to generate the data and then you can do whatever you want and it's
very easy, very fast and robust. I guess I think you no longer have excuse to start doing what you need to do. You can start just use the synthetic gener generic data and you're ready to go.
Most important open source which means no fees are accurate except the the compute fees of course. That's it in a nutshell. Um and going back to kind of our mission statement, we just want to
broaden the access um to data such that people can extract insights and it's not there to replace data scientists or data literate folks. It's to give folks the ability to access data that they
wouldn't normally have access to and then ideulate with it and brainstorm and think what can I do with this highquality data and folks then through the power of LLMs our assistant your
assistant can ideulate and talk a natural language and create a use case and a workflow and a a more finished product to bring to your data scientists. So maybe do 80% of the work and say look I now want to get this into
production can you help me with the last 10 15 20% of the task to operationalize things. So it increases the ability of a organization to innovate based on you know that valuable proprietary data that
they have. Awesome. Thank you so much for John for this uh for this introduction to your Python SDK. Looking forward to to play with it. Cheers.
Appreciate the time Yousef and take care. Bye. I like.