Executive Interview
Omnigent: Open-Source Meta-Harness for AI Agents | Matei Zaharia
Matei Zaharia · Co-founder & CTO · LinkedIn
Hi. My guest today needs almost no introduction in the data world. He created Apache Spark, co-founded Databricks, and is now its CTO.
He's also a professor at UC Berkeley and behind massive projects like MLflow, Delta Lake, DS Pipe, Unity Catalog, and more recently, Omnigen.
Matei, great to have you here.
>> Great. Thanks for having me.
>> And of course, my famous co-host, Quentin.
>> Hey. Hey, Matei. Thanks for taking the time.
>> I wanted to start uh with Omnigen because I mean, we all were in pink, of obviously, and that's what everyone is talking about. And can you tell us more what is it? And what the problem does
actually solve? And also, I have to ask because I heard the origin story involves uh Reynold sending you a wish list before Christmas.
Um and because he was tired of keeping his laptop open while live coding, but that's not the only uh the only >> Uh it wasn't before Christmas. It was like three months ago. Then we started
building uh this. Yeah. So, Omnigen is um what we call a a meta harness. It's basically an orchestrator and control layer uh on top of agents. You have uh many coding agents today that you can use. You build
lots of custom agents. But then you want to work with a bunch of them together.
And you also want certain features across all your agents like governance, control, or even just collaboration with uh with people. If every agent and every app builds those in a different way, you
just have a big mess. And you know, it's very hard to uh to make things work. So, uh we built this layer that kind of wraps existing agents and lets you easily um work with many of them,
whether you're composing many agents to solve the same task, or uh you're trying to control them more centrally, or trying to give them some some nice features like real-time collaboration
with other humans and stuff like that.
The main use case that we started with is for coding because like all the developers we saw that really got into coding agents ended up building very custom ways to manage lots of agents.
Um so it's an easy way for you to use like, you know, any of the coding agents out there and create a collaborative workflow with like your team and with with multiple um people as well and
multiple agents. But um we're also using it to build custom agents. It makes it really easy to build an agent and swap swap out the harness and swap out any subcomponents of it. Um so we think it's
kind of the next layer up for working with with AI basically. It actually kind of started from a few different places.
So first of all, there were um all the engineers coding at Databricks and as I said, anyone who like really got into coding agents were soon building their own way to manage lots of them. Build people were building
their own like desktop apps and stuff cuz they didn't like using a terminal.
Um and um also people were mixing different agents like Codex and Cloud Code and so on. Um so we saw everyone's kind of solving the same problem. If we build a single way to to manage these then they can share
the work and and they can build security once and build collaboration once. And that that was one angle. And that's where Arnold said he was saying well like I don't think you can do it unless
I can also see the terminal version of the agent because uh if it's just a chat app like I'm missing too much. You know, sometimes I want to run stuff in a terminal. Well, lots of engineers had
very different opinions on what they would like and that's how we came up with the idea of like wrapping existing coding agents and building something around them instead of uh trying to
replace them all with like one new thing.
And then the other angle was like building custom agents. So, there are all these teams building custom agents like Genie or internal agents, and they all ran into the exact same problems.
Like, I built my agent, but now I need an app with like sharing and collaboration. Now I need history. Now I want to try a different harness. Like, new GPT model came out, so I want to try
that or whatever. Um, so we we thought, "Hey, they they Actually, we could also build something there." This idea of layer above the harnesses where you could change.
>> So, it means, let's say I'm using uh like a Claude code or Codex at a daily basis, and then I want to switch to I'm using Claude, and I want to switch to Codex, then I can just stay on Omniscient, and
I can just choose the new one, and I keep everything here like all my like >> Yeah, you can move your session from one to the other. So, for example, a lot of the feedback we're getting from users is
uh this is great cuz I'm, you know, when I finish my daily quota of Claude, I just take the same exact chat, and just switch the agent at the bottom, and now I can continue in Codex or vice versa. And the
other thing you can do is um you can use uh like an orchestrator agent that uses multiple of these together. So, we actually include one called Poly, but you can also easily make your own or
just tell them like go launch a sub agent that's using Codex or whatever.
So, for example, whenever um I I write code with this, I I tell like, you know, use one agent to write the code, then use a different one to review it.
Um, or if I have a bug, I just say, "Hey, like, let's kick off two or three of these and look for the bug." Because, you know, one of them is basically you just need to try a lot of ideas until
you figure out what caused the bug. So, it's better to have like models with different biases, and I I'll find the bug faster that way.
>> And And can it help also because I think uh like a commit team is based on what you want to do. Maybe you need a smaller model or like a bigger one. So, could you also like split that into smaller
>> Yeah, definitely. It's very easy to just put in in the prompt for like the these uh, the orchestrator agent, you you can tell it when to use different ones. Like uh, and and they are better at different
things. Like for example, Claude is widely considered to be better at fine-end. Uh, GPT is like widely considered to be better at debugging.
Um, so uh, you you can write your own rules there. Um, we're also now shipping uh, what we call intelligent routing where um, we um, basically have, you know, we trained some some heuristics where
there's an API you can call based on your prompt. It'll like suggest which which model and which agents to use. And we think that's going to get better over time, too. This pattern of orchestrating
multiple agents is is really going to pick up. Most people with coding agents, they start with one long chat session with one agent, and then they keep asking it things to like build this, now
build that, now debug this. But that's actually very wasteful and um, not very effective way to use the agent, cuz you just get a huge context. When the context window gets bigger, the model
gets stupider because it has to pay attention to everything else, and you're paying more. Like you're paying for this like million length context window all the time. And then when it compacts it,
you forget things.
Um, whereas if you kick off sub-agents and you give them tasks, they each have like a fresh start. They just do that.
Um, and then you can have like this planner or orchestrator maybe above them. This move towards like multi-agent, which you see in like Claude Code and Code X too, actually makes that much easier to do
routing uh, on a per task basis and to like actually mix and match and and get the best of all the models out there.
>> That's very interesting because as you mentioned, some some developers used to seek with the same LLM and actually this is bad when you're reviewing the app with the same LLM that's built this app.
And how does Omnigen architecture work behind the scene? So I guess you have a server that's where we can store the history and everything the artifacts and the gateway. But is there anything else and I heard
something about Neons had something?
Like a component that was brought on to Omnigen?
>> There's a server which you can use for all the collaboration features or if you want like central management you can set policies there on all your agents. We we didn't talk about those yet but that's
the other feature that like a lot of people are excited about. And then there's something we call the runner that wraps existing agents. So for example if you just type Omni Cloud or
Omni Codex it launches those you see exactly that thing in the terminal but it also wraps them like listens to all the events and injects hooks and tools in there that let you use the Omnigen
features such as collaboration through the server.
So these are the two big pieces. Both pieces are open source and it's designed to work with any database release so that it definitely works with Neon. It just uses SQL Alchemy and also it supports launching sandboxes
so we we announced data book sandbox. Which is is a very like a VM in the cloud that spins up quickly so you can code in an isolated environment. We also support like Daytona and Modal and like
Kubernetes and other ways of launching these.
So we try to make it easy to integrate with like any infrastructure you have.
>> So how does it work with the sandbox if I type so I'm like I'm on my laptop I go to the server and I type something in in Omnigen?
And then it's sending the so let's it's sending the query to I don't know Cloud Gun.
>> So, the the sandbox is a way to launch the whole session in a cloud VM so you can close your laptop and like do other stuff. And so you can have maybe better security and isolation around it.
Actually, one of the cool things you can do with a sandbox is you can launch it and then you don't have your personal credentials and files and stuff in there. You can actually just share it
with another developer when they're debugging or whatever. Like you can actually do you know, collaborative coding or review that way. But even for yourself, you know, instead of like trying to keep
your laptop running, you you got this like reproducible environment. So, yeah, I will most of our engineers I think code in sandboxes. Now, the only thing some people like to do better on the
laptop is fine and because the developer loop is very fast, but I think even there like about half the people use sandboxes.
>> So, it means you also load all the skills and everything into the sandbox.
>> Yeah, and you have a file system.
>> I think cuz you need to have some persistence or the libraries you're going to be using all the skills to avoid having each time this the like this huge lag to to deploy them or
install them.
>> Yeah, and most of these platforms make it easy to clone a sandbox also or like share your home directory like keep it persistent even when you shut down the VM. So, it can be pretty fast.
>> We've seen that now more and more companies >> Mhm.
>> are struggling when it comes to giving the right permissions to the LLMs. Like >> Yeah.
>> How far should I give LLMs? Can I give them permission to scan my files or some specific files or send them emails? And I saw that we released a blog about contextual policies. So,
what does Omniscience contextual security policies do beyond uh coding agents auto-approve that we we have.
>> So, we spent quite a bit of time talking to security teams, cost management teams, and so on. And um we realized like uh most agent policies are are not very expressive. They're just about um
like if you just get Cloud Code or Codex, they're just um you know, does the tool call it's making match this pattern and like allow or disallow. And then the auto mode is is it instead of like
listing all the patterns, have an LLM decide if that tool call looks safe. But actually, if you think about it, like um it's it's it's very restrictive. Like for example, should the agent be allowed
to send emails? Like you know, probably you say, "Yeah, it it should because sometimes I'm asking it to send an email." But at the same time, if my agent um you know, let's say read just
downloaded some dangerous package from NPM, um that was published like an hour ago, and then it like read 10,000 company docs, and now it's trying to send an email, that's probably kind of
suspicious. It might have been compromised. So, the whether a an action is safe depends a little bit on the context of what happened before it. And if you just outright allow it or
outright ban it, you're limiting the sort of expressivity. So, the idea of contextual policies is like let's give your policies more uh information so you can make better So, let's let them track everything
that's been happening in a session. And this enables some really, you know, like really effective sort of policy designs that are both more usable and more secure than like anything you could have
configured with a static list. Um so, for security for example, like a lot of security teams, their ideal is to do some kind of risk scoring. As you take actions, your score goes up. if you and
if you do too many dangerous things, you now require human supervision. That's actually what they do with humans, too.
Like if you go to a cafe in a country you've never been and use your work laptop, it'll ask you for more checks for security.
Um so that's one one thing. Um another super powerful one is like when I tell my agent at the beginning the task, why don't you narrow down what it can do based on that initial prompt? Like I
told it review this code, so it should not be sending emails. If it does, maybe it's been prompt injected. This is what I think people will want um to to make agents like both more secure and more
usable. Um and then the other cool thing about being able to track state like this is um you can also start having policies about like spend of the agent or of the subagent. So for example, I've
had agents spend like $1,000 by mistake because they decided to like create a giant log file or something.
Um it's actually very hard to make them spend that much if they're doing reasonable things, but sometimes they do sort of stupid things. So I can just tell the agent like, "Hey, put a put a
policy on yourself and um just ask me for permission after like every $10 spent." And then it'll run, you know, when it gets to 10, it'll ask me, "Do you want to continue?" When it gets to
20, it'll ask me, and that way I can like way more easily control this. And I just get a notification then it's asking me to continue.
Um so yeah, I think the idea of adding state or context uh makes policies a lot more useful.
>> Yeah, and I think having also, like I don't know if it's something we are considering, but having all the history, like you can unlock so many interesting use case. Like you can see where it's
struggling, you can see what's so cost ex- like what's so expensive, and then based on that you can like you can improve the system just to make it work better.
>> Yeah, yeah, that's true. Yeah, and I mean that's another pattern we're seeing as um if you've got agents running, whether it's like coding agents or your own app, you really want to collect and look at
all the traces, and you find so many opportunities for improvement. Like we found, you know, people some of the skills and some of the tool descriptions and stuff for our engineers were just
like super long, and you waste a lot of tokens just like reading that, and you don't need it most of the time.
Um so nothing really beats like look looking at the data to improve these.
>> I think I can see great integration with the MLflow 3.0. That was like literally revamped for the LLM area to be able to understand tracing. So this can be like this perfect integration knowing that
both MLflow and Omnizon are open source, so we're like really um very easy to very easy to be used quickly. And I think this is really leads to the question of why did Databricks open source Omnizon under
Apache Spark 2.0 instead of just keeping it keeping the orchestration layer proprietary?
>> So yeah, it's because we really believe that there's going to be this new software layer above agents and above harnesses where uh you know, they're all basically like text in and text out, and you you want
to manage them in a common way.
Um so we thought that that's going to happen. There's a lot of value in this layer. Like for example, all the security stuff I talked about, you can write it once and it applies to like any
agents that Omnizon understands or any custom agent or whatever. It really benefits to have like a common layer where any new agent software can like easily plug in, any new control idea or
like UI or whatever can plug in as well.
Um and that makes sense when it's open source. So um so that's what we wanted to do. And I actually we're finding a lot of companies now that say, "Yeah, I built something like this internally or I was
designing something, and actually I'd love to join forces and collaborate on this because, you know, everyone benefits when we add features to this thing and when all all of them can work
well together.
Um it's not that different from like MLflow, which was for MLops. Like the problem in there is there's so many ML libraries, so many like pieces of infrastructure where you can deploy models and stuff. If you try to
build your own MLops platform like from scratch, you'll be out of date in like 6 months with all the new stuff that's come out. But if you build on this open-source one, uh people just integrate in one place
and like everything can work together and you share that effort with the community of keeping up-to-date. All right?
>> Well, anything we are missing on Omnigens?
>> Uh I mean, I just recommend people try it out, try the collaboration and the live editing. Like now, all all the ways I use agents, even for like non-coding, are through this because it's so easy to
share stuff, to write documents together, um and um and and just to like get many opinions from different models on the same task.
>> Yeah, yeah, just go give it a try. Uh like one of the feature I love, by the way, is how like I have code running on my laptop, and then I just go on the server, and then I just, you know, can
>> If you can use the mobile app, too. It's It's a It's a big deal. Yeah, save a lot of effort. Yeah.
>> Uh that that those big struggle always keeping the laptop open like walking while having the laptop because you don't want your agent to stop.
All right. Well, Matei, thank you so much. And and yeah, just go give it a try, omnigen.ai.