Databricks SQL Performance: 25% Gain, Free Edition, Warehouse Migration

0:00Well, we have like 10,000 dashboards. My first question, do you need 10,000 dashboards? And like we ask ourselves that all the time. Do we need all these dashboards? Actually, I'm of the belief that you need about five dashboards and a genie. Basically, every the reason why you have dashboard variability or variance is because everybody wants to see data their own way. And it all depends upon how high up the food chain you are. If you're high enough, oh,

0:22we'll make you your own dashboard. If you're not, there's export to Excel. Manage yourself. >> We will be happy, >> right? Today I have the pleasure to host my friend Franco Patano for the second time in the channel. He was invited I think one year and a half ago to talk about streaming tables view but mainly how

0:46Databricks is pushing the boundaries with the DBSQL and he's back. Hi Franco. Hey everybody. uh Franco Patano. I'm a principal solutions architect at data bicks. Been with the company uh going on seven years now. Uh I joined uh in the early days before uh we had a SQL product. They I I didn't know it at the

1:08time but databicks wanted to go into data warehousing and my background was data warehousing and DI uh and I had no idea why they originally hired me. I uh remember I had a conversation with one of the co-founders. I was like what am I doing here? It's a company full of data scientists and data engineers. He's like, "Don't worry. Don't worry. Just study, learn as much as you can. Just wait." Uh, and then a year later, we

1:30bought Redash and started our SQL product. So, unfortunately, I haven't had the chance to meet you this year at the data summit. So, I wanted to know first um what was your impressions about this year conference and what was like the announcement that maybe impressed you the most? This one's hard.

1:52So, first off, the reason why uh I think we haven't seen each other is just the growth of Summit over time. When I first joined the company uh in 2019, we barely fit I think in one of the Mosone and now we've completely taken over that part of San Francisco and there I think this was a record year of people showing up.

2:14>> We had 22,000 people on site if I'm not wrong. uh which is >> insane. >> Happy to be here. Uh happy to be on this journey. I used to have crazy wild aspirations of Databricks six years ago, right? People used to say I was nuts like uh and nowadays it's like man even my wildest imagination does not uh

2:36kind of match where we are. We've exceeded all of my aspirations. So it's great. Uh it's also on the flip side like I used to be able to pinpoint things at summit like this thing uh is going to be wild. Now it's like I can't pinpoint just one thing anymore. It's kind of like there are so many great things happening cohesively across the

2:58platform at the same time. One of my favorite things that is not like earthshattering. It's not like amazing tech. It's not like some research thing that came down from the gods, the smart gods that we have at Databricks and and make it happen. It was actually a super duper simple uh thing as Databricks one. How do you put a front door to the

3:22data intelligence platform for the every user, the regular business user who who doesn't know what to do with the notebook, who can't write code, uh who who basically can uh just dictate or type what they want in natural language, right? If you think about these new AI tools, that's what we're enabling is a all new interface to interacting with your data platform. you know, we're still going to have backwards

3:44compatibility code things for the engineers and the scientists, but now there's a whole new type of user that we have to build for. Um, I think it's the early days of this. I think that uh we're going to see a lot more uh simple interfaces to accomplish tasks. You're seeing some of that with agent bricks right now. We're going to be able to deploy agents across these enterprises

4:07to handle all the the repeatable work. That's another amazing thing uh that I think was announced that I don't think people truly understand the impact it's going to have right now because they can't see it. But I think Agent Bricks is going to be uh ridiculous. Um how it's going to modernize these enterprise processes over time. There's

4:31going to be friction. Uh a big part of this I think people gloss over. It's about people and process as along with product. Databricks brings the best product for sure, but like you've also got people in process to worry about. So, it's not always just the product. There are a long tale of I think super important other things that that didn't

4:53get a whole lot of wow. One of them I saw on LinkedIn this past week, somebody said, "Oh, AIBI is amazing." And somebody had a complaint like doesn't even have undo redo. It's like, "Well, maybe you should check again because it's got undo redo." and they released like a ton of features uh at Summit and just a few weeks ago after Summit that basically brings AIBI up to what I

5:15consider like 95% parody with these BI tools to get to I don't think people understand the gravity of this. AIBI wasn't a product 5 years ago. It did not exist. In the past three years, the engineering team has built 95% compatible modern BI tool that just it makes itself. You can text to viz like

5:37you can make visualizations just from a prompt. Uh Tableau couldn't do that in 10 years. Tableau was messing with NLP 10 years ago. I remember I was one of their testers at one of their summits. Uh they were messing with NLP 10 years ago. Tableau couldn't figure it out in over 10 years, right? Uh and our engineers figured out figured it out in

5:59three bringing 95% comparity to the market. I think that's amazing. And I don't think we've done enough talking about it. I think partially because we want to wait till it's 100%. I think my I think our founders and leaders are a little too safe. They're really hesitant to pull the trigger until it's 100%. But I got to be honest, I've got customers that are migrating from legacy like SSRS

6:21uh and maybe some naive PowerBI and they're like, well, it handles 100% of what we need and so why would we pay licenses for a BI product? The usually the second question that comes up is like well we have like 10,000 dashboards and it's like my first question do you need 10,000 dashboards? And like we ask ourselves that all the time. Do we need all these dashboards? And and it's like I don't we don't think you do. Actually

6:45I'm of the belief that you need about five dashboards and a genie uh to back them up. Basically every the reason why you have dashboard variability or variance is because everybody wants to see data their own way. And it all depends upon how high up the food chain you are. If you're high enough, oh, we'll make you your own dashboard. Uh, if you're not, there's export to Excel. Manage yourself. >> We will be happy.

7:07>> Right. >> So, I have one customer that migrated all of their legacy SSIS, SQL Server and SSRS reports to uh, LakeFlow and AIBI and they had they went from like thousand a thousand dashboards to I think 10 and a genie and the business got their their analysts were elated. um

7:30because they had less immers requests because Genie was just taking care of it. So I think it's the if you were to pigeon pull me to one thing, I think I'd pick Databricks one, but I think everyone would be upset that I didn't pick the 25% increase for no extra cost in DBSQL. They're like, why

7:52wouldn't you pick that one? Uh because that isn't that amazing. you basically gave away uh performance for no extra cost. Meanwhile, the competition is giving performance, but they're making people pay for it. Uh they're kind of like recouping some of that cost. Uh whereas database took the other way approach. We're like, "No, here's here's

8:14an extra 25%." Uh our nerds actually took a look at the data and was like, "It really doesn't affect us that much to give them a little extra bump, but it actually impacts the customers greatly." Uh I think Joe just did an analysis and he's like the bigger the longer the queries the bigger the uptake you get and DBSQL is already really good really price performant at small queries. If you have a lot of shortrunning queries

8:37that fire off at uh variable time during the day serverless you cannot beat the value of serverless SQL that we've run a crap ton of benchmarks and we've tested this and we've modeled it. There's no way to beat if you have a like a bunch of short running jobs that have like less than five minutes that spawn every hour from like airflow or something. Serless SQL just can't be beat. It's

9:00it's the best at handling that stuff because you don't pay for ramp up and ramp down. You don't pay for scale up and scale out. It just happens instantaneously. And then on top of that, for the stuff that it wasn't super great at was the longer running queries, we basically just trimmed off 25% off that and made it more competitive with against other products. So, you know, isn't that the best thing? Eh, we've

9:22been giving customers price performance away for the past 3 four years. It's been getting consistently better. There's a chart that we have, probably everyone has seen it. Performance 77% we've given back to these customers. What's an extra 25%. Actually my argument is we've given 77% over the past three year two years and now you give an extra 25%. Is that a big deal? I

9:44mean to the customers it is but in the grand scheme of things at Databricks actually like a little extra performance boost will make the business users happy but that those are kind of the things that were super interesting. Uh you know and then there's a bunch of runners up. The iceberg compatibility uh is fantastic. The unity catalog upgrades were fantastic. uh lake flow you know

10:07super excited about spark declarative pipelines uh really hope the open source community like embraces that those are like some of the things I'm sorry I don't have one big thing it's just I think Databricks is too big now for one thing anymore oh how did I forget oh my gosh Databricks free edition how long have I been complaining internally about

10:29community edition for years no our community edition which was the free version of Databricks just to kind of like set the because there's a huge amount of confusion about this. There is free trials at Databricks that it's two weeks free trial. You get a un a little bit unbounded sizes of warehouse and we have it's all serverless. We include the compute and

10:51the storage but it's only free for two weeks and you get to choose sizes and it's a little bit different of an experience. Data's free edition is the spiritual successor to community edition where it's always free forever permanently. You just get really really small compute like it's really only meant to like train experiment and learn. Give you a great example. >> Yeah, you're a student or maybe a

11:14developer or someone aspiring to I don't know increase or enhance and then like DB knowledge. >> It's fantastic. And finally, Databricks SQL is now available. Actually, what like what it really for all you nerds out there, there's a brand new architecture type that some of you older Databricks customers might know this. Modern Databricks is actually completely different architecture

11:36underneath the covers, completely serverless, managed storage. It's a whole new app for all intents and purposes. It's a whole new app. Databricks free edition is built on this new architecture and that's why it has all the new tools including Databricks SQL because Databricks SQL was the first serverless product to actually be be serverless and now all the other products are. So that's kind of why it's

11:58been a long long time and a pain of mine because we've had the new products DBS SQL out for years. That's been f what 2025. It's 5 years now. We launched in 2020 but it's never been available in the free edition. And so finally now we have DBSQL available on databicks free edition. It's absolutely fantastic. You should check it out. Since you're mentioning you're talking about DBS SQL,

12:21what if you walk us through maybe the latest features we have added because I know a bunch of people were complaining about not having uh store procedures, amp tables, cursive cities and bunch of other features. I call I call these lovingly uh the warehouse hold out features. The last bastion of warehouse holdout features. Before uh we like

12:47go into what we're doing, why why I feel like you have to preamble this with why along with a little bit of context. So so so what what what are store procedures, temp tables, uh multi-statement transactions, multi-table transactions, uh SQL scripting? This is kind of the the suite of features that's the last bastion of warehouse holdout. So why

13:10didn't Databricks in invest in them early on? Well, SQL was really used as primarily a query language in Spark forever like ever since chart SQL uh launched. It was more of a way to just query the data in an analyst friendly friendly way like query side. If you wanted to manipulate data in Spark, traditionally you wouldn't do it in SQL

13:34in the early early days like in the early 2010s, you would do it with Scala. And then over time, I think in 2019 2020, uh we doubled down with the Python community to be more Pythonic in Pispark. And so over the years, actually today in Databricks, it's mostly 50/50 SQL and Python. It's like one or two

13:57percent is Scala and R and Java and the long tail of other stuff uh now that we have spark connect but like it's basically SQL or Python. And so if you wanted to do programmatic what are these things these this glass bastion of warehouse hold out stuff? It's to do ETL in a warehouse. And so like why why

14:20didn't we do these things? Because traditionally you would do the ETL not in the warehouse. you would use like an Informatica, a data stage SSIS to do your ETL. It didn't become popular until I would say like 10, 15, 20 years ago to cut out the ETL tool because Informatica started nickel and dimming all their customers and making it really expensive. Uh Microsoft shot back with

14:43SSIS which cornered the market for a little bit, but then another part of these crafty uh warehouse engineers were like, "Nah, push the compute to the warehouse. Just give me an extra, I don't know, Informatica costs a million dollars a year. Uh, give me an extra 250 grand in hardware on the warehouse. We license the warehouse software. We get unlimited use. Uh, we'll do the ETL.

15:07We'll call it ELT. You load the data raw into the warehouse. We'll process it on prem. When you own the hardware, you own the network, and you license on unlimited use of the software, you could do this. It wasn't efficient. Warehouse engines were not efficient in processing data. They were primarily built the OG use case for a warehouse was get me the data out of this thing and into my BI

15:29report as fast as possible. That was the original use case. Processing data was not necessarily something that a warehouse did. But then these crafty engineers were like, I could save so much money on these ETL tools. And business folks were happy to cut out, you know, a price gouging vendor that was they were totally happy with doing that. And so along came ELT. And then this kind of proliferation of pro

15:53programmatic extensions to SQL. Terod data had its own version. Oracle had its own version. TSQL which was SQL server had its own version. It's programmatic extension to SQL. And what you had now was two versions of lock in. The data was locked up in the proprietary warehouses format and the logic your ETL was now locked up in the warehouse

16:15specific SQL language which the market did not like because what happens if you wanted to migrate off of data stage terod data you'd have to hire a consultant multiple years millions of dollars to migrate off of them. So little bit of the history behind this. It's super contentious. There was no consistent standard. Any did not start creating standards for a

16:37lot of these concepts until well after these v vendors built their own thing. And on Databricks or spark you wouldn't really use SQL to manipulate data. You'd use a real programming language. Fast forward to today everyone has still been doing ELT. And even now you you've got it in the cloud where vendors like DBT and Snowflake actually promote you to do

16:59it. They're like, "No, don't use Arie ETL engine. Just use uh uh push it to the warehouse with DBT and Snowflake." DBT supports other warehouse. But let's get serious. It was built for Snowflake. It supports other things, but it was just natively built for Snowflake. And they bring this anti-attern into the cloud. You know, is it the right thing to do? We have a lot of customers that

17:22have the EMR plus Snowflake pattern where they would use Spark uh to pre-process their data to load it into to Snowflake. And so I was like if other C and or customers would use Databricks plus snowflake where they would ETL the data in Python your data engineers would use Python and Databricks and they would load the data into Snowflake about five years ago. That's what people did. And then Snowflake started developing

17:45data engineering capabilities. they started bolting on Python uh and then data brick started bolting on SQL to do more of these programmatic things. This is a little bit of the history. It's a mess of standards in the marketplace. We didn't for the longest time it was like well why would you want to use SQL? Just go use Python. Even today uh like our the Reynolds Zen our chief architect was

18:08like why are people manipulating data in SQL when there's Python right there? You can test your code in Python. You can't really test SQL. you can test SQL. There's a couple vendors that do it. It's not great. So, that's a little bit of the why of the what. Uh the market is fragmented. Tons of proprietary lock in everywhere. And uh traditionally, people didn't do that in Databricks. But when you cross the chasm from early

18:31adopters into the mass market, you have to start supporting these other tools. And so, we started investing in these kind of these last holdouts to do this warehouse processing uh inside of the DBSQL warehouse. So store procedures is the encapsulation of the code to be able to execute it and get a return. SQL scripting which is you know doing loops

18:53and if conditionals and and error handling and all that stuff that happens inside of a procedure. Um and then you have statement transactions multi-table multi-statement basically begin do all these actions commit or roll back you can kind of handle multi-tables that way. Um and then tempt tables is a super common way uh in old legacy systems.

19:15Generally you put temp DB on on fast SSD storage so that you would get really fast manipulations. So people just think they need to continue doing that even though it works different on the cloud. You really shouldn't do it. Uh but a lot of times we have to support these things to do like lift and shift migrations even though I don't like lift and shift migrations. I think you need to modernize your code more more or less is

19:38to backwards compatibility support these migration efforts but that's you know it's all coming I think the one that's trickiest is the SQL scripting is already out procedures is out uh you can start testing them uh statement tempt tables is coming soon we're just trying to make them a little faster and then uh uh uh statement transactions um you know

20:02that that one's weird because delta uh every DML is a transaction. In every other warehouse, it's not. You have to open a transaction, lock the system, write the transaction, un commit the transaction, unlock everything. Uh with Databricks, there's no locks. It's optimistic concurrency with snapshot isolation on the reader. So, it's just it's trickier to implement um uh in this

20:25type of system, but it gives you better consistency. You get role level concurrency with optimistic uh concurrency. That just means that like you're not going to lock tables and if you're not touching the same role, the users won't even see each other. Like they won't even care. They'll just be writing differently. So, it's going to be better than what the other vendors provide. But that's kind of the last set

20:48of what I would say the hold out warehouse features are to be able to do the these lift and shift migrations for your ETL into into Lakehouse. Mouse.