Databricks Notebooks: AI Autocomplete & Live Debugger

0:00hi I'm youf and today we have a guest it's Jason Maser is his second video with us and today he'll be introducing some exciting features on notebooks hi Jason hey yoseph yeah thanks for having me yeah nice to meet everyone my name is Jason and I'm a

0:22product manager here at datab bricks on the notebooks team and yeah we have a lot of cool stuff I'm excited to show you today yeah yeah so I've been using those features for couple weeks and to be honest they did massively massively improve my productivity and my go was like I need Jason back to show this cool

0:45features awesome well let's uh let's let's dive in let me share my screen here and uh you know I I thought maybe we could do some slides but honestly The Notebook is a great presentation tool and so you know we can just jump right into the notebook and in a live demo of what we have here so just as some back history The Notebook is one of the uh one of the oldest products at data

1:07bricks we've had it kind of from the beginning and uh you know recently we've been putting a lot of effort into making it more simple and more powerful um to be honest you know before it was a little bit intimidating for some of our users there's a lot going on and so we kind of redid the entire UI to make it just simple and focus on the things that matter most to users so like the first thing I I'll jump in and talk about here

1:29is the new cell design and the new result table so if you can see here we have uh the new cell design and it's a lot more minimal it focuses on just exactly what you want to do so we have a nice single click to run button here which is cool we now have sell titles just directly right here and uh they have an AI title button so I can click generate uh via Ai

1:51and then I I get a nice thing um you know the cool thing about this is in order to help improve notebook navigation and organization these titles also show up in the table of contents so I can see right here I have my my modern intuitive ux and then I have the subtitle of my my cell there so um it's really easy to kind of stay organized now if you use AI to autogenerate your titles or you just give them titles

2:14right there we also have a new Focus mode so I talked about this in my my last video but we've been making a lot of uh improvements to this so if you want to just focus on one output in cell at a time you can just go into Focus mode here and this is great for viewing a full screen um output table you know if you want to use the whole width of the screen um and if you really just want to focus in on what you're doing in

2:37a single cell so I know for me I get distracted by you know everything going on sometimes and so this is helpful to kind of just jump into this this Focus mode the uh the next thing that um I I can chat about with the new Celli is um it's really easy to now kind of rearrange your notebook so if I just click on this drag handle down here um it just snaps up and I can just drag it

2:59to wherever I want throughout um the page which is really cool and really convenient for for kind of organizing your notebook and with that we've also have an improved markdown rendering so if I just click in here I don't know about you but I always have a hard time um knowing what my markdown will look like after I'm done with it and so now we have this this show preview button which will show me exactly what it looks

3:21like right now and then instead of having to memorize what different commands are I can you know I can just click this button to make it a a it or if I want to you know make it a heading I can I can click that if I want to make words bold I can just click the Bold button or if I want to italicize I can I can do that and so um you know really easy now to just

3:44document your your notebook and write markdown and see exactly what's happening as you're working so excited to to bring that to everyone too I was struggling with this one because each time I needed to know write H1 then write this and then exit and click on enter to have the result and say okay no it's too big I need to make it smaller and this is really solving this this pain so it's

4:07amazing yes exactly I I'm I'm the same way as as you ysph so cool so that's kind of the new Celli and now if I talk about the new result table this is the next piece that we've really kind of worked from the ground up to rebuild um it's now much more performant Scrolls really really quickly uh it has infinite scrolling which is great and has a lot

4:29of new power ful nice little features so um you know I have like things I can hover over the tool tip to kind of see um you know what uh what data type it is I have data type tool tips so I can quickly glance at all my data and see kind of what it is at a high level and the one thing I'm really excited about is um our new filtering capabilities so it's really easy to add filters now if

4:51if you um if you don't want to write kind of more advanced SQL or or if you just want to stick with just super basic seq just select star you can do that and then if I want to go in here and say I want to see only countries um in the continent of Asia I can just right click and say filter by this value and which which I then have all the values in the continent of Asia and then if I actually want to exclude certain values

5:14you know I can just right click here and say exclude this value and so it's really easy to kind of um chain these these different filters together to really do Advanced um kind of data exploration without having to write any code which is awesome if I go down here I could show you something that's cool too with this is the filtering mechanics are different depending on the data type so right here it's just a um this is a string fi and

5:38if I filter I get the options to like you know contains is one of um starts with ends with you know all the kind of the string stuff but if I go over here and find a date time and I do a filter here I get different options so I get like after before between and then if I click on uh this I get an actual date calendar where I can then go and select

6:00the date that I want this I want to find dates after this so it's really convenient really easy to be able to filter through different types of data and so yeah we're really excited about the new result table and just you know all the ability that it gives you to kind of jump in and explore your data without writing much code I I have to I will mention also we have improved selection mechanics so you know if you really want to if you want

6:22to copy things um you know you can really easily just um copy different columns and rows and uh it's just it's really easy to work with that so hopefully you all will enjoy uh the improvements to the new result table I did so I guess they would enjoy it yes yes awesome perfect okay now the next piece of the Notebook that we've been working a lot on is embedding

6:45Ai and intelligence in it to um you know just make it more simple and more powerful so the first thing that I'm excited to talk about is Databricks assistant autocomplete and this is something we released just a few weeks ago and I'm personally really excited about it so as you can see here I have a a SQL query or a SQL comment here that just says I want to query countries with a population of above 10 million in a

7:07life effect expectancy of above 77 years so now if I just um hit enter here and I'll just type you know I want to do a select here and you can see I get um I get a recommendation for exactly how to write this query so there you go so just uh just did it for me so again really awesome and it's it's powerful if you just want to write a comment and tell

7:29and it will just just tell you how to do it or um if you're writing complex code it can give you recommendations um based on the context of what you're doing so super excited about that one the next thing uh next we're Bing embedding intelligence is with um the SQL language switcher so this is really nice because so there's a python notebook and you know if I want to write SQL here I have to type percent SQL but now we have an

7:52auto language switcher so if I just type select star from um you can see it just automatically added the percent SQL and um now I can just type my code so this has saved me personally a ton of time and hopefully it will save you time I don't know y do does this happen to you yeah Lon trust me those tiny details make the difference I TR I can feel your

8:16pain yes yes well yeah that's awesome glad glad it's helped you and yeah definitely something I do on like a daily basis at this point um so this has been a been a super good timesaver for me um so one of the things we can do in which is really awesome in the in the notebook as you can work between SQL and python so if right here I'm just going to convert I'm going to take the output SQL DF and I'm going to convert it to a

8:38pandis data frame so we can use it later um for some of our other demo stuff but I just wanted to run that um another thing that we uh are releasing very soon that I'm very excited about uh which embeds intelligence is what we're calling text to filter so this is another data set I have here so this is the Titanic data

9:00set so it just has information on people who are on the Titanic and so I showed you how you could add filters um via the UI here but you know why why add them that way why not just tell it exactly what I want to filter with natural language so if I click filter display data here I get a prompt here it says what do you want to do so you know I'm interested in um only males over 70 on

9:23the Titanic don't know how many people there wear that so if I just uh type that in it automatically creates the filters that I want and gives me only males over the age of 70 and again if I wanted to dig in more I could I could type another National prompt and just kind of chain these to really quickly explore my data so this is a super cool uh feature that we're excited to bring very soon to you

9:46to you all yeah another way to improve the data exploration exactly exactly um i' I've shown this one before but I can't I can't give a demo without showing the inline assistant and now it's updated with a cool new icon here so you can see uh you have a a little animation which is cool there so if I click that um again this is a super easy way just using natural language so say um or say

10:09let's say find the top 10 songs based on number of streams and if I just hit enter there boom just wrote me the SQL to do that and if I run it it runs correctly and gives me um the top 10 songs from the Spotify playlist for last year um based on number of streams so again

10:31super awesome way to just explore your data in a very simple way um the cool thing about the assistant is uh it could also do it in in Python too so if I just say like you know just copy um this again and say you know find the top 10 strings using the Spotify DF now this time um it it can do that now um so I

10:56think it actually switch back into spark we'll see if uh I'll see if this works um but there you go so yeah so super easy to kind of uh do whatever you want with the assistance very powerful um and yeah great way to just use natural language to explore your data so again really making trying to make things simple here and and just to to highlight so for those who are uh wondering how uh the AI is able to

11:20find the right table so keep in mind that we you see now we have the possibility to generate automated uh comments uh over uh table and columns and that's what helps you when you're writing you're using AI within the notebook to find the right table yes yeah exactly um so yeah it we

11:42the AI has access to Yeah the more you document your your data and tables the better your AI recommendations are going to be so uh so yeah we definitely encourage you to use UC and to put comments and kind of document what's going on it will help um both your team and the AI to give you better recommendations exactly so finally the third pillar

12:04of ways we're making the notebook more simple and more powerful is by bringing powerful IDE features um select IDE features into the notebook so the first one that I want to call it is just like syntax error highlighting and quick fixing so um you can see here like this is a this is valid but um you know I haven't imported um numpy and so if I just hover over this you know I can see undefined name and I can just click

12:27Quick Fix here and I could do try to do it with the assistant which would do it or I could just automatically do it with um the language service Proto server protocol and just say import NPI as MP and then it it fixes that um so again we're bringing in ID features that that give you kind of that syntax highing to tell you when your stuff is wrong and then a quick way to fix them and then this next one is one I I'm really excited about um I know um a lot of

12:51other people are excited about this too and that is the integrated debugger so if I run this code um you know this is supposed to to plot a graph of the different valences for the different streaming platforms so Spotify apple and dieser and if you can see here I it's it's giving me an air uh NP arrays this is kind of a confusing air not quite

13:13sure what's going on here and so we now have an integrated debugger to kind of figure out what's going on so if I just click right here and I say debug sell um it will take me into um a debug mode mode where now I can do um step by step and kind of go through each line so um if I go through

13:35here um you know so let's let's look at um the values I'm getting so if I step over Spotify average veence looks looks right to me and if I go Apple average veilance that kind of looks like a good thing to me and now these are average veilance I can see oh oh my gosh look here I have um just numbers I forgot to call the dot mean um function here

13:58and so uh can just easily see that that was the error um and then I can just stop and rerun the uh the codell and see if that fixed my air for me and it looked like it did so again just really powerful way and uh I know personally this is this has stopped me from using vs code I'm able to use uh data Brooks

14:21and just kind of step by step and do big data debugging which is uh which is super helpful I couldn't agree more with you so each some you're bringing new stuff so I can't wait to invite you a third time just to see what you GNA bring yes no well yeah thank you so much for having me and and yeah hopefully everyone uh everyone enjoys the uh the

14:44the improvements uh we're going to be at dat and Summit so definitely if you're there stop by we'll have a session where we'll share even more we stuff um and and yeah thank you Jason and make sure to register to attend the data as either in person or you can still register for the webinar thank you