Rivery ELT: 200+ No-Code Connectors for Databricks

0:00a historical bottleneck of a very centralized data team data engineering team where essentially business teams need to uh wait for data to be available what people should be doing with their kind of data architecture they do basically everything from extract [Music] transform hey today I am with Taylor

0:25from Rivery and we have an amazing session for you can you introduce yourself um what you do at rivy sure yeah hi Yousef I'm happy to be here um my name is Taylor McGrath and I lead Solutions engineering at rivy um I started in the data analytics Consulting space about 10 years ago uh managing data projects and Advising on Data

0:49Solutions to a bunch of different types of companies and organizations and I moved to Rivery about 5 years ago uh to lead pre-sales and uh my team's role is really to bridge the gap between you know our customers business problems and then the Technical Solutions they they can find with Rivery and and our partners that's very interesting and can you tell me more about uh

1:13Rivery yeah absolutely so Rivery is uh a modern integration platform so we're really in the elt space so extract load transform and it's all about uh building and maintaining data pipelines to essentially move data and then transform data you know once it's there and orchestrating these processes Al together as they go it's uh it's no code

1:37so it helps uh data engineers and data analysts alike to extract and load data from various sources such as you know third-party services like like Salesforce your marketing platforms like you know Facebook ads Google ads as well as uh databases to do database replications from you know operational databases into your your cloud orous or lake house like like Databricks so you

1:59know it offers a simple no code experience it's fast to set up and requires you know very little maintenance because we are a we're a SAS platform so there's nothing to you know install on premise uh let's say we also do have self-service an easy way to create custom connections so that can be a custom source to any rest API as well as even a custom destination so for like

2:22reverse ETL let's say if I want to take data from my lak house and put that back into an operational system um yeah that's amazing I think I remember I used Rivery to uh get data from from C force and then I like I I did some Transformations then I push it back to Genie to start asking like natural

2:46language start asking questions like what are my weekly sales on this project what are the top best teams and it's very easy to use Rivery that's that's for sure and that's that's I think that's that's also if you can give us some example where data brakes customers use Rivery and maybe some of the common

3:08use cases I already mentioned that like ingestion from Salesforce but do you have any other examples yeah yeah so we really say two main use cases I think go really well where Rivery and Databricks fit really well together the first one is a little bit of what I explained before which is what I'll call you know simple and flexible data ingestion so that means being able to move data from essentially any Source

3:31into day Brick's lakeh house and by flexible I mean uh even even post ingestion being able to for example kick off a Databricks uh workflow or kick off a DBT job to run on top of Databricks or just do any sort of SQL that's happening within within Databricks so whatever that next step is of automation

3:53of your like data engineering life cycle being able to connect connect the dots more more easily uh and flexibly with all the configuration options while still being easy to use the second core use case that we're seeing is for larger organizations there is this shift or this move to be more decentralized in

4:17their in their data operations meaning there may have been a historical bottleneck of a very centralized data team data engineering team where essentially business teams need to wait for data to be available with Rivery we have a multi-tenancy that we can actually align really well to Databricks Unity catalog so that means data

4:41platform administrators can create these like curated silos where business teams actually can go deeper Upstream into the data engineering life cycle and create like endtoend data products without having to wait or depend on you know Central it organization because it's easy to use and because it can be governed right to the per permissions

5:03and there's visibility over all the tenants these these are very very great use cases but I'm really like I want to know specially or I'm interested in the ones where rivy and data braks are used together to decentralize the development of data pipelines and as you mentioned like maintain this proper governance can

5:26you give us more details about that or maybe to have a customer story yeah absolutely so we have a customer story I can give I I I can't say the name but it's a it's a very large uh Toy Company I'll say and the core use case with them is very aligned with the second example I gave you where essentially they're looking to

5:48decentralize kind of along the Paradigm of like a data mesh principles of their you know all their data operations internally their core Warehouse Lake housee everything they run on is Databricks internally so essentially what they were looking for is can we have an endtoend platform where we can we can Leverage The Power of Databricks but

6:10also connect to any data source easily so that these teams uh they call them data product teams can go end to end right without without needing um you know there their Central it organization so what we did there was we aligned our environments in Rivery to the different teams where each team essentially has you know like a Dev QA production and that aligns to the

6:35access that they have in data braks as well so the provisioning process is all linked where when they link or when they provision a new team they provision them the necessary Databricks access and they connect that to the place in Rivery where they're provisioned so that those users have full control over the part uh the piece of Databricks that they should right so they can go end to end

6:58but it still governs at a global level uh internally as well first this is um a more general joint architecture both Rivery and Databricks so if we see on the left where we see ingestion these are kind of buckets of all the different things that we can connect to easily so we have fully managed connectors the ability to create custom ones something called kits which

7:21we'll get into in the demonstration which is essentially these pre-built models of both the ingestion and the push down transformations in Databricks change data capture or database replication as well as event streaming and then the idea is to move the data into Databricks usually building some sort of data modeling what's what's common is what's called you know maybe a medallion

7:43architecture where you're building out the different layers of your Lakehouse of like raw data to more structured and curated data for specific business needs so that's when we start talking about data products and something that'll actually you know provide value to the business and you may do this in a number of ways whether that's DBT native Databricks orchestration and Transformations uh where is flexible you know with with

8:06all of those and then ultimately uh there's some sort of activation or value insights being had whether that's analytics in the form of you know our traditional like dashboards and Reporting using that data for you know data science ml even gen now data breaks has like a great Suite of features there and we talked a little bit about

8:28sending data back to business apps as well so being able to push it even Downstream you know outside of the Lakehouse and back into you know an operational system so it's the general architecture of what in the use case I was talking about they want individual teams to be able to do and to end kind of any any piece of this puzzle or even to be able to draw the line in the

8:50middle maybe you want um your core engineering team to still service this side of things but then once once you get the data into Databricks that's when you want people to you know other teams to be able to connect do their own data modeling and then serve their lines of business and so when we start to think about data operating models I've talked

9:12about you know decentralized that's really the core kind of shift that we see happening especially at larger organizations where the bottlenecks are much more painful because you have to you know maybe wait go through a few different teams there is a laundry list of requests to get access to data even U we're seeing a shift to the right of

9:36where these data platform teams are established to govern and manage the data platform and to enable Downstream teams to be able to selfs serve all the data on the platform so it becomes the ideal state is these like democratized operating models where we have the centralized data platform which would be for storage and and

9:58transforming data that sits in Databricks would be Databricks but then also making sure that all the other tooling that we're integrating with Databricks has the same security the same governance principles Etc that we can serve these domain teams so all these things are standard while they're doing their own data product building separately so

10:21that's that's the idea and that's exactly what the use case I was referring to or or the customer store story is here where you know they really started with their like marketing domains so all the marketing platforms being able to Ser get the data from The Source transform it in Databricks and then ultimately serve it back to the

10:46business yeah it's amazing how how easy it is I think this is the most important part how easy is to use uh Rivery since you're already sharing like your screen can you show us how Rivery works like can you open the platform and walk us through through it yeah absolutely where I'll start is actually in the Databricks partner

11:09connect so river is in datab bricks partner connect it's super easy to just spin up spin up an account so you basically just page through this menu it'll create you a Rivery free trial account and it'll actually automatically create a Databricks Connection in our account as well so you can just get started immediately setting up your sources uh

11:32which I'll show you here so if we jump over to Rivery here we see the Rivery console we see the Rivery console and this is initial landing page of Rivery so kind of all the different types of data pipelines you can work with as far as pipelines we referred to them as you

11:54know rivers to be very on brand here but let's say I wanted to connect some different applications to my Databricks and you know set up automation to ingest from those sources it would just be a matter of clicking to open this type of pipeline we call a source Target so getting data from sources Landing them in our our Target which is datab bricks

12:16and we have over 200 different connectors out of the box and the ability to create your own custom connectors so just to start with something quick um I'll show you an API one so if we if we choose Facebook ads so let's say you know a very common marketing platform I would choose the source that I want to connect to and then establish the connection to that

12:39source and I can always create it from scratch if I don't have any connections yet and then really there's two routes I can take one is I choose predefined reports and I essentially can page through this menu so I select uh Databricks as my target I can set up the connection if I already have one or I can always uh go in edit it or create a

13:03new one so I'll set up all the different pieces here uh and then I would choose okay what catalog in unity catalog do I want to load data into my schema Etc and how I want to actually load data once it's starting to merge into other tables let's say so do I want to set up a merge

13:25process just a pen data or always overwrite it let's say in the downstream tables and then choose the actual reports that I want to pull from Facebook so that's as simple as just saying okay I want to pull everything or pull specific reports I can go into each one see what the actual columns will be I can edit and choose you know customize

13:47any of this in terms of casting renaming you know Etc you name it and then ultimately just put that on a schedule to automate so you know hourly daily even custom cron expressions for like Advanced scheduling and really that's end to end of just if I want to get data very quickly into like the standard

14:09reports and common use cases of these sources I can use this predefined option now when we're thinking about being highly configurable and flexible uh we also offer custom reports so that means I actually can curate and Define the exact Fields the dimensions granularity uh that want from these sources so for

14:31Facebook and many other uh apis they they tend to be aggregated so based on the columns I'm selecting that's going to be the level of granularity of the output let's say so I can say all right I just want to see my accounts by uh clicks so I can select these two Fields choose uh choose the data level choose

14:53any other additional breakdowns this is all we try to make our sources as oneto one with with what's available in the API documentation uh by the source apis as we can so there's kind of this this two route option of where I can get very custom I can essentially pull anything that these apis offer um in a highly

15:16configurable way or if I'm just a time to Value speed I just want to replicate um all the common reports and tables then I can do that as well so this is when we're talking about flexible ingestion something that's simple yet highly configurable this is this is what we mean yeah so technically to technique if I understand well like maybe a data

15:39analyst can just open the platform select any type of resource like I don't know Google analytics Facebook or whatever like choose the catalog the schema it's going to up search merch or overwrite send the data to the data Lake and then from data brakes go I don't know do transformation then send it to pobi or publish reports quickly

16:04without the need of like any data engineer typically yeah exactly that's amazing this what I just showed is like that first the first step you mentioned right so just getting data there uh and it's very similar like for you know applications like a Salesforce or even a database it's it's the same exact flow what I just showed except I'd be choosing entities or tables let's say if it's a database instead of API

16:27reports but then once I do that I can actually uh choose what happens next Downstream so our next type of pipeline we call a logic which is essentially a series of logical steps so I can say okay my very first step I want to run an existing River so maybe the Facebook the Facebook one we just

16:49created and let's say I also have you know a Salesforce process I've I've built so I'll do you know Salesforce load and these I can actually even kick off together so I can say I'll create a container I'll drag this one in run these in parallel that means these two jobs are going to be triggered at the

17:11same time kick off all my ingestion together and then I can choose I have a few options for how I want to kickoff transformations to happen Downstream so we do have a built-in like Databricks functionality for just running data brick SQL so this will happen you know in the actual uh command is sent to datab Bricks it's using Databricks resources so here I can just do you know

17:34really any SQL query or ddl typescript run that in Databricks and then choose what happens to my results so maybe I want to do some sort of transformation load that data into a different table in Databricks I can do that with this step I also can kick off we call an action which is our kind of generic rest API component to do a number of things

17:57so this could kick off a DBT job this could kick off a datab bricks workflow that's already defined and we have a lot of these use cases because we see them a lot with our customer base we have them already preconfigured and we build them into what's called a kit that really any user can go and download and start using quickly so here let's say I want to actually kick off a Databricks workflow

18:21I have a job here called datab bricks job trigger where I may just need to send in or configure some variables like my job ID making sure you know I'm I'm configuring and triggering exactly what I want to run but this essentially will connect the dots between my ingestion platform and all my ingestion processes and that next step of the data engineering life cycle and so if we go

18:44over to kits here I'll show you what these look like essentially they are a library of these predefined packages so if we just search all the ones we have with Databricks here you'll see I have Databricks job trigger so that's exactly what I showed before if I preview this I can click use kit and essentially it would download or import

19:06those pre-configured pipeline components into my Rivery account so I can just get started and use them so I don't have to you know go and create them from scratch Etc and so here as well you can see we have a number of other kits so we have some of them that are more like models I'll say where we do both we package both the ingestion and the

19:29the sequel or the Transformations that happen after the fact to create more consumable tables for business users so it's essentially tying together both the initial loads of the raw data and creating essentially a you know a full data model here and so it'll give you some info on exactly what's happening and we're creating kind of a final

19:51performance table this example is just with you know Snapchat data to data breaks but essentially a user can import that get all that preconfigured and then also tweak it and customize it you know to their needs so just one question kids is it the equivalent of workflow for the orchestration for Rivery or it just like some pre-bundled

20:15scenarios yeah so I'd say there I would consider them like starter templates right so for various different use cases so some will combine ingestion and transformation some like the datab bricks job trigger these are actually just more of an automation component so they're various different Rivery components packaged together easily configurable for users to just get started with so they they really range

20:38kind of all the all the functionality okay that that's cool and I saw that you can trigger workflow from from the process so can you for example build up on a successful completion of that job trigger another process for example I don't know getting data from Salesforce

21:00will trigger I don't know pushing back to another system and then getting back data from another tool for example yeah yeah so that's exactly what I was showing where these jobs actually are sequential so the next one is not going to run until the previous one completes so let's say you know if I do have my Salesforce load that's loading to Databricks and then then I kick off

21:25you know a Databricks trigger where maybe this is doing Transformations inside Databricks then after this completes only after this completes I can have other things happen Downstream too so you can manage kind of a full end to-end orchestration here um or it can simply just be hey I want to do my ingestion here and then I'm going to kick it off to Databricks and

21:47then the rest of that let's say is managed by Databricks so it really depends on the use case in the company kind of where they want their bird's eye view to sit as far as like seeing all the pieces of a of a workflow complete okay and and I'm curious to know like who are uh the main users of rivy because for data breaks we can we have

22:09lot of several personas we have data analyst data scientists data Engineers but who are like the main users of rivy I'd say it's it's everything from your your you know more technical data engineer uh a little bit more Downstream to like your analytics engineer and even your business analyst are the core user users of Rivery um and that's because

22:31for even the more technical data Engineers it makes a lot easier and quicker um their their job right so they don't have to spend as much time managing pipelines can clear up um their time to do um you know more more challenging tasks maybe more um you know bespoke tasks that are you know to optimize for their business whereas um

22:53for business analysts Etc it really just extends what they can do in terms of going more upstream and creating their own data products not needing to wait on other teams so I think what's nice about that fit with datab bricks is it's really extending the amount of users that can work with the datab bricks platform right traditionally Databricks

23:15is was initially built very much data Engineers machine learning Etc but now with um things like you guys providing data briak SQL and unity catalog and the dashboarding suite Etc there's there's kind of this opportunity for it to be further Downstream in the hands of the business and I think uh it's a powerful pairing of rivy Plus Databricks for

23:38that yeah that the that that's very interesting and just for information I think Rivery war was the first company from partner connect I've ever used when I joined the company like few few few years ago and um I've seen like several uh companies with in the same let's say Market

24:01etls what's the main different differentiator for Rivery versus their uh competitors yeah good question I think uh you're right there's there's many ETL elt vendors it's super saturated Market I would say so we I'll break it up into the two I guess categories or types that we see so first we have what I'd call

24:25more like Legacy ETL vendors where it's they're very robust products they do basically everything from extract transform Etc but a lot of them were built before really things were built on the cloud and for the cloud initially uh so the main difference between uh a platform like River and these is that everything comes with a modern Cloud native

24:48solution the ease of use the time to Value the near zero you know Administration uh so it comes down to a total cost of ownership since there's really no Hardware or like version upgrades to manage Etc a lot of that is just kind of taken away shipped away from from the users responsibilities and managed by by us the second category

25:11I'll say is we have a set of more like modern vendors often referred to as elt tools similar to Rivery they offer a number of data Integrations as well out of the box but however most of the those vendors are quite limited even they're like either they're Limited in just they only do

25:34ingestion so that's kind of where it stops there's no flexibility really to enable things to happen Downstream to connect into other other tooling via you via either you know like our examples of pushing pushing Downstream to DBT sending data to other systems triggering uh Databricks workflow and really anything you can do with with a rest API

25:56you can you can automate with Rivery in addition to you know its transformation and orchestration capabilities so Rivery can help you simplify your stack and run it alone with something like Databricks or add specific Tools in addition but not necessarily needing a series of like four or five additional tools to your

26:20lake house right for one for ingestion one for transformation one for orchestration one for reverse ETL it can really kind of simplify that stack and that also you know there's a big cost component of that as well with with every new tool there's a new pricing model there's new cost to to keep an eye on some are consumption based some are not so I think it's it's

26:43the simplification of the data stack in in times like today as well do you have any tool to track the cost within Rivery like for every single part how to track the current cost or not yeah I think I'm still sharing my screen I am so we do have what we call uh so there's a couple ways to do this

27:06in Riv so one is right in the user interface we call the activities we're essentially down to every execution you can see the amount of data moved you can see uh here let's go into one you can see the amount of data moved you can see the um the credits what we call rpu or R pricing unit that were incurred so all of these are particular runs so I can see okay if I go in to one of these I

27:30can see the status so this completed uh I can see there's one credit for that run how much data was moved the parameters and all of this is available via API as well so that means I can grab all this data I can automate that process as well we actually have a kit specifically for tracking um Rivery activities that you can bring the data these data sets down into like your data

27:53bricks tables and then even create a dashboard on the top let's say to manage alerts we have some built-in alerts too so based on on the package that you have will even do you know popups in the UI for you know when you're reaching certain thresholds or certain limits that that's amazing and I can see there is something called environments can you tell us more about this yeah so

28:18this is exactly what I was talking about with the example before where there's there's I'll say two really main value propositions here around our environments and multi-tenancy so we're talking about a use case of you know democratizing you know data operations so I'll say first what you're seeing here is you know I'm in an environment

28:40right now called development and in an account called tailor demo so this account actually has you know seven odd environments and these can be configured however I like so that means I could do my classic Dev QA production and even do repeat that for different teams so here you see like business unit one Dev business unit one prod same with business unit 2 so depending on the

29:04the lay of the land of my company and my data organization I can curate these tenants and then I actually can once something is in an environment that means uh permissioning and role-based access control can happen there and that also means that I can set up deployments deployment packages from one area to another so I can migrate content pip

29:28lines Etc from one place to another so what that looks like is I see my deployments here I can just create a deployment package where I say I want to go from you say deploy uh I'll go from you know maybe bu1 Dev to bu1 prod select objects there so I select exactly what I want to

29:51deploy and then continue to that deployment and essentially what that will do is migrate that content deploy it from one place one environment to another and what's nice is I can manage these here you see all a bunch of variables down the down the side and across the top we have all my environment names so that means I can parameterize any piece of

30:14these pipelines so that when I'm deploying from one environment to another uh I can have it uh evaluate to different values so that means if I say okay maybe I have a catalog in my in my development environment that I always want to use my development catalog but when I deployed a production I want it to the entire configuration of my pipeline I want to be identical

30:37except I want to parameterize certain pieces of it so that's where these variables can come in and I can get a a bird's eye view or Matrix of all of my environment variable definitions and lastly is when it comes to governance of actual access so I can have access at the user level teams of user level um and I can decide

30:59essentially for each user or team what environments they have access to and what level of access so here you see a a user Nam named Ron and he has no access to a particular set of environments that means when he logs in he won't even see these and then uh different levels or different roles across other environments as well so this is when we're talking about a master a data platform team who's administering this

31:22platform they can align these environments to their setup in Databricks or Unity catalog and make sure that the roles make sense for what you know a data Steward should be doing what a a developer should be doing so that aligns with their data strategy everyone's Drawing the Line somewhere between what's centralized upstream and maybe what's more

31:45decentralized Downstream that they want business teams to own so this allows flexibility for them to curate exactly where those lines are I think Simplicity is the keyword here when it comes to using Rivery yeah yeah for sure yeah simple yet yet flexible that's

32:08that's what we try to that's our that's our philosophy yeah so thank you thank you so much Taylor for this work through the platform I think you've covered almost everything like how you can get started with Databricks and Rivery how to create a river how to have a look at the kits the

32:30different environments and we have a glimpse look at the all the connectors more than 200 connectors you do have plus the custom one you can build I I think now like the audience have a clear understanding of what Rivery what can Rivery does and how data breaks and Rivery can be combined together thank

32:53you so much for your time and before we close do you have any any advice for for the users yeah I mean I think um you know we always come up against questions of you know differentiation or even just questions of what people should be doing with their kind of data architecture um they might already you know a lot of times they already have something in place like like a Databricks where they say hey this is our

33:15this is our strategic you know storage and what we're using to serve the business but then as far as all the other pieces of architecture I'd say yeah like you said just you know keep it simple and unless it's something that's going to give a very customized opportunity to your business that you want to develop manually and customly I'd say you know keep keep it

33:37simple where you can try to uh consolidate and and allow your users to you know access data and and play with data and uh ultimately serve serve the business with insights that's perfect thank you so much uh Taylor for joining me during this session and see you soon yeah thank

34:01you you you too