Rivery & Databricks: No-Code Data Integration for Teams

0:01one hi I'm Yousef and today uh I have a very special guest I'll let him introduce himself hey thanks youf um yeah so my name is zachi I'm a Solutions engineer here at Rivery um glad to meet you sir likewise so can you tell me more about River yeah sure thing so river is a

0:24modern data integration platform mainly focus on simplifying like the elt process uh so we really do is we help data Engineers or even data analysts extract kind of load as well as um run Transformations on their various sources so when we talk about sources like what we're talking about is like a lot of different like third-party apis like Salesforce maybe like Google ads but as

0:46well as like some databases as well like you know oracle mongod DB you know SQL server and we help them take data from those variety of sources and pull that into like Databricks or other data lakes and warehouses um and we offer like a platform to do this that's in a very like simple no code experience um that's pretty fast and easy to set up and requires very little maintenance um

1:09so we're seeing this become a lot more important as we see teams kind of investigating some you know of their data as well or even some of the modern like you know AI use cases where they're trying to get their data in like a normalized fashion within their like data warehouse or data Lake um so once the data kind of lands within Databricks Rivery can also help run Transformations um data activation or like ETL back into other systems as well

1:32and our users also are those who want to save time by avoiding you know coding you know building out those manual Integrations maintaining Integrations or managing too many tools to create like a single data pipeline um so it really gives them the ability to kind of focus more on those you know high value tasks uh that support their businesses as opposed to those you know individual maintenance tasks that seems very very interesting

1:54like simple platform to use and that's that's that's amazing and since you mentioned like data breakes so can you show me how I can connect uh River with data breaks yeah so there's actually a few different ways so I think the easiest way usually is just to kind of show you how this works um there's two options really So within uh Databricks we're

2:16actually one of the first like Databricks uh Partners to be utilized within like their partner connect um so if you have like a Databricks instance you can just go into like the partner connect on the bottom left search for Rivery and you can click on Rivery here and we'll automatically spin up account for you so we'll spin up like the rivy account and connect your datab bricks instance as well um alternatively you can actually also go into or like once you've created

2:40a Rivery account um you can create one on our website as well and we offer like a free trial uh once you create that account you can actually just click on the connections over here you can click on a new connection and you can just search up Databricks and we'll give you instructions here on the right hand side on how to configure your data BRS instance and how to pull in those different parameters needed um and you can kind of add that

3:02information here on the left hand side and you can also test that connection to make sure that it works yeah it's very very clear so either do it through partner connect or through uh your uh your platform amazing so now can you show me how Riv works I think it's very very important for our audience yeah sure thing um yeah so how

3:24about I just go into like the Rivery platform and I can show you a little bit about how it works uh give me one second while I sh my screen yep here you go um so this is what the Rivery platform looks like um there's you know a very brief kind of introduction into Rivery is that what we what we call rivers are going to be those data pipelines so we have three different River types or data pipelines

3:46and we have a source to Target a logic River as well as an action River um so what a source to Target River does is that's more of like that basic like extract and load you know so taking data from one source and then loading that into a Target um we go a step further from there into what we call a logic River and this adds another layer to that where we run orchestration of your different

4:08pipelines as well as like a transformation step and we offer transformations in both like SQL uh push down SQL that would kind of land in your datab bricks instance as well as some like python Transformations as well um and lastly we have what we call an action River and these action rivers are able to run both get and post requests to custom rest apis so a lot of our customers are using that either to pull

4:31data from you know longtail applications or like other third party Services um or they're using that to post data back out or run like a data activation um so in that post um example you can use that to connect to like um DBT you can use that to run like you know Tableau refreshes or you can even use that to call like a datab bricks job to run as well so I'll kind of show you kind of

4:55what the platform looks like and kind of hop into the different elements of it but there are kind of a few examples or you know some information I also wanted to review there's this idea of like variables within the platform as well and this is kind of like a very important function because it really makes the Rivery platform very Dynamic uh you can use these variables to parameterize certain parts of your Rivers so that you can use you know you can pass through one variable from one step to another you can dynamically

5:18change those variables as well but it also leads into like our environments function where you can then build out deployment packages uh from like a test environment to like a Dev environment to like you know production environment and you can see how those values change or those variables change from like the test demo or like a prod environment in this instance but we'll hop into like the real meat of the platform in those like Rivery queries or Rivery Rivers

5:42themselves so we'll start with the source of Target and then we'll hop into the action and lastly we'll hop into this logic River kind of show you how they all combined with each other so with the source of Target River these are all the different sources that we support so wide variety of like marketing analytics sources you know rdbm M sources no SQL CRM source as well as well as some basic storage units like S3 GCS you know Azure blob FTP um

6:08but we also have things like you know web hooks where we can receive data from a specific endpoint as well so from here all you have to do is like if you're looking for a specific connection let's say you're looking to connect Salesforce you can just select Salesforce over here and on the right hand side it'll give you instructions on how to connect you can add new connection and on the right hand side it'll give you instructions on connecting or adding your Salesforce information

6:30and you can just fill out the information on the left hand side you can also test the connection on the bottom left and you can just select your connection from the drop down over here all our various sources actually kind of follow the same format right so that was Salesforce but let's say we want to connect mongodb we can just select mongodb click on the new connection and again on the right hand side it'll give you instructions on how to connect and

6:53you can fill out the information on the left hand side so Salesforce was more like an API mongodb is going to be more of like a rdbms or like a no SQL Source uh and for those sources we actually offer a few different ways to connect we offer like a um you can connect through SSH reverse SSH Azure AWS private link as well as a VPN option as well and for all our databases we

7:16actually offer a few different options of extraction you can extract using standard extraction or using like a CDC method or in mongodb's case change streams as well so we can give this a title at the top and just call this mongodb two Databricks and we can even give it a date so February 2nd 2024 and on the target side these are

7:38all the targets that we support So if you're using Databricks you just select Databricks over here same thing if you haven't connected your Databricks connection as we kind of reviewed earlier you can actually just hit new connection and go through that same information on connecting Databricks again uh fill that information on the left hand side one thing I did want to note is this idea of like a custom file zone so by default like how loads the

8:00data is in this case we take the data from uh load it into a custom file Zone like S3 and then load it from S3 into Data breaks uh that S3 instance by default is managed by Rivery but what we let you do is that we actually let you utilize like a custom file Zone as well so you can stage your that intermediary data within your own S3 or within your own blob environment and this becomes

8:22useful for a lot of um security teams especially because like all the data can be within your VPC and stay within your VPC uh but it also helps where none of the data is like you can handle a retention so you can handle like auditability of like what that data looks like you can go back and see what that raw Json or raw CSV looks like um and you can also you know create that like data Lake layer automatically where

8:45you have like that you know reference within S3 of what that data looks like once you have that data braks instance set up you can just select the specific catalog that you want to pull from and we also have like helper functions on or you want to write to sorry and and we have helper functions as well so this would just be pulling like a different list of catalogs available um and we do also have like a few other things um here as well that

9:07make it a little easier so like once you select your catalog let's say we're using like Rivery test we it'll automatically load the schema as well um just give me one second pull up so I believe uh river is supporting Unity catalog yeah so we uh we support both Unity catalog as well as a high store as

9:30well so once we load the schema you can just type in like sample and you we offer different loading modes like upsert merge append or overwrite as well uh so upsert merge just loads on like a primary key you know overr right trunk case and loads and a pen just adds at the bottom but from here you can just select the different you know schemas or collections in this case

9:54that are available within your mongod DB instance you can just kind of look through here look at anything that's available and just select anything that you want you can either select all the different you know collections or you can select individual ones as well and you can kind of edit each one like the target table that you want to load in within um your datab bricks instance or you can select like you know the load function the specific extract method as

10:17well so a lot of different configurations here that are available I really love the simp I really really love the Simplicity and how everything is made accessible for everyone that's really amazing yeah it yeah it makes it super easy and I feel like the instructions are also there to make it really easy for you to kind of like you know without like much

10:40information like if someone's not too familiar with like but they still need to pull in data it makes it super easy for them to kind of just pull that through without like much you know you need some like guidance but Rivery does provide like a lot of the guidance or instructions for each one and then it's pretty much just like a no code experience to kind of writing all that different information through so let me actually just rename this we'll call this R underscore and

11:04on the schema side we'll just select the different tables so there's a few I wanted to kind of run through just to kind of you know I'll show you a little bit later how it all kind of ties together but let's look for the specific I want to type in like user types maybe environments rules uh we'll put in like I think that

11:26should be good uh and then from here you can just this to run at Whatever frequency that you want uh the highest frequency that Rivery supports is every 5 minutes and then the lowest frequency is going to be once a year but obviously you can also just schedule this ad hoc to run as well and then this will just schedule or just run whenever you kind of run this process you can also get notifications on failures or Warnings or if it exceeds a run threshold as

11:50well and then once you hit run this is just automatically going to run this is going to take data from mongodb right into our Databricks instance and you can see this on the activities as well where I'll pull in that specific run you'll be able to see those metrics over here and you can see a broader activities panel as well that pulls in like all metrics from various you know all your different pipelines as well then you can filter on the specific

12:12date time that you'd like as well and you can just see like that process like these are in the running status and these are the ones that I've run previously with they completed time we also have other things where we can pull in like you know email data as well so if you have want to look at the specific River you can just look at it over here and this is one that I've pre-created that's pulling you know email like an Excel sheet from email so this is just

12:35pulling from my own email and it's just pulling this Excel sheet that we can then see here and same thing we're just loading that directly into Data brakes as well that's really any questions or anything so far youa no to be honest everything is clear and thank you for those explanations I mean it's very simple and your explanations are very concise so I don't

12:59think I have I have but yeah I have a question actually so in your experience when it's the best to combine rivy and data breaks yeah so I would say in my experience um like the best way to there's kind of like three different use cases I would say where like data teams benefit the most from like combining Rivery and Databricks the first is you

13:21know for like f data ingestion for like databases you know various Cloud systems and maybe some like custom apis as well um I didn't show much about the custom API connections but essentially like you're able to pull from like a variety of different sources using our rest API connector um and this is something I can show as well if you'd like uh but really it can kind of just pull data from any like custom rest API and that's kind of

13:44done through what we call like this action Reverb here um this is especially common for like marketing analytics use cases as well uh where many marketing kind of related data sources such as like ad channels social media and other marketing tools are kind of integrated within Databricks um and then they also have like those longtail applications that they may pull from using like that custom

14:06connector um kind of the Second Use case so that's kind of like the first case I'd say the second one um is I would this I would say like large organ organizations that need to manage like data pipelines across like multiple teams so instead of having like a bottleneck where like there's one centralized data team that kind of creates all the different pipelines for the organization um they share like a segmented environment across like

14:29different teams so like the data team is going to own like one environment but using like this environment functionality they can actually create multiple different environments where all the different teams that would need it so that way they can like parameterize or like you know organize like you know one Department could have access to what their own environment and the data team can kind of govern that and control over access uh between all the different you know pipelines that they have but the centralized team would

14:52kind of own that and then each department would have their own like sandbox essentially to play with um I would say that's kind of like the second most popular like use case um and really enables like data management approach especially in like a data mesh way as well uh so the third popular use case probably is for organizations that want to find themselves managing like too many tools and then wasting too much time as a cost of that um so you know

15:17usually some people are using a different tool for ingestion as well as like orchestration and another tool for transformation and then another one for like data activation rather than having like four different tools for that they can kind of just have Rivery be used for like that entire scope or end to-end process um so that's where we've really seen where like that uh River has been beneficial especially in like that tool consolidation so hopefully like those three different use cases uh can kind of

15:40help you know explain like the best use cases where you can combine like Rivery and data break sensor yeah that would that would make sense and I believe I should thank you for your time and I'll make sure to add all the useful links in the description of the video including the links to Rivery the documentation

16:03and in case you want to reach out the team thank you thank you Zary no problem than you sir bye bye