Rivery
Modern Data Engineering
one hi I'm Yousef and today uh I have a very special guest I'll let him introduce himself hey thanks youf um yeah so my name is zachi I'm a Solutions engineer here at Rivery um glad to meet you
sir likewise so can you tell me more about River yeah sure thing so river is a modern data integration platform mainly focus on simplifying like the elt process uh so we really do is we help
data Engineers or even data analysts extract kind of load as well as um run Transformations on their various sources so when we talk about sources like what we're talking about is like a lot of
different like third-party apis like Salesforce maybe like Google ads but as well as like some databases as well like you know oracle mongod DB you know SQL server and we help them take data from
those variety of sources and pull that into like data bricks or other data lakes and warehouses um and we offer like a platform to do this that's in a very like simple no code experience um
that's pretty fast and easy to set up and requires very little maintenance um so we're seeing this become a lot more important as we see teams kind of investigating some you know of their
data as well or even some of the modern like you know AI use cases where they're trying to get their data in like a normalized fashion within their like data warehouse or data Lake um so once
the data kind of lands within data bricks Rivery can also help run Transformations um data activation or like ETL back into other systems as well and our users also are those who want to
save time by avoiding you know coding you know building out those manual Integrations maintaining Integrations or managing too many tools to create like a single data pipeline um so it really
gives them the ability to kind of focus more on those you know high value tasks uh that support their businesses as opposed to those you know individual maintenance tasks that seems very very interesting
like simple platform to use and that's that's that's amazing and since you mentioned like data breakes so can you show me how I can connect uh River with data breaks yeah so there's actually a few
different ways so I think the easiest way usually is just to kind of show you how this works um there's two options really So within uh data bricks we're actually one of the first like data
bricks uh Partners to be utilized within like their partner connect um so if you have like a data bricks instance you can just go into like the partner connect on the bottom left search for Rivery and
you can click on Rivery here and we'll automatically spin up account for you so we'll spin up like the rivy account and connect your datab bricks instance as well um alternatively you can actually
also go into or like once you've created a Rivery account um you can create one on our website as well and we offer like a free trial uh once you create that account you can actually just click on
the connections over here you can click on a new connection and you can just search up data bricks and we'll give you instructions here on the right hand side on how to configure your data BRS instance and how
to pull in those different parameters needed um and you can kind of add that information here on the left hand side and you can also test that connection to make sure that it works yeah it's very very clear so
either do it through partner connect or through uh your uh your platform amazing so now can you show me how Riv works I think it's very very important for our audience yeah sure thing um yeah so how
about I just go into like the Rivery platform and I can show you a little bit about how it works uh give me one second while I sh my screen yep here you go um so this is what the Rivery platform looks like um
there's you know a very brief kind of introduction into Rivery is that what we what we call rivers are going to be those data pipelines so we have three different River types or data pipelines
and we have a source to Target a logic River as well as an action River um so what a source to Target River does is that's more of like that basic like extract and load you know so taking data
from one source and then loading that into a Target um we go a step further from there into what we call a logic River and this adds another layer to that where we run orchestration of your different
pipelines as well as like a transformation step and we offer transformations in both like SQL uh push down SQL that would kind of land in your datab bricks instance as well as some
like python Transformations as well um and lastly we have what we call an action River and these action rivers are able to run both get and post requests to custom rest apis so a lot of our
customers are using that either to pull data from you know longtail applications or like other third party Services um or they're using that to post data back out or run like a data activation um so in
that post um example you can use that to connect to like um DBT you can use that to run like you know Tableau refreshes or you can even use that to call like a datab bricks job to run as
well so I'll kind of show you kind of what the platform looks like and kind of hop into the different elements of it but there are kind of a few examples or you know some information I also wanted
to review there's this idea of like variables within the platform as well and this is kind of like a very important function because it really makes the Rivery platform very Dynamic
uh you can use these variables to parameterize certain parts of your Rivers so that you can use you know you can pass through one variable from one step to another you can dynamically
change those variables as well but it also leads into like our environments function where you can then build out deployment packages uh from like a test environment to like a Dev environment to
like you know production environment and you can see how those values change or those variables change from like the test demo or like a prod environment in this instance but we'll hop into like
the real meat of the platform in those like Rivery queries or Rivery Rivers themselves so we'll start with the source of Target and then we'll hop into the action and lastly we'll hop into
this logic River kind of show you how they all combined with each other so with the source of Target River these are all the different sources that we support so wide variety of like marketing analytics sources you
know rdbm M sources no SQL CRM source as well as well as some basic storage units like S3 GCS you know Azure blob FTP um but we also have things like you know web hooks where we can receive data from
a specific endpoint as well so from here all you have to do is like if you're looking for a specific connection let's say you're looking to connect Salesforce you can just select Salesforce over here and on the right
hand side it'll give you instructions on how to connect you can add new connection and on the right hand side it'll give you instructions on connecting or adding your Salesforce information
and you can just fill out the information on the left hand side you can also test the connection on the bottom left and you can just select your connection from the drop down over here
all our various sources actually kind of follow the same format right so that was Salesforce but let's say we want to connect mongodb we can just select mongodb click on the new connection and
again on the right hand side it'll give you instructions on how to connect and you can fill out the information on the left hand side so Salesforce was more like an API mongodb is going to be more
of like a rdbms or like a no SQL Source uh and for those sources we actually offer a few different ways to connect we offer like a um you can connect through SSH reverse SSH Azure AWS private link
as well as a VPN option as well and for all our databases we actually offer a few different options of extraction you can extract using standard extraction or using like a CDC method or in mongodb's case change
streams as well so we can give this a title at the top and just call this mongodb two data bricks and we can even give it a date so February 2nd 2024 and on the target side these are
all the targets that we support So if you're using data bricks you just select data bricks over here same thing if you haven't connected your data bricks connection as we kind of reviewed
earlier you can actually just hit new connection and go through that same information on connecting data bricks again uh fill that information on the left hand side one thing I did want to
note is this idea of like a custom file zone so by default like how loads the data is in this case we take the data from uh load it into a custom file Zone like S3 and then load it from S3
into Data breaks uh that S3 instance by default is managed by Rivery but what we let you do is that we actually let you utilize like a custom file Zone as well so you can stage your that intermediary
data within your own S3 or within your own blob environment and this becomes useful for a lot of um security teams especially because like all the data can be within your VPC and stay within your
VPC uh but it also helps where none of the data is like you can handle a retention so you can handle like auditability of like what that data looks like you can go back and see what
that raw Json or raw CSV looks like um and you can also you know create that like data Lake layer automatically where you have like that you know reference within S3 of what that data looks
like once you have that data braks instance set up you can just select the specific catalog that you want to pull from and we also have like helper functions on or you want to write to
sorry and and we have helper functions as well so this would just be pulling like a different list of catalogs available um and we do also have like a few other things um here as well that
make it a little easier so like once you select your catalog let's say we're using like Rivery test we it'll automatically load the schema as well um just give me one second pull up so I believe uh river is
supporting Unity catalog yeah so we uh we support both Unity catalog as well as a high store as well so once we load the schema you can just type in like sample and you we offer different
loading modes like upsert merge append or overwrite as well uh so upsert merge just loads on like a primary key you know overr right trunk case and loads and a pen just adds at the bottom but
from here you can just select the different you know schemas or collections in this case that are available within your mongod DB instance you can just kind of look through here look at anything that's
available and just select anything that you want you can either select all the different you know collections or you can select individual ones as well and you can kind of edit each one like the
target table that you want to load in within um your datab bricks instance or you can select like you know the load function the specific extract method as well so a lot of different
configurations here that are available I really love the simp I really really love the Simplicity and how everything is made accessible for everyone that's really amazing yeah it yeah it makes it super easy and I feel
like the instructions are also there to make it really easy for you to kind of like you know without like much information like if someone's not too familiar with like but they still
need to pull in data it makes it super easy for them to kind of just pull that through without like much you know you need some like guidance but Rivery does provide like a lot of the guidance
or instructions for each one and then it's pretty much just like a no code experience to kind of writing all that different information through so let me actually just rename this we'll call this R underscore and
on the schema side we'll just select the different tables so there's a few I wanted to kind of run through just to kind of you know I'll show you a little bit later how it all kind of ties
together but let's look for the specific I want to type in like user types maybe environments rules uh we'll put in like I think that should be good uh and then from here you can just this to run at Whatever
frequency that you want uh the highest frequency that Rivery supports is every 5 minutes and then the lowest frequency is going to be once a year but obviously you can also just schedule this ad hoc
to run as well and then this will just schedule or just run whenever you kind of run this process you can also get notifications on failures or Warnings or if it exceeds a run threshold as
well and then once you hit run this is just automatically going to run this is going to take data from mongodb right into our data bricks instance and you can see this on the activities as well
where I'll pull in that specific run you'll be able to see those metrics over here and you can see a broader activities panel as well that pulls in like all metrics from various you know
all your different pipelines as well then you can filter on the specific date time that you'd like as well and you can just see like that process like these are in the running status and these are the ones that I've
run previously with they completed time we also have other things where we can pull in like you know email data as well so if you have want to look at the specific River you can just look at it
over here and this is one that I've pre-created that's pulling you know email like an Excel sheet from email so this is just pulling from my own email and it's just pulling this Excel sheet that we can
then see here and same thing we're just loading that directly into Data brakes as well that's really any questions or anything so far youa no to be honest everything is clear and thank you for
those explanations I mean it's very simple and your explanations are very concise so I don't think I have I have but yeah I have a question actually so in your experience when it's the best to combine rivy and
data breaks yeah so I would say in my experience um like the best way to there's kind of like three different use cases I would say where like data teams benefit the most from like combining
Rivery and data bricks the first is you know for like f data ingestion for like databases you know various Cloud systems and maybe some like custom apis as well um I didn't show much about the custom
API connections but essentially like you're able to pull from like a variety of different sources using our rest API connector um and this is something I can show as well if you'd like uh but really
it can kind of just pull data from any like custom rest API and that's kind of done through what we call like this action Reverb here um this is especially common for like marketing analytics use
cases as well uh where many marketing kind of related data sources such as like ad channels social media and other marketing tools are kind of integrated within data bricks um and then they also
have like those longtail applications that they may pull from using like that custom connector um kind of the Second Use case so that's kind of like the first case I'd say the second one um is I would
this I would say like large organ organizations that need to manage like data pipelines across like multiple teams so instead of having like a bottleneck where like there's one centralized data team that kind of
creates all the different pipelines for the organization um they share like a segmented environment across like different teams so like the data team is going to own like one environment but
using like this environment functionality they can actually create multiple different environments where all the different teams that would need it so that way they can like parameterize or like you know organize
like you know one Department could have access to what their own environment and the data team can kind of govern that and control over access uh between all the different you know pipelines that
they have but the centralized team would kind of own that and then each department would have their own like sandbox essentially to play with um I would say that's kind of like the second
most popular like use case um and really enables like data management approach especially in like a data mesh way as well uh so the third popular use case probably is for organizations that want
to find themselves managing like too many tools and then wasting too much time as a cost of that um so you know usually some people are using a different tool for ingestion as well as
like orchestration and another tool for transformation and then another one for like data activation rather than having like four different tools for that they can kind of just have Rivery be used for
like that entire scope or end to-end process um so that's where we've really seen where like that uh River has been beneficial especially in like that tool consolidation so hopefully like those
three different use cases uh can kind of help you know explain like the best use cases where you can combine like Rivery and data break sensor yeah that would that would make sense and I believe I
should thank you for your time and I'll make sure to add all the useful links in the description of the video including the links to Rivery the documentation and in case you want to reach out the
team thank you thank you Zary no problem than you sir bye bye