Robust data pipelines with smart contracts Webinar

Captions as YouTube publishes them, usually machine-generated, so names such as Daml and Canton can be misheard.

1. 0:00

   hello everyone and thank you very much for uh joining us today uh today we have uh a special special webinar a special event and a special topic and special people today with us and why it is a special event because um this is a joint webinar together with the frankfurt smart contracts community uh we never did a joint webinar together with the smart contracts community but the people who are running this community they are residents long time residents and my friends

2. 0:37

   residents of the frankfurt data science and just my friends and also the people who gave talks at frankfort data science and today we're going to see how two technologies are coming together and how smart contracts are optimizing data science and particularly data engineering so um let me go through quickly uh through the agenda so what's the plan for today so after a quick introduction we will have a talk a keynote talk

3. 1:09

   then we will have a few lightning talks then q a round so feel free also to use the chat and the q a section to ask the questions so the questions and the chat bots will be moderated and then afterwards after the main part after the webinar we will have also a chance to meet in a more like easy going atmosphere uh in a zoom meeting so by turning on the cameras and just uh have a chat together yeah so um

4. 1:40

   let's go let's start i want to tell a few words about the franco data science so i'm elder and i'm one of the co-founders of the franco data science um data science is a community of people passionate about the data and analytics and it was i should say a local community but right now with a current situation i think we grow out of the location frankfurt and orion mine right now we do

5. 2:11

   uh events with other data science communities across europe and we really grew uh quickly so uh we have quite a large uh network and we organized uh many on site as well as online events at the moment and we also supported for example many people to connect with each other and also when someone was looking for a job we also uh trying to leverage different channels like youtube channel for example this webinar

6. 2:41

   will be also posted later on our youtube channel and we work closely with we are trying to work closely with different organizations companies and that's very important uh point because today for example our content is powered by guys from the dml so who are the experts basically in this field so coming back to franco data science uh we have a quite uh wide range of people with from different industries with a different interest

7. 3:11

   but most of the the most interesting topics for the community are around data science machine learning so we have quite a techie community so uh these are the the companies organizations with whom uh we worked we have a partners with whom we work closely and we have also contributors who help us for example uh with the content we organized a lot of events on different different topics starting from the demos where people could show exactly like

8. 3:41

   leery hands on no powerpoint we did large events together with two other communities for 600 people we did more recruiting oriented events with also other institutions and a few events that we are planning uh next week so next week we're gonna have this webinar about the object detection i will post a link to this webinar uh soon in the chat and also next week we will have another webinar which is made which is organized together with other data science communities

9. 4:18

   in europe and as you can see the topics are really uh wide and we are looking forward to see you also among the people who will visit our webinars uh right now i would like to give award to miguel from the smart contracts frankfurt community and he will host this webinar further miguel uh miguel i think you're unmute yeah i i mean i was just saying that it was the first for us

10. 5:03

    after saying hi to everyone and uh you can see that it's the first because i forgot to uh turn off the unmute um or turn on the unmute so uh it is a first in um two ways it's uh first it's a joint effort with somebody that we know very well and uh uh very much like to work with uh the frankfurt data science guys and elder specifically it's also a first because we uh intentionally decided not to uh have online webinars for or online meetups for quite some time because we

11. 5:34

    really like to meet people shake hands and uh engage in in discussions with a beer in our hands well you know that's not possible anymore so um and we didn't want to continue absent so we've been preparing this for quite some time and it's a pleasure to have a different format than what we usually have um first let me know who let you know who we are so it's me ali pasha and and uh rem um we decided to start uh this community two years ago

12. 6:06

    and if you could go to the next slide or if i could go to the next slide then you know we we didn't want to be a blockchain meetup we wanted to be something specific we wanted to be smart contracts um because we've been um working in this in the space of blockchain digital assets to student ledger and we figured out that um in order to make things at work then you actually need to separate three different components uh technology

13. 6:37

    business and the law um and in order to make things work then you actually have to align all of these things so it was the purpose of having this alignment this discussion and having three different communities actually meeting that we started something that's not blockchain so not technology related but something that is use case related so the smart contracts uh frankfurt um so we bring together legal people uh business people and technologists and try to make sure that they have a proper setup

14. 7:09

    where they can talk and discuss uh the ideas and try to figure out where you know what's the point of it what can we do um how how to proceed with uh with with something that with a field that isn't as mature as a data science field right we're still maybe in the stage where the data science community was uh 2014 2015 uh where you have the first enthusiasts um we've had a couple of events um

15. 7:40

    we did one that was technology related in april um 2019 but we also try to also stay healthy if you could move on please we did some events that were more uh specific to topics um so like topic of payments um as you can see with people from business but also with a lawyer and people from uh some well-known startups here in the in the region in the frequent region so that's uh who we are um that's

16. 8:18

    um it's we wanted to change a bit the the uh the form at this time we wanted to change a bit the topics as well we try to find something that is smart contracts related but could help us uh work together with the frankfurt data science but also especially with some uh with a company that we think is um doing something that aligns exactly what we have as value which is uh damo and i would like to give the work to francesco to explain briefly what tamil is

17. 8:51

    and uh what value they bring uh to this space so floor is yours that's good thank you miguel uh yeah weekly intro by myself uh i'm francescondroli and uh um i'm based in the zurich office i work for a company called digital assets their headquarters in new york i'm currently there a developer marketing lead so i spend most of my day um helping bringing new people to demo and getting them the support they need

18. 9:22

    when they're using it or anybody that don't know demo basically is a functional programming language build oh i see the folder and mostly the functional programming language that is designed for building multi-party agreements it's open source and you can deploy it in a variety of distributed ledger or central age ledger like sql um it's a very uh quite short language so it's extremely focused on business logic and handles

19. 9:54

    all your backend concern for you and i just want to say we know each other with with miguel since quite a lot of time and very happy that uh we are doing this webinar today also year 12 and uh let's see maybe in the future we'll do it all the physically thank you mike you think you're still mute yeah i'm um i need to get that one better yeah um so thanks francesco for uh the intro uh very looking forward to both the

20. 10:28

    presentations um also with guido in the end um i would like to give the word to ali pasha so that he makes a bit of an introduction to the uh to the topic that we want to discuss today so how did we put these two meetups actually together and put tamil in the middle um and then we would have a small discussion with uh with michelle um followed by the talk short talk by francesco and then at last then uh the the talk by guido

21. 11:10

    that will bring the topics together um so adi are you ready hello everyone i hope i'm ready let me see if i can share my screen um full screen so um now i need to change sorry guys um and i do want to explain um maybe in the meantime something about that to share you mean it's just a share screen that's true but i updated my pc and i have lost

22. 12:13

    all the permissions of this uh you should have the permission no i mean for my new pixel operating system so location contact so meanwhile i can see the results of the poll and where we see that we have a mixed audience today here so many people from franco data science and i can see also the names and many of the people i know so hi to everyone uh and we also have people from the uh

23. 12:54

    franco data science but uh what is really interesting that uh there are zero people who are attending uh events uh from the both communities and i think this is one of the goals today so to bring those two communities so people know about the technologies so data scientists know about the smart contracts and the smart contracts people can see how smart contracts are used in a data science and i'm looking forward to it alipasha

24. 13:29

    i am i'm working on it okay so maybe maybe we did just change it uh the order and then we will start directly with the and then do the uh we continue with you is that okay okay is it okay with your miser uh yeah okay cool um so when we started when we started um working on this meetup working on this topic uh we really wanted to have somebody that could uh we have an intuition right and the intuition is that smart contracts can help

25. 14:11

    um and we really wanted to have somebody that can explain to us where you know what the current pain points are you know how does it really look like from the uh from a real production or really large scale data pipeline somebody that has the experience somebody that has uh been working on the infrastructure somebody that has done trivial use cases um and we were very happy with misha um

26. 14:41

    except to tell us a bit more about uh you know what what his work is uh what he does in this space um um could you maybe quickly introduce yourself um that'll be great okay so yeah um my name is um michelle or michael um i'm actually studied physics and worked as a physicist in fundamental research for about 12 years and so what um what we did was we were in a big accelerator collect data and analyze the data and um maybe you would not have heard about cern and the large hadron collider

27. 15:21

    so i was part of this project and essentially what we did was also we did we built um data pipelines for a large scale like tens and and around up to 50 petabytes of data that needed to be analyzed over and over again to look for example for some correlation signals so that's the background is let's say a mix it was a mixture of physics um computer science data science um and also some let's say data engineering and and uh yeah real um

28. 15:55

    soldering engineering so um then one one year ago not quite one year ago i moved out of academia and i started new position with dxc technology that's a large company which came out of the merger of csc and an hpe uh the the part of hp so was formed also only i think a few years ago and i started as solution architect and in yeah the the realms of autonomous driving development so we are not

29. 16:32

    developing it but we are building the infrastructure the platform the tooling and doing the integration necessary for automotive companies that are oems so the car manufacturers but also the tier one suppliers to be able to develop and validate this functionality which is needed for autonomous driving and so when we talk of autonomous driving we think of fully automated vehicles like that's called well level five in the in this sae

30. 17:05

    jargon that means you you have no steering wheel anymore basically but we are not there yet so what is currently being developed is called driver assistant functions so highly automated driver assistance functions and these ones are for example things like what tesla has so that you can take your hands off the steering wheel in a highway situation for example and you can the car will will manage to break to keep the lane and so on so that's the the current um main

31. 17:37

    development focus and um so compared to my previous experiment experience actually uh that's not such a big difference because the amounts of data are actually similar it's in the tens to hundreds of petabytes and the processing follows a bit similar path so you have to process this data over and over again and you have a similar pipeline from let's say data collection which is happening in the vehicle itself

32. 18:10

    so the the car manufacturers they operate large fleets of test vehicles and these test vehicles are equipped with lots of sensors so radar lidar lots of cameras ultrasonic gps whatever you have the the vehicle data from the vehicle itself so um how to develop these these functions is you you collect huge amounts of data and then you have this data so we store this uh when when we implement such solutions stored in a big data lake

33. 18:42

    to have it available for processing and then when you process you develop a function so let's say you want to develop functionality to keep the lane so this needs of course you have to detect the lanes and you have to generate the commands to steer and then you have to test this and so one way to test this is just to run your algorithm on your data again so you build a big data pipeline where you input

34. 19:06

    all your data or all the relevant data and then you have your algorithms in the loop and you analyze the output by means of some kpis and then you have the next development step and you do this again and once you are sufficiently sure that this algorithm is actually making sense then you can start try to implement it in a vehicle so the first step is always you work on i mean compute clusters large compute clusters and you work on on big data lakes before you make the step to

35. 19:39

    put a function in a vehicle and and how um so two questions on this so what you're describing is mostly the the work that you need the data pipeline from the data guard reading so there's research there's analysis there's a lot of dialogue there's a lot of preparation and then you put the data inside the vehicle or you put the function inside the vehicle what would you say are the biggest differences between these two um you know

36. 20:11

    these these these two different setups so are they using so what's common with them and what's what's different from from each other okay so so if you're developing let's say a perception algorithm and then typically you have a lot of gpu you also don't care about the speed if it's executed a bit slower or a bit faster and you know you don't have real-time processing and you have a complete different tool set so this development of data science tools is centered around i mean python or the open source tools

37. 20:42

    or the machine learning libraries but once you go into the vehicle it's a completely different setup i mean you have a limited compute capability you have specialized hardware and you have very strict requirements in terms of traceability and in terms of audits and security and these kind of let's say extra tests which normally you wouldn't do so um the step i mean from this is actually a big step from um going from a let's say working

38. 21:13

    solution which is working in an in a big compute cluster on a computer to something which is actually working in the vehicle in real time and this is also i mean it's not a step we are doing but we are supporting it by building tools that for example automatically test your algorithms in such setups and so on one thing for example is that you really load this algorithm onto one of these hardware components you have in the vehicle and you let it run in this exact same hardware and you feed it the the

39. 21:44

    data stream which you already have collected previously so that would be uh one type of of data pipeline where you would let's say you would have to connect the worlds of let's say big data and you have to connect it to the in vehicle tools cool and um so that's so that so it's two different setups it's two different functions and that took two completely different sets of requirements right

40. 22:09

    i'm especially interested i have to say in the requirements uh for the runtime setup because i think there's a lot of verifiability traceability auditability that is required i mean every time the tesla crashes then um you know we we can't stop hearing about it list reading about it on the internet um how but could you look could you guide us a bit through what is in these data lakes what kind of processing there so what kind of technologies are you using

41. 22:41

    um are you cloud-based is it everything on premise what kind of skill sets do you need um could you guys a bit through the first kind of setup yeah sure so um these it's it's actually both it's cloud and it's on-prem and that depends largely on the customer so they are companies with a cloud strategy and they move everything into the cloud and there are some companies who are strictly on-prem so um this

42. 23:11

    this really um depends but in the end we tried of course to use the same technologies on both of this so we use large scale kubernetes clusters for example um for for uh cloud processing we use uh kubernetes variants for on-prem processing like openshift so we we make we make use of of containerization um wherever possible and uh for example if you think about storage we use actually we use different

43. 23:44

    technologies so whenever you're in the cloud you try to use make use of the the cloud native services um and then of course on top of this uh we are using um the the let's say big data tools so um using spark uh hadoop um also kafka sometimes so uh for for yeah data processing and uh but this is only i mean let's say this is the big data part but then i mean you see all the automotive manufacturers they have their own tools their own tool

44. 24:16

    chains and they are normally not in this parallelizable fashion so you have to let's say you have to parallelize and you have to orchestrate all these workflows let's say manually i mean you cannot always use ready available tools so you try to have orchestration based on docker if it's possible but there's a lot of legacy software which is windows and you have a lot of different software components which have to work together and be able to let's

45. 24:48

    say exchange the data or read and write the same data so typically i mean if you want to simulate a car um or you want to let's say you want to test your algorithm in a car if you want a realistic result you need to um you need to also realistically get the data through your sensors and if you look at a camera something as simple as a camera it's not just a camera which writes pixel output it has already software built in by the manufacturer of this camera so you have to

46. 25:21

    make sure that for example for this camera you if you put in the video stream that you already properly take into account which software version you're using there because if you change the software version and maybe this changes the output of this camera it could change your whole whole chain and you have to keep track of all these changes you are doing so you have to keep track of all these software versions

47. 25:43

    and now there comes maybe also a possibility for things like smart contracts so what is complicated is that whatever algorithms are in development typically the sensor suppliers for example they don't want to give all their ip out of their hands so then you're left with let's say the option that you ship give them the data they do this part of the of the processing and they give it back and so that is also happening that's

48. 26:14

    something and of course if you look at it it looks uh quite inefficient that you that you ship data instead of shipping the algorithms to the data which considering you have petabytes it would make a lot of sense so but there is more to this than only the technical part and this is also for example this part that there are a lot of different suppliers and everybody uh has his own format his own um

49. 26:38

    let's say way of processing this and the challenge is in the end to get everything work together so that you in the end have a have a let's say a setup which is as close as possible to what you will have in an actual car okay and so i i guess that that's something that you definitely need for the production case you know when you put the function or the functions orchestrated live um

50. 27:03

    tell us how many people work uh for example and and when you think of a function you know how many people are working in that sort of specific driving function how many people are working in this so how big are the teams uh so that would be uh question number one and then um if if you think about this orchestration bit um there's a lot of inconsistencies as you said in the data formats um how how how close are we to having standards

51. 27:36

    um and uh who is dictating the these standards is it the is it the constructors or is it the open source community um what are you seeing from your side so let me start with the second um question so on on data performance there are actually several standards but then there are variants of this plus there are many um i mean there are many software companies building software with their own data

52. 28:08

    format there are car manufacturers or or suppliers automotive suppliers building their own format because everybody thinks that uh the formats which are existing they don't suit our purpose so let's build our own it will be the best ever and then in the end you end up with with a lot of different formats so um there is some there is also some there so also a format which which came a bit out of the open source um community but in the end typically uh what is done is if you want to use

53. 28:39

    these formats and you want to feed them into machine learning algorithms for example and you have to transform the data to let's say from from some automotive mdf4 to a numpy array okay okay cool and there's a question the question on the on the team size so so i don't have um let's say on the developers which are developing the driving functions i don't have direct insights but let's say on the typical uh functionality um just

54. 29:11

    i mean defining requirements testing and engineering this um this is typically i mean this is uh for for um for several functions which are then existing it's uh probably it's the order of a thousand uh people working on this not from us but from the from the oams and from the tier 1 suppliers and so what what we are we have considerably less people in this because we are providing let's say platform orchestration

55. 29:41

    automation and and scale things up but uh we are not the ones um our company is not the one who was developing now the the function itself okay cool so that's that's a very very big setup to coordinate from uh the human but also from the technology point of view uh so i have in um additionally to what you said about the the guarantees from the smart contracts i have to think whether you know these standards that we have for

56. 30:13

    uh putting logic there could also help um so thanks misha there's a there's an outstanding question from the audience about streaming formats um i i i would like to ask if it's okay uh to move into the qa in the end we would like to move on and as we start talking about smart contracts um we would like to ask you how familiar you are with uh with that topic um so you're asking me

57. 30:46

    um yeah you okay in the audience okay so so i can tell you that i'm not too familiar with it i only know it from let's say i know blockchain technology to some detail and uh smart contracts on on ethereum blockchain um but let's say i have not um i didn't dive into into the depths of this but let me just tell you one one thing where i think this might be um uh might be of use actually um i told you about all these suppliers

58. 31:18

    which have to work together and exchange data data and typically it's you always have a dispute about who has delivered how many data to whom and so where this data is and of which quality the data is and everybody is keeping track of trying to keep track of everything and then occasionally you you have a meeting and you um try to align all your um tracking of everything all this data and i think that it would be a potential use case for a technology like a smart

59. 31:49

    contract on on a blockchain where you can even i mean you could even verify that everybody has received the same data by using cryptographic hashes and you could also get let's say this whole process this whole coordination process in an automated fashion which will not be then by oral and written contracts but uh yeah by by uh let's say contracts as code and we try to define everything as code right i mean

60. 32:20

    all pipelines infrastructure everything we try to find as code so we can uh we can versionize it and yeah i think there would be there would be potential applications of that cool we we agree so elder do you want to show the results do we have results already cool there's somewhat familiarity um maybe we'll see whether this improves with uh with uh with the with the other talks um hopefully that's the whole point um the insights from guido francesco and then

61. 33:05

    first uh alipasha giving the work to you oh oh thanks michelle so should i try again let's see if it's working can you hear me can everyone hear me okay good so let me share my screen okay all right so um michael talked um wonderfully about the complexity of the pipelines and since we are in the middle of joining two different technology camps together there are still a lot of people which i talked about before this

62. 34:00

    meetup who were asking why what is it pipeline and why do we need it and i think michael got more into the detail about the python himself i will try to explain the complexity of the data science and bring an example of why we would need something like this data pipeline so this is me i do technology providing programming and i am a data enthusiast i don't call myself scientist because i use it more i work a lot with data but

63. 34:37

    more like from responsibility and point of view so as you know in 12 2012 that the term data scientist became popular by calling it the sexist job of the century which brought a lot of people's attention to this field however it's not usually as sexy as people think because you do have a lot of data overload when you're working with in this data science area and there are so many different factors to consider it's not just that there is a lot of data but there are different structures of data

64. 35:18

    and there are many processes there is this understanding of that eighty percent of the work of data analysis is about cleaning and preparing which has also been explained in this external project data mining and data cleaning books by devon johnson and has also been populated very happy with richard who thankfully created a model of for a process of getting the data under control by by having processes for inputting the data by tidying the data which basically

65. 36:02

    means you start building a structure for your data without getting first into any meaning of the data but just having it prepared so that it's useful just by its structure and then you move over into understanding the data by transforming the data and building your models trying to visualize parts of your data and then go back and remodel and re-transform and once you have a good feeling and understanding for your data

66. 36:35

    then you decide to make it more uh robust so that you can communicate with the confidence about your understanding of the data but also you get ready for automation so um why do we need that automation we need that automation usually because there is a lot of data and it gets complicated let's take an example with the current issue a situation with coronavirus and covid19 and let me make it disclaiming that i'm oversimplifying a very complex issue

67. 37:16

    that is ongoing and hasn't been solved by anyone yet but it helps to understand that it's more than just grabbing data and building a track so first of all we have um we have to know what we are going to measure and why we need that and we have this case fatality ratio which is the proportion of introduced uranus disease who died from that disease okay but this actually is only reliable at the end of the outbreak once we have all the data available so then we can

68. 37:54

    make some conclusions about it since we are not at the end of the outbreak and we are in an ongoing process we need to estimate cfr and we actually need to estimate the cr we already see that we have to add the number of recovered cases too to have a more reliable cost unfortunately there have been so many different versions of here at the beginning and the ascfr by itself does not eliminate all biases from the system

69. 38:31

    so even then we need to take one step further um and so that we prevent uh misleading statements because it's about a lot of things going on right it's about lockdowns it's about economic survival it's about it is about um isolation of family members and so on so the next um ratio that people um work with is the infection physicality ratio and this one is more robust but it is also very hard to estimate

70. 39:11

    because you need to do the real um the testing of people to exactly know if they do actually have the disease rather than just um matching some antibodies so this one is slow it's costly and often only only partially available i mean if you have only partially available data then you cannot do a quite predictive analysis so even here we need to do some predictive analysis for ifr so because we even with ifr we can only be sure about

71. 39:46

    the numbers at the end uh we need to have mechanisms to uh predict that which is basically what data scientists do right so we take data that we have and we try to make predictions so that we give pm responsible decision makers tools and uh guidelines to to move forward but that's not all because we see that comparing data is complicated between different countries and even uk had to remove the comparisons that they started to bring out countries count deaths

72. 40:26

    differently that's the simplest and probably the most interesting point to make of course so many different decisions that are made but this one is very crucial i think belgium is one of the countries that have been counting from their understanding rightfully but for the understanding of other countries as a over county while we have had less and less data from china so we couldn't

73. 40:56

    work with those efforts anyway so that's important um so all countries also test differently so there are different versions of the test so that also has to be calculated but it's also uh you need to also adjust for age sex and health and that hasn't been done by all countries with the same standard so so you need to also take that into consideration but that's even not enough because you also have to obey the law and there are some

74. 41:29

    human rights and privacy rights that need also to be taken care of right so you need to know which kind of data you are going to use what's the time limit for that data how you are going to use that data and all of this information is crucial but even that is not enough because beyond obligations and scientific understanding you also have a social responsibility of how you are

75. 41:57

    going to deal with your data and communicate them so as you see there are so many different factors that you have to consider you have to consider methodologies fatality rates regions individuals uh special cases privacy laws human rights social responsibility and if you want to have um do this study on an ongoing basis because you are building your estimates again and indeed as the data changes

76. 42:27

    you have to have a system that is reliable that is going to produce a result every time by acknowledging all of those criteria because of errors are too high the process needs to be verifiable especially if you are going to verify that you are about obeying the law and you have to repeat the process again and again and it this is just only one part of so many other factors that need to be considered so that's basically why i'm saying that you needed

77. 43:00

    a pipeline i hope that helps a bit to understand the complexity of the data and the process going on i think so that's uh thanks very much ali um and it's uh kind of a very um important topic um to having proper decisions being made from data right how do you kind of can you be sure what the data means how can you be sure where the data is coming from okay cool um i would uh um like to give the ball then

78. 43:39

    to francesco yes thank you our special partner for this meetup thank you very much and you guys were really a spot on on on you michael speaking really about the supply chain use case and ali on the was that on the 60 um timing that you're taking for actually you know filtering and uh and not actually the focus but anyway let me just start um i have just a minute so i try to uh use it as much as i can and uh

79. 44:13

    just quickly introducing i will quickly introduce you what's the problem on our side and how um and how we can solve it from uh from the demo perspective and then guido will take over and we show you a nice solution for a practical example uh so as you can see here in my slide and i'm using a couple of monitor i hope i'm always looking at you guys but uh let's let's hope that so uh traditional problems between enterprise uh today is mostly that we

80. 44:44

    operate in silos right and and blockchain was meant to integrate different enterprise and to bring them on on the same page at the end what we're trying to uh to talk is about application with the enterprise so um as you probably may know any business processes is comprised of multiple business application they have one two three four or unlimited business processes and each of those application portfolio

81. 45:16

    have always their own database or they often have um have to keep in sync with each other through constant reconciliation so basically they have to pass data from any application on or from to one other application and in order to say sync um so i just try to actually that's very similar and uh and great uh michael that you mentioned this supply chain i i have a similar example that i want to show you guys and uh that's uh easy i mean a very simple business process that you you can think is about

82. 45:51

    the credit card process uh there are multiples let's say process tab processes with multiple database multiple database so i mean you can you can get from an application to a credit card that goes into a system the credit um from the tension department it's also maintained it's in own database um every all the data is also stored in an e statement system and you can see my point is basically here is uh that a simple business process is being executed by so many different

83. 46:29

    applications and each of them are having their own database so and this is actually very similar to what ali mentioned it before because at the end of the day if you want to run analytics or having any predictive modeling afterwards uh what you need to happen end up doing is is aggregating data from all different sources and basically what we are showing here is that mostly up to 60 of your efforts are finishing up on aggregating and cleaning and what actually your goal

84. 47:08

    will be is focusing on uh on the fourth point right and basically the solution is coming in here right uh so this is basically the problem that smart contracts are solving and the way we do that is to bring all of this application um on on one common business workflow model using uh smart contracts in this in that case demo and you can still maintain privacy so that's that's basically the point you can still maintain privacy between

85. 47:41

    different applications but you still allow them to operate on a single we call it golden source of true right and let me show exactly the same example but how we look like so basically um basically this is reflecting to this example so if if you see here there are different application and all of them have their own internal silos but at the same time here that's what the difference is they're using smart contracts and models that they

86. 48:14

    allow them to have a common smart contract store so basically which me so this is basically meaning that at any point of time if you want to get a snapshot of your business you can clear your smart contract store and fetch the state um and that's actually quite an interesting value proposition because the let's say the the other process that you should normally do is fetch this data from any from every different database from each application

87. 48:46

    so this i will probably like send over those slides afterwards but i want to focus on the main point and and here uh basically i want to say a couple of things so demo is purpose-built and we probably didn't mention but in a couple of weeks we have this hands-on workshop that we can show you uh how does it work really building a quick demo and a workflow in front of your eyes but the other things that i want to show is basically

88. 49:18

    this interval ability them also it's interoperable that means that you can have your application build it today on your um let's say sql a centralized database and you can move on in couple of years if you want in a fully decentralized ledger so basically demo can atomically connect those different networks together and most importantly allow also to connect those networks seamlessly with in the internal network itself

89. 49:48

    yes i know have i don't have much time but basically i want to show a quick quick demo from from our colleague anthony this is a tracking a beer tracking um uh landscape where you offer and own beers and the demo is basically build it on on top to understand how to model those uh multi-party agreements and this is how he looked like the the backhand so it's less than 50 um both lines and uh it's quite interesting because it's not the same as the front end right

90. 50:21

    i mean as you probably uh you guys are doing that's quite a long um a long difference but uh but but yeah um you can check it out also it's open source and the application is uh is available for everybody i want to end up saying a couple of words right we have uh uh we have the forum and any that's a dlt enthusiast or demo um enthusiast is asking questions there so if you have any question about this talk or about the the webinar itself please feel

91. 50:54

    free to to register there we have question news tutorials or guide and uh and we are also probably posting the recording i assume um from from this if you are also beginner and you want to start in terms of starting smart contract modeling in general i i really recommend you to check out this demo.com slash learn there is a very similar to the openshift one you can basically build an app on your browser without uh installing all the dependency and then thirdly and we have this

92. 51:27

    workshop we are not doing it next week because of thanksgiving probably in in two weeks and it will be fully hands-on and will be uh probably we're still debating on a supply chain use case or a more workflow use space we will build in front of you a quick app and everybody can see what's going on so um i think i had everything and we have a q a i think afterwards we can definitely handle the q a and i will give the word now to guido that he will show you

93. 51:58

    a practical example thank you guys okay thank you francesco let me just share my screen i hope you all can see it right okay so first a little bit of introduction about uh about those about genesis studio i am the co-founder of genesis studio uh we are a startup based in portugal um we started focused on dlt and blockchain technologies but but we are escalating to artificial intelligence as well machine learning rpa and uh business intelligence too so we are escalating to all of these different worlds because

94. 52:41

    we truly believe that they are all interconnected in in the in the short term um so this is our stack our best is uh okay so from our perspective what what do we understand about uh about the difficulties for data scientists uh currently uh as francesco was telling to we currently have this landscape of different applications with different databases uh probably one master application managing all the workflows and other applications trying to get data from there

95. 53:16

    then if you want to do anything with that data you still have to go to the traditional etl process before you can get you can do something with that data with smart contracts uh what uh what can we see that that may happen um all this etl process if we if we are using smart contracts can be truly simplified because we are we are using smart contracts as the the main workflow for all applications uh

96. 53:45

    means that probably in the short to mid term individual databases may may be phased out you will have your golden source of truth that can serve multiple multiple applications and also it gives you freedom to change your applications whenever you want because they do not have the business logic on the on themselves so you can well today it's react tomorrow it's angular or it's anything else and you still have your workflows on only one one place so you can rely on that and just change

97. 54:18

    all the application that that you want so for this what we have we have our product our gentle uh it is a little bit early stage yet but we we see it as a smart contract driven orchestration engine what this means uh we have all our workflows running on our on our application uh serving different front-end applications rpi we have apis to deal with rpi technology as well um so for this

98. 54:50

    the the smart contract store is really the the central piece for this it's the golden source of truth uh so all the steps that the your business workflow is doing your business process is doing all the interactions uh everything that's happening it's being recorded uh out of the box it's uh the auditability that comes through with with them all with demo smart contracts this gives you more or less control of

99. 55:20

    all the data that's coming to a single place and you can start building uh your algorithms on top of that so the idea the main idea is really not to put your data into our algorithms but instead push your algorithms to your data not having all that work off and now i have to go to these 50 different databases collect the data clean it reconcile it and then yes i can start thinking about applying any kind of algorithm

100. 55:50

     being machine learning algorithms being used in business intelligence applications you name it on top of this smart contracts also gives us some some automations for from demo side in this we are we are looking and using more and more the the triggers and functionalities so this allows us to have different automation processes on the data side and not on the application side again giving you freedom for your applications to be whatever you want only on the front and then being as less

101. 56:27

     as business logic in there as possible that makes our data more automated as well so a really concrete use case that that we already built with gen flow and we can demonstrate it whenever you want we have been doing this on the insurance industry uh we are doing the insurance policy life cycle and insurance claim life cycle so everything from the simulation the quotation the exception

102. 56:58

     or rejection of the the policy uh changing vehicles then reporting an accident involving different parties on the accident being other drivers even the policy the police if you want to to define who is guilty or not if it's not a friendly accident uh so all of this is is done using the the demo smart contract uh we are already integrating with the liferay in this case it's our cms a portal front-end integration with different with different applications

103. 57:30

     via our apis and then with with our golden source of tools we can build consolidated reports and policies and claims on incidents on whatever we can extract from that open source of truth from there we can really know pretty much anything that that we have so how does this work for for these type of processes um we have our our own database our golden source of use we are currently using

104. 58:00

     well just a positive sql database um from here we can extract via api raw data json data from here we will pre-process it clean it try to analyze all the attributes defining which ones will be used for for training the algorithms which one will be used to really run the algorithms and then we can feed it also to reporting tools business intelligence tool like power bi tableau all of those those tools that we can

105. 58:31

     that we can do by the end of it we can have like any kind of result from from this from this process like a forecaster validation recommendation well a variety of things that that can be the outcome of this um well and i think this is about it i think i was faster but yeah this is pretty much it so this is our contacts feel free to reach out to us visit our website cool that was uh that was that was awesome that was very fast

106. 59:05

     uh so i guess that uh that's also leaves us a bit of time for some questions q and a yes so i give the people time to warm up and maybe start with one myself um so just just to get a better general feeling so is it so are you more working with online so is it is it more for a data warehouse for an offline machine learning training setup or is it more for something that is online so like streaming data real-time decision making so where do you place

107. 59:44

     yourself well probably both i would say uh as we can aggregate all these data in a single source of truth and golden source of truth we can probably take care of both both use cases really okay and so if it's both are there uh what about the execution time so is it what kind of logic can you can you use uh or have you tried what kind of complexity do you have there um well in this case on our insurance use case for instance we are

108. 1:00:16

     we are putting machine learning examples to try to do cross-selling and upselling of different insurances uh to people so imagine that you have a vehicle insurance only try to set up like anything else and then health insurance as well try to understand your behavior on on your online journey on our on our website for instance while you do your your simulation try to get the exact

109. 1:00:42

     feed product to you and that is all from these type of algorithms and uh from from this type of data okay and and what it what specifically does damol help you there with is it more the verifiability aspect is it the fact that you have maybe different participants um working on the workflow is it the compression power of the smart contract language for us it's more of the keeping the workflows only in the in one place and a more business oriented workflow because

110. 1:01:18

     as francesco showed it is a really easy to read language so this is one of the biggest advantages and then all the built-in advantages that it gives you as well so the the rights and obligations of course that is one of the biggest advantages you can clearly define who can do what on which time in each in each template and each contract so that type of features that you have built in on the smart contracts removes complexity and programming on other type of on the application side

111. 1:01:50

     really so that's the the two of the of the biggest advantages that that we see on demo yeah cool thanks we i see we have a question from uh from the from the chat um the question is where do this by contract uh sit in the last example that you've shown so could you maybe maybe guide us a bit through the deployment and the infrastructure of that example yes let me just bring it up again

112. 1:02:25

     okay so i would say yeah probably in this one it's the the best one to explain so the smart contracts will sit on this side this is a traditional this is our our site so the applications are still there the databases may or may not still be there but the smart contracts will guide all the data related stuff so all the business process for one side and on the other side all the data that is coming from the applications into into into here into the business process

113. 1:02:56

     so in here it's where all the smart contracts are from here on it's different of course it's not the smart contract related but here it's where where it all resides okay hope it is clear for me it was but if people from the chat have further questions or maybe adi francesco yeah maybe i can add something i mean at the end of the day i mean yes all the smart contract part as uh as either showed right it's before the the storing but

114. 1:03:35

     any use case that you think about on on basically modeling uh different parties behavior it's it's actually great uh to model on uh i mean theoretically you will do it on on any smart contract uh platform but uh it's quite important that you're building something fast and you can show a an mvp for example but yeah i mean any just to resume any um use case that have multiple parties engaging with each other that's uh that's that's a great use case right and uh i really want to touch up the one from

115. 1:04:09

     michael that he spoke about the supply chain i mean it's great right and uh and uh yes question from my side mr so what would you say are when you look at that kind of pipeline and if you think of the kind of problems that you need to tackle right so what what do you think are the requirements or things that you think the smart contract layer should be able to do you know both

116. 1:04:33

     both functional i mean from the functionality but also you know processing time um parallelization support of streaming so what do you think what are your thoughts and when you look at something like that so so my first thought obviously is that this this smart contract uh it it's something where you do not want to have a lot of data in right so that's not something you can scale to a

117. 1:05:02

     very large volume so what you would use it for is let's say some kind of metadata right and you were talking about giving getting a let's say a common view on on something i mean you could use obviously you can you can you can do that for let's say code or so directly but for large data volumes you would not i mean you would not like put your in this in this kind of pipeline you would not put your your normal full processing right um so so from my

118. 1:05:34

     side what i what i would see is that it needs to be something which can be integrated in let's say other existing workflows and which would let's say add a layer on top and only serve let's say one specific use case let's say for example you want to have all a common understanding of where which data is or so so that would be something you could you could um i i could see that this would be a

119. 1:05:58

     use case which could be rather easily implemented in such a framework yeah and you see more the live use case or the training use case so deploying the function or coming up with to the function well i would see more let's say a bit offline use case so that uh so making sure that the model was properly trained right so however if you're now looking at a model for example it would it would be let's say you you could say

120. 1:06:33

     which model you're using right so you could keep track of which model was provided by for example a supplier but i was i mean what i was talking about about exchanging data and so on was a bit it was even more general i mean it was uh it was basically um the the normal supply chain i mean offline supply chain but with data so who supplies which data and yeah i think that that is what what i i mean currently see as uh as a problem so i spent uh actually

121. 1:07:07

     uh most of my day today to fixing these problems yeah i guess i guess uh ali pasha would agree right with his example from covid exactly so so basically uh with data from resources and once you have the data you have some kind of a time stamp that you know that from the point of your responsibility the data has not been tampered with that would be also something you could use smart contracts for

122. 1:07:38

     and then also something that is important is why we move we want to have some move towards smart contracts is because up until now a lot of the times we have talked about verifiability of the data but the processes by themselves are also important right so if you start to switch between a standard of measurement or a calculation to another you need to have that uh take that into discount not that

123. 1:08:10

     someone comes up and updates the algorithms and change that without having any kind of um verifiable um uh kind of um record about that so we need to know uh for example if things change quickly how are we moving from a certain kind of a process to the next one and to the next one so that if you go back and want to evaluate the data that we have had and the the responses that we have

124. 1:08:40

     the results that we have created to always know that it's not all made with the same kind of uh um process so that's also helping with smart contracts because they also have intelligence and of course access control is very something that demille does really good so that's also very good because it's it's readable within the code that you can see who has the right to to call certain kind of functions without having very complicated access control those tools

125. 1:09:13

     and api calls to some external force like you don't see that in your code yep definitely definitely agree so last call for questions for the chat otherwise um yeah so francesco maybe you want to show us something more about the upcoming webinar so the workshop yeah for sure um so basically um yeah i invite you all i will put the link in in the chat itself but uh we will have this uh hands-on workshop and the idea is

126. 1:09:59

     basically having uh having a live um a life built together right so we we are going through this uh um this demo and we're building it from from scratch and uh we're still debating as i said before right to have a cool use case for for data science in general but but something like that and we are planning to do that on in two weeks wednesday and uh and yeah and everybody is welcome especially for people that

127. 1:10:31

     really also never heard about demo they want to check it out how you assign you know different parties accessibility and and looking how easy is to do it because as this is a functional language it uh it's it's quite i think i mean on my side it's more it's more intuitive than an object-oriented one so yeah join and uh and uh yeah i'm happy to see you guys uh in two weeks cool and thanks that will be there

128. 1:11:06

     yeah so let's make sure everybody knows about it and gets the link uh it's in the chat um yeah otherwise if there are no more questions ella do you want to say something or should we just thank everyone uh that was a very enjoyable hour a bit more um i learned a lot was good to hear about um you know what's uh what's required for having some driving assistance in a real world production case sounds like a lot of work with a lot of people involved um

129. 1:11:44

     with a lot of technical and non-technical challenges probably also ali thanks for bringing up such a good example from the um from from from our current uh real life setup uh and francesco uh for bringing uh the topic of smart contracts and obviously for the example that wraps it all from guido that was very enjoyable so thanks everyone um i think you will get news from further events from the current channels eller do you want to

130. 1:12:18

     say something yeah so uh thank you very much of course uh to our speaker so we have we have today we had today global coverage and uh this topic smart contracts that uh used in data science we didn't have until now such topic at our events so i hope that it was also interesting for the community to hear and see what could be done and also join the workshop which francesco just shared in the chat

131. 1:12:48

     so thank you very much for everyone thank you very much thank you thank you guys bye bye