Fluree’s Post

Transcript

Everyone, thanks for coming out to my talk today. My name is Andrew Johnson. I'm a lead software engineer at Flurry. And at Flurry we think a lot about collaborative data, which which I'll dive into in a second, what that means exactly what it would mean to open up a world where we can work with collaborative data. And I will just say most of the talk is going to be kind of a thought experiment about what an architecture of collaborative data would look like. It's not going to be really a product conversation at all, but we are a. A company dedicated to making open source data management and data infrastructure tools. So everything that you see today, if you want to try it out, you can go to Flirt dot E and try all of it out for free. But with that, let's dive in and talk a little bit about what we mean about working with and building application technology on collaborative data, what what we mean by that data, that worldview. And then at the end of the talk, maybe last 10 or 15 minutes, we'll actually do some live application development. Show what it would look like to find data that you're interested in and immediately start building utilities on top of it with 0 integration. So let's talk a little bit about first just what collaborative data could or or should mean. A little bit of this we'll we'll be talking about what comparative data isn't. But I think when we, when we hear the term collaborator, we think about data that we can participate with, we can produce or publish to a data set, we can consume from that data set. Very similar to what we might think of with GitHub and collaborative or open source code, right? Like I can go on GitHub, find code that people are shared with me immediately start building or extending that building. Application technology that leverages that code. And I think the goal is, and I think what a lot of us want is to be able to find that same collaboration in the space of data. There's really incredible tools out there, of course, Data dot World and Cagle that allow us to find open source or public datasets if we want to build applications on them. We still have to somehow transform that or load that into an operational data store. We then have to think about how we want to kind of secure that data once we're working with it. And the goal of collaborative data is very much like that GitHub model we find. Code or site, we find data that we want to work with and then we just start working with it. We start building application technology on top of it. But and if you're already thinking, you know that's a pipe dream or that sounds nice, but that's not the data architecture we live in. I think that's a great starting point. Let's actually talk about the data architecture we live in and think about what might be holding us back from traditional architectural stacks for for this kind of vision. So, you know, this slide does not show our traditional data architecture. We'll start layering on the kinds of inner infrastructure that often exists between our applications and our data. But this is kind of the goal of what we want, right? We find a database or a data set that we could build cool application technology around and we want to just start doing that. But what's in the way of that? And and when I say what's standing in the way of that, let me clarify. I don't think any of what I talk about in these next couple slides is bad. I think we need these concerns, concerns like security concerns, like modeling and integration interfaces. But let's talk about what they look like today for a little bit. One of the things that usually sits in between consumption of data and the data itself is just something like who is accessing this from the perspective of a different application? Clients and also individual identities? Who is at the end of the day trying to retrieve data and what data should they be retrieving? More directly, we often take some kind of authentication or identity provider service and then we start making sense of that and realizing that against the data with a layer like a data API data retrieval layer. Why is that necessary? Because, again, I'm not saying that it's not. It's. It's necessary because on one hand, you know, most of our data structures don't allow for data to be or or make it quite tedious or difficult for us to imagine querying or manipulating data from an application layer directly, right? We don't want to, as developers and probably for very many security reasons, shouldn't be directly manipulating SQL strings. We don't want to manipulate strings at all often when we're working with structure and queries. So if I. Make it possible for us to have a kind of operational data model and translate that back and forth with queries and transactions against the database layer. API layers are often where a lot of our authority, authorization security lives as well. And of course we need that, right? We don't want anyone to be able to do the kinds of things in an application that maybe only an admin should be able to do. So our API layer is a kind of a combination of integration utilities and security utilities that we typically need. And just finally we don't want our database often to just sit out in the open so that the interface with the actual kind of database layer itself is publicly available. So we have just general perimeter and firewall security that's even behind that kind of API layer, let the API be public or semi public, but the database is is you know closed off in its own private environment. And again I want to clarify these are. These concerns are correct. I think that we need things like trust and security, and we need things like integration technology to make it possible to work with data. With that said. Most of what holds us back from collaborative data or immediately integratable data are these two kinds of topics that that we see in all three of those layers. We have issues around data integration. We have issues around data trust and security. So not to read off of the slide, but I'm going to read off of the slide, right. We can't. Oftentimes we need some kind of an API layer because our data, our application speaks a very specific profile or or has a very specific operational view of what data modeling looks like and we need. Something to translate that layer, we need REST end points that might make our database state and whatever schema exists in return data that's operational to an application profile. And if we have one database that could possibly power multiple applications. Often what we find is that. You know, we need multiple APIs and we needed duplications of that data so that that data can speak the language of an application. That's an integration issue, right? Similarly, we don't if we're working in, you know, Sequel, we're working in a query language that is hard to manipulate. It's not clear enough how we could directly work with that data from the app. And we need a REST API to make that data useful in ways that our application can understand and and and useful in ways that we can manipulate requests and response easily. Finally on the on the data trust side, let's say that we had perfect data integration. Like we didn't need an API layer to model that. We could structure queries exactly like we wanted. We could get data back not just in its native shape, but in the shape of the application profile we want to build that we that we want to use to interact with that data. We still have a massive issue around trust and security, right? Let's say that we have multiple people producing data in the same database. You know, how do we know we can trust those individuals? How do we know who actually issued what update or what interaction? If we have the dream of one database but multiple producers and multiple consumers, what about all the anxieties about the provenance of that data? Or what about the fact that we might be developed developing ML models on top of that data, but people are still producing data to it, right? It's slipping out from under our feet. These are kind of the concerns that affectively justify the architecture we were looking at, but still are what are in the way of this kind of dream of collaborative data and just to go back one more time again. This architecture makes sense, but if we think about what that does to our data on the far left, it means that every time we have data that's useful to an app, we have to architect it into a silo. And then every time we might want to share that data with someone else within our own company or just within an open source community like in a kind of GitHub model. Every time we want to do that, we effectively have to re silo that data and rebuild that architecture. A new API for each app, a new data copy for each app. So there's lots of data technology that's being about this today, right? There's folks familiar with the semantic space or with the Knowledge Graph space know that there are W3C standards out there to make it possible to have material data in one shape, and to be able to virtualize or infer that data as if it existed in a totally different shape. But often those kinds of those kinds of 0 immigration infrastructures. Don't they think about this in terms of, you know, I'm in a larger enterprise, I'm an analyst, I need, I I need an already deserve to have access to a wide variety of data. But we don't have to think about the idea of inside of that enterprise, the kinds of trust that we lose once we start to have that kind of that Open Access to a wider data ecosystem. But if I genuinely want to share this, if I want to share this data with you, if I, if you find my data, want to start adding, collaborating with it, I'm not only need the integration or the the standards that make that kind of virtualization. In 0 integration possible. I also need to take all the things that really matter that are important. About trust about security and provenance, and I need to not have to reduplicate those in an API stack every time because again, I might have 0 integration, but I'm going to have to reintegrate it for every single app. So on the other side of this data technology that does think about moving some of those concerns into the data itself, that's great, but we need that with this year integration combination. So collaborative data, something like this target in the top right, right where we have data that can function across vocabularies that can be directly integrated. Without an API layer in front of that database, but also where we actually open up the possibility of a kind of multi producer, multi consumer relationship because we've taken the kinds of trust and security we need and put them directly in the data itself, we only really can share and collaborate on data when data can defend itself. It's kind of it's kind of the thesis I'm going here with with what would make what would open up the door to truly collaborative data. So what would that look like? Right? Let's look at integration and trusting. But think of a scenario where we had native integration and native trust and let's then talk about what that would open up. So native integration, instead of having to rely on on layers in between soloing or isolating layers in between our data and our application, if we had the semantic standards that would allow us to materialize data once but virtualize it or represented or interact with it as if it could. Exist in multiple vocabularies and application profiles at once. We never need an API to kind of specifically model data in a really application centric way. If the kinds of ways we queried for data and the responses we got were already object modeled, then we also wouldn't need the API to kind of translate REST endpoints into SQL queries and then resolve resolve those responses through an ORM system. On the flip side of that, if we we had that and we also had the ability for data to defend itself. If we took the concerns of the API layer and the perimeter and we moved them into the data, then we actually could we we actually could take our database and put it into a public space, encourage people, not only do we not want a perimeter to silo that off, we actually would want that perimeter list as much as possible, right. We would want to invite collaboration under those circumstances where collaboration is permissioned and appropriate. And we would also want guarantees that if I found a dentist that I wanted, wanted to again train a model on, it knew that that data was going to keep changing. But I wanted to be able to reproduce the same data state, evaluate bias, etcetera. We want things like lock INS on immutability or we want things like native audit trails of, hey, this data has changed since I last looked at it, who did what and when and what were the permissions involved with allowing them to do that. So this is kind of the picture of a natively or immediately collaborative data space and I think this paints the age of what it would look like. To have a kind of, you know, a world of open source collaboration around data. I have a database I've constructed. I've I've populated that with significant data. I want that data to get enriched by people who have other analytics. I want that data to get enriched by other applications. I want other applications just from a purely, you know, either because it's important to my enterprise or from an open source ethics perspective, I want that data to be immediately accessible to other folks. And as we do that, when we start to see is again, we don't need the perimeter, we don't even want it. We don't need the API either for modeling. Data handling data request for modeling responses. Nor do we need it for an authorization perspective. And we don't, you know and and I I will I'm going to hand wave at this a small bit because we could have a full conference on decentralized identity or kind of self sovereign identity or you know the good and the bad of web three, you know wallet managed identity but. You know, if we, if we were able to identify folks in a variety of ways, maybe it's in, you know, individually issued API keys. Maybe it's down to the level of cryptographic key management. But if we could identify folks without that API layer or an IDP, or if we treated IDP systems as just their own collaborators to a database, we wouldn't even need this kind of session management as well. And So what we end up then having is not just this pure picture of immediate interaction between one app and one database, but really like a single database could power multiple applications applications could consume. Data from multiple databases at once. And with that, let's actually look into what this could look like and test this out a bit.