Beyond the relational database

Chances are that most of the transactional data in your organization are still stored in relational databases like SQL Server, Oracle and DB2. When modernizing your application landscape this will rapidly change. Why? And what does that mean?

The problem with relational databases

Modern software solutions are service oriented and very distributed, where the systems of record are still hardly having a user interface but do have good, mediated APIs that expose business logic as services. And the tailormade solutions are created through low-code and integration on top of these APIs. The composition of these services exposed through APIs is taking place on a higher level. That is where the differentation takes place. That’s where you use small apps that do exactly what you need, fully integrated with your application landscape. The underlying systems of record will in the meantime also be modernized by the suppliers, where these will also move away from the silo approach and relational databases. Mark my words.

There are three great problems with relational databases:

  1. Relations are hardcoded, to ensure referential integrity. This cannot be used anymore in a software architecture based on microservices and apps.
  2. Transactions are executed in the context of this one database. That no longer works in a distributed application landscape. Commit and rollback are phenomenons of the 80’s that cannot be used any longer.
  3. And, not to forget, the costs of relational databases are way to high, because most of the times they are end-user based.

In practice, lots of relational databases are misused to store non-relational data. Just because the thing is already there and operated. And because it easily goes with the regular backups. That’s a wrong strategy. It should change. Now.

How to prepare?

Now is the time to slowly say goodbye to relational databases in your tailormade solutions. For every need of data storage, try to find out how relations with other data can be created. These days, there are lots of different storage methods, especially in the cloud. To name a few:

  • NoSQL or Document database – here you can for example store JSON objects and search easily and flexibly. These databases don’t use schemas.
  • Graph database – this is where you can store all kinds of information and dynamically create n:m relations. Ideal to create profiles and have the data that is interesting for you actually come to you.
  • Data lake – here you can store all kinds of varieties, volumes and volocities of data and do big data analyses. This will often be machine generated data, such as with IoT scenarios.

In a (micro)service architecture it is important to cater for eventual consistency. The database cannot take care of that for you anymore, simply because that cannot be done overarching multiple databases. How you organize your services and APIs will become more and more important, including the data architecture belonging with that.

So, what about business intelligence and analytics? These solutions will have to change rapidly as well. More and more data that is needed to show in dashboards or reports will come from non relational databases. Traditional ETL cannot be used anymore. Cubes cannot be created that easily anymore. Artificial intelligence as part of your analytics solutions, even for the traditional hindsight applications (“wat happened?”), cannot be avoided. Let alone if you want predictive (“what will happen?”) or prescriptive (“what do I have to do to realize this?”) solutions. You’ll have to start thinking about these kind of things and adjust your architecture to it. Waiting for “the hype to blow over” is not a good strategy. The relational database is already clinically dead IMHO. You can throw your SQL skills in the bin in no time!

Cheers, Gijs

Serverless. Basta!

I was once a system programmer in a Unix world. Brilliant OS. The first version I got to know really well (and I mean, deep down really really well) was System V. Later on, I worked with (Open)VMS and also spent quite some time porting stuff to Linux (Redhat). I mean, communication protocols, compilers, etc. The really hard stuff. And fun it was. Getting stuff to work was really very fulfilling. Especially in a time where you had to build your own “platforms”, like 4GL, RDBMS and integration middleware in order to provide your internal consumers with better solution building productivity. Building all this stuff was awesome!

And then I was introduced to Windows. Lipstick on a pig it was. And in quite some cases still is. We were serving 8 developers on a 486 with Redhat Linux and with Windows 3.11 on the same machine we only got to serve 1. Aaaargh. But hey, developing really visual stuff was a nice change as well.

But the best thing today is: we don’t have to care anymore, because we’ve got PaaS now! Who cares about servers? We just want to run our workloads serverless. Let the hardcore developers build this cool platform stuff (and make it very very easy for us), that we ordinary folks can just use to deploy everyday workloads.

But, lately I got introduced to this relatively new phenomenon of containers. I think that’s a step back. At least, for us people who just want to deploy common workloads. I understand that for architects and developers working for large scale B2C companies (Facebook, Google, Amazon, etc.), containers and K8S and stuff like that is great. But for the average company, it’s overkill. And overly complex. And back to virtual machines, but just a bit smaller and more contained. And somewhat easier to deploy.

But, we don’t need that (in our platform). We just want less complexity. And more less. Serverless. Basta!

Just my 0.02 Q* of course.

Cheers, Gijs

*Want some for free as well? Just register here through my personal invite link.

A Darwinian view on algorithms

A common notion these days is that “data is the new gold”. My opinion on this is different. Data are just common natural resources. Not that scarce and therefore not worth that much. But when combined in a smart way, it can become gold. So, what is the new gold is algorithms.

“Algorithms are the new gold, not data.”

Companies who found this out, after they initially still quite naively started providing neat features to end-users are Facebook and Google. But, by now they have become behavior manipulation empires (see Jaron Lanier’s great TED talk on this). The algorithms these companies apply are opaque. We don’t have a clue what’s happening there and to be frank, sometimes they probably don’t know themselves either.

As has been seen in the news the last couple of weeks, these algorithms can turn into real Frankensteins sometimes. But I presume lots of smart data scientists work at these companies and they know what they do. Hopefully.

But what will happen when not only the data that is used as input can be manipulated (for example by generating fake data through the use of trolls, from Russia or wherever), but also the algorithms? What if these algorithms can be injected with malicious code? And produce results that better fit the people or organizations that put the code there?

We get manipulated by algorithms more than we know and probably is good for us. The singularity is already here, but we don’t want to admit it yet. By far the best example is that probably more AI generated babies are born these days than the good old fashioned true biological ones.

What if the evil people can influence these algorithms? And start generating “odd” (but “more desired”) couples by influencing Tinder or other platforms that people use to find potential life partners? What if during WW2 big data would have been available and algorithms could have been manipulated? Would we all have spoken German now? That’s a pretty scary thought. And there are plenty of dictators or dictator-like folks in high places right now that could do this.

So, what’s next? How can we protect the algorithms? More quickly (and openly) get to know when systems have been hacked? Is open source the way to go? Shouldn’t we let these platforms get too big? Do they need more inside and outside governance? These will be the major topics we have to deal with in the short term, or otherwise evolution will not be going slowly and gracefully anymore and Darwin’s insights will be obsolete soon.

Cheers, Gijs

The El Niño of the digital ecosystems

After digitizing current processes we started optimizing them. All purely focused on efficiency. Saving costs. Improving margins. Happier customers. Now is the time to start the real digital transformation. A revolution is needed.

Many organizations are working on process optimization. They are looking at eliminating human interaction by means of digitization, integration and robotization. Organizations working LEAN will go a bit further and will continuously try to improve the efficiency of processes. Mostly, this is all based on current revenue streams and business models. Also, it is assumed that the organization has a relevant function and will keep that function in the ecosystem.

What’s happening around us?

Every organization is part of an ecosystem. Ecosystems that become more and more digital. By exchanging information digitally between parties in such an ecosystem, these ecosystems will become more and more efficient. First we used EDI and today, in modern application landscapes we use APIs more and more. APIs that are becoming smarter and smarter and can find eachother more or less automagically. APIs that are a direct interface to an existing application. An application that most of the times is a system of record. Not a very “interesting” application most of the times, only capable of handling basic administrative needs such as handling orders and ship notices.

The middleman in crisis

There are only a few systems of record that really deliver true value add for an organization. The real value add of an organization usually is in the uniqe position in an ecosystem and the processes it delivers by means of a combination of all kinds of software systems, on-premises and in the cloud. But what happens when your combination of services suddenly is not relevant anymore? Because ecosystems have found a way to solve the isssues in another, much better way? Your organization actually doesn’t have a function anymore? Because you are actually the middleman? Crisis!

We are seeing everyday that new ways of working together in ecosystems evolve rapidly. The most important unique selling point of blockchain is to cut out the middleman. This is possible because the system enables collaboration without having trust in place. You don’t have to trust eachother. The trust has been digitized. By means of smart contracts. Look around you. Blockchains that enable peer-to-peer lending, that make wholesale and banks obsolete, that replace marketplaces and trading systems, etc. etc.

Blockchain is a revolution

Blockchain is the El Niño of the digital ecosystems; it changes traditional collaborations between companies and disturbs the balance. We will probably see an Al Gore like person in the near future who will warn us of the dangers. But, just like in nature, it will take a while but a new balance will evolve. After which the landscape will be thoroughly changed. There will be many victims, but sometimes that is needed to make the next step, as a whole.

If your organization will become a victim depends on how much value add you deliver. Oftentimes this depends on the need for physical assets or infrastructure. For example: A supplier of green energy who does not have assets but only handles the trade and contracts can be replaced by a blockchain. The supplier that also owns the wind- and solar farms and the infrastructure for homes and factories will be much more difficult to replace. Until the time that you’ll be able to generate 100% of your own energy need. But if energy surplus needs to go back into the net, you’ll be needing infra again. Uber can easily be replaced by a blockchain. Uber doesn’t have assets. And really no value add. Blockchain is a revolution. In every boardroom this should be on the agenda permanently. Make sure you understand the technology and its impact. Don’t be an American member of congres who doesn’t understand Mark Zuckerberg’s business model and technology. Work out a solution and really start innovating!

Cheers, Gijs

 

Serverless bitten in the ass by relational databases

Thomas Erl wrote this wonderful book on the SOA’s 8 principles of service design quite a while back. Together with Hophe and Woolf’s Enterprise Integration Patterns that’s basically almost all you need to know by heart, when starting out as a rookie (integration) developer.

No matter what software architecture used, the things learned from these books should always be applied. So, also in this new serverless era, the same applies.

But, the hardest part of every software architecture is the data architecture. You can fantastically modularize functionalities and integratie them at will, but if the data behind the modules is one big blur of relational mess, good luck with your APIs, principles, patterns and Devops then!

Is SQL Server (or any other relational database) actually witholding us from creating true serverless, reusable and autonomous APIs? Will the Graph or DocumentDB or CosmosDB help us out? Is big data coming to the rescue? Schema on read? CQRS? Blockchain based transaction ledgers?

Probably 80% of data in any given relational database is not relational. So why use a SQL Server database then? Why shoot ourselves in the foot right at the start?

If we want to become true serverless, not only do we need to adhere to the principles of service design and integration patterns, but also need to rethink our data architecture.

Eventual consistency looks like a nice concept, but in a real-time world quite hard to implement and make successful. But maybe it’s our only option? In any case, the use of relational data storage will in my opinion decline quickly. SQL Server will *have* to morph into something new, with less emphasis on the relational aspect. I know Microsoft have already started that transition.

Getting the right architecture guidance on serverless in combination with data storage is going to be crucial in the coming time.

Cheers, Gijs

 

The rise of the Ethereum blockchain frameworks

Lately I’ve done quite some research on blockchain. I’ve been involved in a number of inspiration sessions for our customers, trying to come up with good use cases for blockchain in their respective industries. We’re in the process of defining and executing some exciting PoCs (proof of concepts) right now, mainly in the logistics vertical.

The Ethereum blockchain seems to be(come) the dominant platform for all kinds of initiatives. Ethereum is also doing quite well from a token market value point of view at the moment and that’s not hard to understand. It’s the goto platform for anything that has to do with smart contracts. A lot of current ICOs (initial coin offerings) run their technologies on the Ethereum blockchain. Some of them are good and probably have a bright future, some of them are hyped but basically hot air, and some of them are right out shady and probably scams. But hey, a new crypto sucker is born every day as Microsoft’s blockchain principal architect Marley Gray said during a keynote on a blockchain conference.

On the Microsoft Azure platform, it’s quite easy to setup an Ethereum blockchain. With the CoCo framework, Microsoft has built exciting preview stuff that can run on multiple blockchain platforms. Check out the paper here.

For me it’s clear that the blockchain technology itself is not the interesting part. Of course having immutable records and a consensus model to cut out the middle man is *very* important, but the blockchain itself will become mainstream like any other database technology, like SQL or NoSQL. What makes it worthwhile is the concept of smart contracts. And that’s what the Ethereum blockchain is quite good at. It is however quite hard to develop and test smart contracts. I foresee that in the short term, lots of startups will come up with smart things around smart contracts.

I’ve bumped into two of them that are worthwhile mentioning. Also because they are both legitimate and did their ICO’s in North America:

  1. Blockmason. The have developed the Credit Protocol on top of Ethereum, which takes care of a very badly needed smart contract for handling credit (on which this world turns), including the automatic settling of it between parties. They have developed this technology before they did their ICO. And they are SEC compliant, which is a first in crypto land. They have interesting partnerships, like the one with Coral Health who are doing a pilot with their technology on settling payments between doctors, patients and insurance companies. Without the need for a third party. Very interesting technology, for which they have applied for patents. I think lots of initiatives will use their technology to implement similar scenarios. Their token is named BCPT. Checkout Blockmason.io for full details.
  2. Etherparty. They have created the technology to make the development of smart contracts easier. Basically they do for smart contracts what WIX did for websites. Without any programming knowledge you can develop smart contracts that run on any compatible blockchain, but the most used one is obviously Ethereum. I foresee that they will come up with lots of out-of-the-box templates for smart contacts making the implementation of blockchain initiatives a lot quicker. Their token is named FUEL. Checkout Etherparty.com for full details.

So, just like we had frameworks on top of SQL databases and integration software, we’re now seeing the rise of smart frameworks and templates on top of blockchain. We’re definitely coming out of the blockchain stoneage. Exciting times!

Cheers, Gijs

BizTalk open source: a win win?

Yesterday, Microsoft announced that its on-premises integration middleware product BizTalk Server will become partly open source. First step is that all the 10K+ schemas (mostly B2B, EDI) have been released and are now available on Github.

Next step will be to make it possible for the community to contribute to the adapters, pipelines and tools.

My take on all this, is that it has the potential to become a win win for both Microsoft, partners and customers, provided that a number of things are executed well. Let me try to explain how:

  1. Microsoft BizTalk Server is rapidly turning into the on-premises LOB proxy (or gateway) that makes it possible to bridge legacy on-premises applications to the Azure iPaaS (mainly Logic Apps and API Management, plus Service Bus, Event Grid, etc.). This is how Microsoft IT has positioned it during Integrate 2017 in London. Bottom line in this (great!) architecture: BizTalk = legacy gateway and iPaaS = all the logic and modern integrations.
  2. Becoming (partly) open source, means that the community can contribute to the quality and number of schemas, adapters, pipelines and tools. This makes the role of BizTalk as an on-prem LOB proxy even more relevant, enabling even more legacy applications to bridge the gap to the public cloud. BizTalk basically has the potential to become an even greater on-ramp to the public Azure cloud.
  3.  Microsoft will remain focused on making sure the core BizTalk engine remains relevant and can run on the latest versions of their own platform (Windows, SQL Server, Visual Studio, .Net) and provides a terrific bridge to the public Azure cloud. This includes the non-functionals like end-to-end hybrid monitoring and management.
  4. The community has to be supported and made enthusiastic about contributing to the, what we can basically call “on-premises LOB adapters”. This is going to be the hard part of this open source endeavor, in my opinion. But, as we have seen in the past with ISVs leveraging the popularity of BizTalk to position and sell their adapters and basically “using” BizTalk to become more successful themselves, open sourcing the adapters will potentially have the same impact. But this time it’s not about leveraging BizTalk, but leveraging the hybrid integration stack. Time will tell. In the meantime, Microsoft can stay focused on the core and the bridge to the public cloud and in the meantime probably can transfer a couple of engineers to the iPaaS teams.

My $.02 only.

Cheers, Gijs