Frankenstein IT solutions

At the various organizations that I’ve been involved with regarding IT strategy and architecture it always strikes me that they so often “bury” themselves in technology. What started out as a nice architecture has derailed into a Frankenstein solution, as I’d like to call it.

Often, this happens due to a combination of the usage of monolithic (and bought licences of) software products, the phenomenon “sunk cost” and architects who, as a result of that, as a carpenter (have to) treat everything as a nail.

Examples

What do I exactly mean by Frankenstein solutions? A couple of examples:

  • Tailormade software is needed to make the ERP system fit the company specific business processes. It is decided to build the tailormade solution into the ERP system, in such a way that it’s completely interwoven. With, as a result, updates of the kernel break the system. They should have chosen a loosely coupled solution, but that wasn’t possible because the IT landscape didn’t cater for that, didn’t have the right tools.
  • The integration middleware offers features to build business logic and store data for the longer term. This is used to fix issues in the back-end systems related to data quality and the lack of certain data.  At the same time the mistake is made to make the middleware team responsible for the solution created.
  • A NoSQL data platform has been acquired and solutions are built on that platform to connect relational data sources and harmonize data. The resulting solutions are very complex and nobody understands them anymore and authorization and privacy issues arise as soon as relations in the source system change.

In all three examples, the tools have been used as a hammer. But the solution is oftentimes not a nail. Sometimes you need a screw, glue or a click-system. But, because we can only hold a hammer, we try to make it fit anyway. This, while you have learned as a kid that you cannot fit a triangle in a square hole in your block box. Unless you of course try to fix it in a creative way, by turning the block and making it fit. Or hit it hard with a hammer.

How can we escape this?

With the arrival of PaaS (cloud platform-as-a-service) services, it has become possible to choose the right service for the right challenge. What seemed to look like an unsorted box of legos of which many users were scared because of the complexity that such a large number of different blocks could bring with it, this turned out to be a blessing in disguise. Together with the arrival of better governance and monitoring tools in the cloud, the solutions you build with these PaaS services become more manageable.

It will make the architect’s life also easier. Instead of making choices based on already acquired, expensive licences of product x or y, the architects can now easily choose new PaaS services if the challenge asks for it. It has also become easier to “fail fast”, because if the solution doesn’t seem to work as expected in a proof-of-concept situation, you can easily turn the cloud service off again.

It does however require a lot of knowledge. The architects not only need to know everything about the various cloud services, but also about the applied architecture patterns. And the operational costs of the services. Luckily, the cloud suppliers have become more an more clear on this. Every supplier provides “open source” best practices and patterns that you can easily use. And a calculator to estimate the operational costs.

Finally: It’s always difficult for any organization to change an already chosen direction. More so, if lots of money has already been invested. But it is very important to also think about the maintenance cost. And the job satisfaction of the people who have to keep the solutions afloat is also very important. It too often is the case that 80% of your IT budget goes to maintenance and thus only 20% to real innovation. And as a result, many IT people leave the company dissatisfied. So, make sure that the architects in your organization have more than a hammer in their toolbox!

The end of monolithic platforms

With the rise of functionally rich PaaS services delivered by the so called cloud mega vendors such as Microsoft, quite a lot has changed. First of all, it has resulted in confusion in the market, but right now we can see that organizations are really starting to understand the philosophy and embracing it.

Think in services

Especially in the early days it was difficult to understand for both consumers as suppliers of cloud services. For data- and integration products it was the hardest to grasp. In yesteryears, it was quite easy for organizations with a Microsoft-unless policy: if you wanted a solution for data you got SQL Server and if you wanted a solution for integration you got BizTalk Server. Very easy to compare features between those products provided by Microsoft and other vendors such as Oracle or IBM. Work through a checklist, score and choose.

Nowadays, these kind of data- and integration functionalities are no longer delivered by products but by a set of cloud services. This has at first resulted in great confusion. What do we compare with what? Especially when comparing a set of cloud services on the one hand and a monolithic platform on the other.

Say goodbye to platform “products”

Even for Microsoft this was difficult. Because on the one hand there is a very powerful message because Microsoft Azure provides all building blocks to build a mature integration platform, but on the other hand Microsoft has to compete with some pure-play vendors that provide monolithic productds. Products that are very good at one thing. And in which the various components are 1:1 dependent on eachother. Microsoft also provided one of those products: BizTalk Server. A fantastic integration product, but monolothical and not very service oriented. The processes in such a product are designed beforehand and therefore “baked-in”. And therefore not very flexible.

However, we saw that competing with the pure-plays was (and still is) not easy. At one moment in 2018, Microsoft chose to rebrand and bundle the services in Azure that are needed to build an integration layer to “Azure Integration Services”. The same happened to “IoT Suite” for example. Of course this was done to make it easier for potential customers to make product comparisons. But personally, I thought this was an admission of weakness by Microsoft. It basically does not make sense at all. It is simply the power of the platform to provide a set of loosely coupled services with which you can build very flexible integration layers. And besides that, you get the whole Azure platform with it that you can make part of your end-to-end processes, potentially being integration-, data-, app- or collaboration oriented solutions.

The same goes for data solutions. These days we have relational data stores, table stores, document stores, blob stores, data lakes, etc. It has been tried to put that all in SQL Server in that past. In Microsoft Azure, these are all “loose” services that all run in their own scalable containers and that can be connected to eachother through (among others) Azure Data Factory, to create a data platform that suits your organization best. And on top of which you can deliver all kinds of analysis services such as Databricks or PowerBI in order to create information from the data.

Non-functional aspects

What is very interesting, is that Microsoft has not only provided all these great features as services, but at the same time has invested a lot in architecture guidance and non-functional aspects. Where you had a silo application such as BizTalk Server and SQL Server (and comparable products from other vendors) before, in which all the management and monitoring tools where built-in, you can see right now that on top of all cloud services you have tooling for end-to-end management and monitoring. For example, for deploying workloads by means of infrastructure-as-code and to gain insight in the relationships between services. Right now we can deal with loosley coupled platform solutions that at the same time are very manageable and quite easy to monitor. That’s real progress! I predict that monolithic integration-, data-, collaboration- and app platforms will die a slow death. Long live PaaS!

Cheers, Gijs

Serverless. Basta!

I was once a system programmer in a Unix world. Brilliant OS. The first version I got to know really well (and I mean, deep down really really well) was System V. Later on, I worked with (Open)VMS and also spent quite some time porting stuff to Linux (Redhat). I mean, communication protocols, compilers, etc. The really hard stuff. And fun it was. Getting stuff to work was really very fulfilling. Especially in a time where you had to build your own “platforms”, like 4GL, RDBMS and integration middleware in order to provide your internal consumers with better solution building productivity. Building all this stuff was awesome!

And then I was introduced to Windows. Lipstick on a pig it was. And in quite some cases still is. We were serving 8 developers on a 486 with Redhat Linux and with Windows 3.11 on the same machine we only got to serve 1. Aaaargh. But hey, developing really visual stuff was a nice change as well.

But the best thing today is: we don’t have to care anymore, because we’ve got PaaS now! Who cares about servers? We just want to run our workloads serverless. Let the hardcore developers build this cool platform stuff (and make it very very easy for us), that we ordinary folks can just use to deploy everyday workloads.

But, lately I got introduced to this relatively new phenomenon of containers. I think that’s a step back. At least, for us people who just want to deploy common workloads. I understand that for architects and developers working for large scale B2C companies (Facebook, Google, Amazon, etc.), containers and K8S and stuff like that is great. But for the average company, it’s overkill. And overly complex. And back to virtual machines, but just a bit smaller and more contained. And somewhat easier to deploy.

But, we don’t need that (in our platform). We just want less complexity. And more less. Serverless. Basta!

Just my 0.02 Q* of course.

Cheers, Gijs

*Want some for free as well? Just register here through my personal invite link.

The can-you-do-that-guys

Here at the #Integrate2017 event in London (26-28th June), I loved the keynote today by Jim Harrer (Microsoft Pro Integration Group PM). During the last 5 minutes of his presentation, he nailed it!

As I wrote before in another blog post (“Integration is just one of the skills needed”), iPaaS is not about just integration, it’s about creating business apps. Integration is part of the multi-disciplinary teams that build solutions. And these solutions are more and more built by using the 80+ Azure PaaS building blocks (see my most recent blog post “iPaaS, what else?”). These building blocks are not just about moving information from one location to another (including from and to hundreds of SaaS apps), but more and more also include Big Data and AI (artificial intelligence) capabilities. Making it possible to integrate things like cognitive services, machine learning, etc. Creating real end-to-end business apps that the business wants, now! With technology that until a year ago, was just not available (at a reasonable cost) to smaller sized companies.

Being an integration guy, you do have a special role in the teams. You are the guy that connects the building blocks and makes sure that the business app actually is resilient. And that you can properly monitor and manage the solution.
The time-to-market for these apps is phenomenal. Instead of weeks or months, you can create value in hours or days! And the speed at which Microsoft is adding not only the functionalities but, more importantly, the non-functional features is amazing. They build the platform, we build the solutions!

During the conference we’ll of course learn about new features that have just been released or will be released in the very near future. But to me, that is not the most important part anymore.

The IT world to me is clear now: integration folks have to become “the can you do that guys“.

We need to show our customers what is actually possible by assembling all these great building blocks into very valuable business solutions. Just do a PoC or Pilot and show the customer you’re at what you can build in such a short time. Sooner or later, your customer will also be saying “iPaaS, what else!“. Our customers are all becoming software companies. We can help them do just that!

Cheers, Gijs