Atomic Integration

Atomic Integration, I like the term. And I like the concept. Of course it has disadvantages, but don’t the advantages outweigh these? Let’s explore.

When the Integration Monday session “The fall of the BizTalk architect” (by Mikael Sand) was announced I was immediately triggered. It was the combination of the title and the presenter (Mikael often rocks) that made me create a note to self to watch it later on. And finally I did. And it triggered me to write this blog post.

For years we have been telling our customers that spaghetti is bad and lasagna is good. And we’ve also been telling them that re-use is the holy grail. And that by putting a service bus in the middle that everything becomes more manageable, integrations have shorter time to market and the whole application landscape becomes more reliable.

But at the same time, we see that re-use is very hard to accomplish, there are so many dependencies between solutions and that realizing this in an agile manner is a nightmare if not managed meticulously. Especially if we need to deliver business value quickly, talking about createing stuff for later re-use and having depencies on other teams is a hard sell to the product owner and thus the business.

Another thing that is really hard with any integration archtitecture and the accompanying middleware with its frameworks that have been built on top of the actual middleware (and thus becoming a part of that middleware) is ownership. And that is a two-headed beast. First of all, ownership of the frameworks that run on top of the middleware, and second the ownership of the actual integrations.

The first one is already a hard nut to crack. Who pays for maintaining these frameworks? Does it get financed by projects? Very hard to sell. Does it have a separate (business) owner and funds? Never seen that before and it probably wouldn’t work because the guy that pays most gets his way and it doesn’t necessarily mean that the framework will be the best usable framework of all times.

The second one is even harder to manage. Who is typically the owner of an integration. The subscriber and thus the receiver of information? Or is it the sender a.k.a. the publisher of the information? Who pays for it? Is that the owner? And will he be managing it? And what happens if the ingration makes use of all kinds of other, re-usable custom features that get changed over time and in which the actual owner of the ingration is not interested at all?

Why indeed not do more copy-and-paste instead of inherit or re-use? The owner of the specific integration is completely responsible for it and can change, fork and version whatever he likes. And as Mikael says in his presentation: TFS or Visual Studio Online is great for finding out who uses certain code and inform them if a bug has been solved in some copied code (segment). And of course we still design and build integrations according to the well known integration patterns that have become our best friends. Only, we don’t worry that much about optimization anymore, because the platform will take care of that. Just like we had to get used to garbage collectors and developers not knowing what memfree() actually means, we need to get used to computing power and cheap storage galoring and therefore don’t need to bother about redundancy in certain steps in an integration anymore.

With the arrival of cloud computing (more specifically PaaS) and big data, I think we now get into an era when this actually is becoming possible and at the same time manageable. The PaaS offerings by Microsoft, specifically Azure App Service, are becoming an interesting environment quickly. Combined with big data (which to me in this scenario is: Just save a much information about integration runs as possible, because we have cheap storage anyway and we’ll see what we’ll do with this data later), runtime insight, correlation and debugging capabilities are a breeze. Runtime governance: Check. We don’t need frameworks anymore and thus we don’t need an owner anymore.

Azure App Service, in combination with the big data facilities and the analytics services are all we need. And it is a perfect fit for Atomic Integration.

Cheers, Gijs

Is NoESB a euphemism for “more code in apps”?

Yesterday I tweeted the title of this blog post. But at that moment I didn’t know yet that this was becoming the title of a blog post. And now it is :-).

I recommended the use of an ESB to a customer. Oh dear. Before I started working on that recommendation I organized workshops with all kinds of folks within that organization and thoroughly analyzed their application landscape, information flows, functional and non-functional requirements. The CIO had told me at the start that he was going to have the report reviewed by a well known research firm. And so happened.

The well known research firm came back with a couple of comments. None too alarming. Except for one: the advice I gave, to implement an ESB was “old fashioned” because ESBs are overly complex centralized solutions and therefore we should consider using API Management. Whaaaaaat?

First of all, let me explain that I have found out that when we talk about an ESB, quite some people automatically think that that is about on-premises software. That’s a wrong assumption. ESBs can run in the cloud as well. As a PaaS solution indeed.

Second, an ESB can also be used to integrate external apps and not only apps that are within your own “enterprise”.

Third, API Management is not an integration tool. With regard to integration capabilities, it’s merely a mediation tool. With strong virtualization, management and monitoring facilities, for both admins and developers. Developers that use your APIs to create new apps.

The well known research firm also said that NoESB is a new thing that should be considered.

I think that we are just moving the complexity elsewhere. And the bad thing is that we are moving it in places (mostly apps) that are developed by people who don’t understand integration and everything that comes with it. People who have issues with understanding design patterns most of the times, let alone integration patterns. People who most of the times don’t have a clue about manageability and supportability. People who often say things like “I didn’t expect that many items to be in that list.” or “What do you reckon, he actually uses this app with mimimum rights. We haven’t tested that!” or “A 1000 should be enough.”. Famous last words.

Of course the ESB has caused lots of troubles in the past and it lots of times still does. But that’s not the fault of the ESB. That’s the fault of the people implementing the ESB, using the wrong design patterns. Or using ESB technologies that do not (fully) support design patterns. Or ESBs that are misued for stuff that should have been implemented with other technologies and just bridged using federation.

I’m a bit worried about the new cloud offerings currently becoming popular rapidly, which are still in beta or even alpha stage. Cloud services that are offered for integration purposes, which can easily be misused to create horrific solutions. Not adhering to any design pattern at all. And not guiding the powerusers (DevOps) in applying them either. Do DevOps even know about design patterns? (sorry if I offend anyone here).

I’m afraid that in the near term, lots of new distributed apps will be created with brand new, alpha or beta stage technologies by people who are neither developers, nor operations folks. People who know little about architecture and patterns and who will create fat apps with lots of composite logic. And integrate with NoESB. Disaster is coming soon. Who’ll have to clean up that mess? Who you gonna call? FatApp Busters! 🙂

Cheers and happy hacking, Gijs