iPaaS, what else?

Integration is more important than ever, in all the systems of innovation we create.

With multi-disciplinary DevOps teams, we build and operate solutions that encompass all tiers; from UX to back-end and everything in between. Middleware specialists are juse part of the team and share their knowledge on integration specifics through cross team guilds. More and more, we build those solutions based on the service paradigm. If we don’t need endless scale (if we’re not doing B2C basically), we just build it SOA style using PaaS technology. If we do need enormous scale and continuous innovation (because we don’t want to loose the consumers that get bored easily), we’ll do it microservices style. But we also still have to cope with existing systems. They can be legacy, but they can also be SaaS. Especially in a SaaS before PaaS before IaaS environment, migrating commodity applications to their SaaS counterparts are quick wins. But, these SaaS solutions are still most of the times silo applications that we have to cope with. For these applications, on-prem legacy or more modern SaaS, we need to create wrappers, so they can expose their task- and entity services or API’s so we can compose them into greater solutions. As a whole, we’re building agile solutions with a mix of silo applications, (micro)services applications and a set of building blocks provided by the iPaaS and aPaaS platforms we use. And in the Microsoft world, we’re talking about Azure then.

In those environments, to me it does not make sense anymore to build new solutions with legacy middleware. To me, implementing a BizTalk Server on-premises (or in IaaS for that matter) just does not make sense anymore, in a greenfield environment. Of course, when you have already invested lots of time and money in an on-prem BizTalk based ESB, you should leverage that. And those environments will keep on running for years to come. But when it comes to new integrations, why use BizTalk? I can really only come up with 1 reason, and that is: “I really don’t want my integration middleware to be running in the cloud” (for whatever reason). But that decision does not have anything to do with technical capabilities.

Let me try and clarify that.

First of all, what we often hear is that it doesn’t make sense to use cloud integration middleware (iPaaS) if most of your systems run on-prem. Uh, why not? With an On-prem Data Gateway and an ExpressRoute connection, latency is not an issue. And talking about latency, the biggest latency in any integration built with BizTalk Server is the freakin’ MessageBox hops!

Second, integrating SaaS applications is not something else (from a location point of view) than integrating on-prem applications. Most SaaS applications don’t run on Azure. Even Office 365 doesn’t run on Azure. So, what is “the cloud” anyway? From an integration perspective, integrating Salesforce from within Logic Apps is the same as integrating an on-prem SAP system when it comes to location dependencies or preferences!

Third, legacy integration is not really that much easier with “out-of-the-box adapters”. It’s fairly easy to either use the On-prem Data Gateway or create a custom API app to talk to the legacy application. Most of the times, the number of interfaces is fairly limited. And, in a modern API based integration world, broad transaction scopes are not used anymore, so relying on idempotency and compensation logic is much more the norm. These type of interfaces are really easy to build as an API app.

Apart from these three very important reasons, the last important point I want to make is this: We have been building monolithic ESBs. It is very hard to deploy individual orchestrated task services because of all the dependencies in BizTalk Server. BizTalk Server has in fact become a monolith (or makes it very hard not to implement monolithic solutions) which in most cases is very hard to manage and monitor. iPaaS makes it much easier to deploy, manage and monitor integration solutions. With ARM, Application Insights, Azure Monitor and of course the Azure portal, it has become really manageable. And as soon as Service Map also makes it to PaaS, we’ve really evolved into a much more mature iPaaS than BizTalk Server has every been. I’m really a fan and don’t see any reason anymore why we should implement new integrations on-prem!

iPaaS what else

Cheers, Gijs

 

 

On the desperate need for (micro)services governance

Am I becoming an old integration cynic (or a Happy Camper as @mikaelsand so sarcastically called me a couple of weeks ago during an #IntegrationMonday)? I don’t think so, but correct me if I’m wrong, please!

My current view on the latest Microservices platform craze is the following: Old wine in new bottles. Yeah, yeah, yeah, we can now run on the latest SQL Server (but not SQL Azure and not AlwaysOn) and run things in different containers. And we have the 3rd or 4th incarnation of a business rules composer that does exactly the same but in a new frame. And yet another way of composing services and building transformations. And we store and exchange things using JSON. Great! 😉 But to me, this all still falls in the category “platform alignment” or “platform modernization”. This is not innovation. This is not adding great features that help our customers solve real life issues in a much better way. This is like adding airbags and anti-lock brakes (that any car has today), but not building a car that’s 60% software and really innovative (like Tesla). This is just making sure that you use the latest technologies to be able to run your basic functions, but even these are not providing the bare minimum.

We are still not fixing what we really crave for: end-to-end design- and runtime governance. And better productivity. But that’s a couple more bridges too far right now. I know that governance is a boring word. But I also know that my customers desperately need it. How else can you manage and monitor your more and more complex integration architecture? When only running on-prem integration, it’s relatively easy. But we lack the real tools here already. We need 3rd party tools to manage and monitor our environment. When adding SaaS, Mobile Apps and IoT to the mix, it becomes more and more complex. Let alone when adding Microservices.

My customers want the following. Today!

  1. An easy way to register and find services
  2. An easy way to compose services (aligned with business processes)
  3. A proper way to handle configuration management and deploy & manage those services (and really understand what’s deployed)
  4. Real end-to-end insight in composed services, including root cause analysis

For enterprises that have a “Microsoft unless” policy, today it’s impossible to use only Microsoft technology for integration purposes. They need 3rd party tools for management and monitoring. Tools that only solve part of the design-time and part of the run-time governance issues we have. Tools that can’t be used end-to-end. Tools that are built by companies we don’t know will still exist in 2 years time. Tools that are not Microsoft tools. Tools of which we even need more than one to provide for end-to-end, fully manageable services.

When, for example, my on-prem BizTalk Server exposes a back-end SOAP or other legacy service as an externally faced REST service (this is called mediation) we can more or less manage and monitor that. In a quite limited way still, because there is no real repository that can register my SOAP and REST services for solution developers to be able to find and start using them. And BAM needs quite some config. But, the major problem is that as soon as the REST service is consumed by API Management which exposes it to Apps and Portals and other consumers, we have no way to find out what happens anymore; no end-to-end tracking & tracing  is possible. Unless we built it ourselves. By coding. Yes, you heard me: Coding; a 20th century concept of building software solutions. So 5 minutes ago!

What we need is platform development (by any vendor by the way, including IBM, SAP, Google, etc.) that takes supportability first seriously. This means: first think about how your customers are going to be able to manage & monitor this when going in production before you start building features. Really cater for management & monitoring. Not build software that cannot or only poorly be managed and monitored. And once we have that, we’ll be more than happy to get better design and development productivity as well. But for now, being able to get rid of the bulk of the cost of any integration environment, namely supporting it on a day-to-day basis (the part of the iceberg that’s under water), would be a good starter indeed.

Thanks for listening. Please spread the word. 🙂

Cheers, Gijs

p.s. I think Microsoft are still way ahead of most other vendors. Pure-play integration vendors don’t have a bright future. It’s a platform game.

On Immutability, Microservices and SOA

Triggered by a tweet by @rseroter (thanks for that, Richard), I decided to write this little post.

One of the major challenges with the implementation of service oriented architecture (SOA) and therefore Microservices, is the underlying data architecture. How can loosely coupled services, that cannot keep state, make sure that the business data remains consistent, reliable and trustworthy?

In every implementation of a (enterprise) service bus or other form of integration middleware, having to cope with for example back-end applications and services that are not idempotent is a nightmare.

What if we could make this all a “thing of the past”, and run our services on an immutable infrastructure? No more updates to existing records. No more keeping track of separate audit information in logs, to record who did what, when and why. The infrastructure just records every bit of information as new information, thereby “elaborating” on the current status. That way, auditability is built in. That way, restartability is not an issue anymore. That way, idempotency is not an issue any more.

Pat Helland writes in his paper: Many kinds of computing are “Append-Only”. Observations are recorded forever (or for a long time). Derived results are calculated on demand (or periodically pre-calculated). And: Normalization is for sissies!

Is immutable infrastructure the holy grail of service oriented architecture and microservices?