API Management

BizTalk open source: a win win?

gindafield

Yesterday, Microsoft announced that its on-premises integration middleware product BizTalk Server will become partly open source. First step is that all the 10K+ schemas (mostly B2B, EDI) have been released and are now available on Github.

Next step will be to make it possible for the community to contribute to the adapters, pipelines and tools.

My take on all this, is that it has the potential to become a win win for both Microsoft, partners and customers, provided that a number of things are executed well. Let me try to explain how:

  1. Microsoft BizTalk Server is rapidly turning into the on-premises LOB proxy (or gateway) that makes it possible to bridge legacy on-premises applications to the Azure iPaaS (mainly Logic Apps and API Management, plus Service Bus, Event Grid, etc.). This is how Microsoft IT has positioned it during Integrate 2017 in London. Bottom line in this (great!) architecture: BizTalk = legacy gateway and iPaaS = all the logic and modern integrations.
  2. Becoming (partly) open source, means that the community can contribute to the quality and number of schemas, adapters, pipelines and tools. This makes the role of BizTalk as an on-prem LOB proxy even more relevant, enabling even more legacy applications to bridge the gap to the public cloud. BizTalk basically has the potential to become an even greater on-ramp to the public Azure cloud.
  3.  Microsoft will remain focused on making sure the core BizTalk engine remains relevant and can run on the latest versions of their own platform (Windows, SQL Server, Visual Studio, .Net) and provides a terrific bridge to the public Azure cloud. This includes the non-functionals like end-to-end hybrid monitoring and management.
  4. The community has to be supported and made enthusiastic about contributing to the, what we can basically call “on-premises LOB adapters”. This is going to be the hard part of this open source endeavor, in my opinion. But, as we have seen in the past with ISVs leveraging the popularity of BizTalk to position and sell their adapters and basically “using” BizTalk to become more successful themselves, open sourcing the adapters will potentially have the same impact. But this time it’s not about leveraging BizTalk, but leveraging the hybrid integration stack. Time will tell. In the meantime, Microsoft can stay focused on the core and the bridge to the public cloud and in the meantime probably can transfer a couple of engineers to the iPaaS teams.

My $.02 only.

Cheers, Gijs

iPaaS, what else?

gindafieldLeave a comment

Integration is more important than ever, in all the systems of innovation we create.

With multi-disciplinary DevOps teams, we build and operate solutions that encompass all tiers; from UX to back-end and everything in between. Middleware specialists are juse part of the team and share their knowledge on integration specifics through cross team guilds. More and more, we build those solutions based on the service paradigm. If we don’t need endless scale (if we’re not doing B2C basically), we just build it SOA style using PaaS technology. If we do need enormous scale and continuous innovation (because we don’t want to loose the consumers that get bored easily), we’ll do it microservices style. But we also still have to cope with existing systems. They can be legacy, but they can also be SaaS. Especially in a SaaS before PaaS before IaaS environment, migrating commodity applications to their SaaS counterparts are quick wins. But, these SaaS solutions are still most of the times silo applications that we have to cope with. For these applications, on-prem legacy or more modern SaaS, we need to create wrappers, so they can expose their task- and entity services or API’s so we can compose them into greater solutions. As a whole, we’re building agile solutions with a mix of silo applications, (micro)services applications and a set of building blocks provided by the iPaaS and aPaaS platforms we use. And in the Microsoft world, we’re talking about Azure then.

In those environments, to me it does not make sense anymore to build new solutions with legacy middleware. To me, implementing a BizTalk Server on-premises (or in IaaS for that matter) just does not make sense anymore, in a greenfield environment. Of course, when you have already invested lots of time and money in an on-prem BizTalk based ESB, you should leverage that. And those environments will keep on running for years to come. But when it comes to new integrations, why use BizTalk? I can really only come up with 1 reason, and that is: “I really don’t want my integration middleware to be running in the cloud” (for whatever reason). But that decision does not have anything to do with technical capabilities.

Let me try and clarify that.

First of all, what we often hear is that it doesn’t make sense to use cloud integration middleware (iPaaS) if most of your systems run on-prem. Uh, why not? With an On-prem Data Gateway and an ExpressRoute connection, latency is not an issue. And talking about latency, the biggest latency in any integration built with BizTalk Server is the freakin’ MessageBox hops!

Second, integrating SaaS applications is not something else (from a location point of view) than integrating on-prem applications. Most SaaS applications don’t run on Azure. Even Office 365 doesn’t run on Azure. So, what is “the cloud” anyway? From an integration perspective, integrating Salesforce from within Logic Apps is the same as integrating an on-prem SAP system when it comes to location dependencies or preferences!

Third, legacy integration is not really that much easier with “out-of-the-box adapters”. It’s fairly easy to either use the On-prem Data Gateway or create a custom API app to talk to the legacy application. Most of the times, the number of interfaces is fairly limited. And, in a modern API based integration world, broad transaction scopes are not used anymore, so relying on idempotency and compensation logic is much more the norm. These type of interfaces are really easy to build as an API app.

Apart from these three very important reasons, the last important point I want to make is this: We have been building monolithic ESBs. It is very hard to deploy individual orchestrated task services because of all the dependencies in BizTalk Server. BizTalk Server has in fact become a monolith (or makes it very hard not to implement monolithic solutions) which in most cases is very hard to manage and monitor. iPaaS makes it much easier to deploy, manage and monitor integration solutions. With ARM, Application Insights, Azure Monitor and of course the Azure portal, it has become really manageable. And as soon as Service Map also makes it to PaaS, we’ve really evolved into a much more mature iPaaS than BizTalk Server has every been. I’m really a fan and don’t see any reason anymore why we should implement new integrations on-prem!

iPaaS what else

Cheers, Gijs

 

 

iPaaS should not become your Trojan Horse

gindafieldLeave a comment

I’m currently involved as a cloud architect at an insurance company, working on the hybrid cloud reference architecture. They are making a move from a hosted environment to a hybrid cloud. And cloud engineers are working on the detailed designs for networking, storage, subscriptions, identity & access and integration. Integration between the private cloud and the public cloud as well as enterprise integration and B2B integration.

The customer is currently using BizTalk Server for enterprise- and B2B integration. The hosted environment is managed by a 3rd party as well, which means that the customer’s IT department only focuses on the functional management of applications. This also applies to BizTalk Server. New services are provisioned by filling out a form, waiting a couple of days or weeks and getting access to the new service. All is well so far, except for time-to-market, cost and being able to make use of the latest and greatest technologies.

Hybrid cloud is needed to shorten the time-to-market and innovating business processes and at the same time decrease IT spend on infrastructure. The whole thing about (hybrid) cloud is the on-demand characteristics and also being able to move away from traditional solution development to agile solution development, deployment and operations. Because provisioning is so fast, solution development can become much faster as well.

This is not an easy endeavor I can tell you.

Apart from the organizational aspects (devops is a topic on its own; is your organization ready for it?), the constantly evolving cloud and at the same time the constantly applied pressure from the business (“hey, we now have shorter time-to-market; let’s see it!”) makes life not really easy for the IT folks. We’re working on the reference architecture, but in the meantime several projects are underway to deliver cloud based solutions. Some SaaS, some PaaS and some we-don’t-really-know-what-kind-of-aaS. Every day we run into issues with regard to security and governance aspects. You’re really going to store customer senstive data in a NoSQL database running on a public facing linux box? Let’s think a little bit more about that. Refactoring the (hybrid) cloud solutions that have been delivered so far is the first thing we have to do once the reference architecture and the detailed hybrid cloud designs are done. That, and making sure the organization can actually cope with hybrid cloud deployments, management and governance.

In the “good old hosting days”, security was designed, applied and governed. Processes were in place and everything worked fine. Today however, checking or unchecking a single box in the Azure portal can have quite the impact. Suddenly, data leaks are possible. And we thought all was covered. Not.

Infrastructure as Code (which not only applies to IaaS but also to PaaS), is a mandatory thing. Clicking in a portal should be avoided. Cloud resources have to be deployed and managed by means of code. Versioned code. Code that has been reviewed and tested. Since full DTAP environments are rapidly becoming a thing of the past, your DTAP process has to be in place pretty well in order to prevent screw-ups with production data.

Why is the title of this blog post about iPaaS (Azure Logic Apps, API Apps, Functions, API Management, Service Bus) and a Trojan Horse specifically? Integration is at the core of everything when it comes to hybrid cloud. All aspects of the hybrid cloud architecture are touched here. Before you know it, things are tried out and put in production with all the potential risks as a result. Let’s protect ourselves from that.

Four things are important here:

  1. Have a reference architecture for hybrid cloud. New rule apply here! Hybrid cloud is not private cloud. This should at the very least contain the architecture principles and high level requirements that apply to all hybrid cloud solutions deployed. Example principle “Passwords should be stored in a safe place”. High level requirement resulting from that “Passwords used in scripts and code should be stored in Azure Key Vault”.
  2. Document the Solution Building Blocks. Azure is box of legos. Make sure that you know how to use all those building blocks and make sure that everybody knows the rules about how to use them in which scenario. Solution Building Blocks are not evil, but necessary artifacts. When do you use SQL Database, Blob, DocumentDB? How does security relate to these choices?
  3. Hybrid cloud needs hybrid service management. Make sure your IT service management sees your private cloud, hosted cloud and public cloud as one hybrid cloud and is able to manage that.
  4. Design and apply the right level of governance. Architectures, principles, requirements and solution building blocks are completely worthless if you don’t make sure they are actually used (in the right way). Peer reviews. Signed-off solution designs. Random inspections. These are all necessary things that you should cater for in your devops teams.

And remember, these things apply to your organization, but also to your IaaS, PaaS and SaaS solution vendors.

Let’s keep the Trojans out of your hybrid cloud!

Cheers, Gijs

The business value of Microsoft Azure Logic Apps

gindafieldLeave a comment

I’ve written a paper on the business value of Microsoft Azure Logic Apps for Integration. Mainly useful for CIO’s and IT managers considering Azure PaaS services for their Integration needs.

It describes the components of Microsoft Azure, and then drills down into aPaaS and iPaaS to position Logic Apps, API Apps and API Management. Furthermore it describes common integration needs in complex application landscapes such as keeping data in sync, creating composite apps, involving supply and demand chains and integrating apps and portals.

Next it describes the real business value, so that you can explain it to your business stakeholders as well. This includes creating value add on top of commodity SaaS apps, leveraging investments in legacy applications (your Systems of Record), decreasing time-to-market, channel renewal, agility: basically digital transformation using Gartner’s pace-layer application model.

Lastly it describes the different integration themes that Logic Apps can help you with.

Enjoy!

Please share as you like.

Cheers, Gijs

Is NoESB a euphemism for “more code in apps”?

gindafieldLeave a comment

Yesterday I tweeted the title of this blog post. But at that moment I didn’t know yet that this was becoming the title of a blog post. And now it is :-).

I recommended the use of an ESB to a customer. Oh dear. Before I started working on that recommendation I organized workshops with all kinds of folks within that organization and thoroughly analyzed their application landscape, information flows, functional and non-functional requirements. The CIO had told me at the start that he was going to have the report reviewed by a well known research firm. And so happened.

The well known research firm came back with a couple of comments. None too alarming. Except for one: the advice I gave, to implement an ESB was “old fashioned” because ESBs are overly complex centralized solutions and therefore we should consider using API Management. Whaaaaaat?

First of all, let me explain that I have found out that when we talk about an ESB, quite some people automatically think that that is about on-premises software. That’s a wrong assumption. ESBs can run in the cloud as well. As a PaaS solution indeed.

Second, an ESB can also be used to integrate external apps and not only apps that are within your own “enterprise”.

Third, API Management is not an integration tool. With regard to integration capabilities, it’s merely a mediation tool. With strong virtualization, management and monitoring facilities, for both admins and developers. Developers that use your APIs to create new apps.

The well known research firm also said that NoESB is a new thing that should be considered.

I think that we are just moving the complexity elsewhere. And the bad thing is that we are moving it in places (mostly apps) that are developed by people who don’t understand integration and everything that comes with it. People who have issues with understanding design patterns most of the times, let alone integration patterns. People who most of the times don’t have a clue about manageability and supportability. People who often say things like “I didn’t expect that many items to be in that list.” or “What do you reckon, he actually uses this app with mimimum rights. We haven’t tested that!” or “A 1000 should be enough.”. Famous last words.

Of course the ESB has caused lots of troubles in the past and it lots of times still does. But that’s not the fault of the ESB. That’s the fault of the people implementing the ESB, using the wrong design patterns. Or using ESB technologies that do not (fully) support design patterns. Or ESBs that are misued for stuff that should have been implemented with other technologies and just bridged using federation.

I’m a bit worried about the new cloud offerings currently becoming popular rapidly, which are still in beta or even alpha stage. Cloud services that are offered for integration purposes, which can easily be misused to create horrific solutions. Not adhering to any design pattern at all. And not guiding the powerusers (DevOps) in applying them either. Do DevOps even know about design patterns? (sorry if I offend anyone here).

I’m afraid that in the near term, lots of new distributed apps will be created with brand new, alpha or beta stage technologies by people who are neither developers, nor operations folks. People who know little about architecture and patterns and who will create fat apps with lots of composite logic. And integrate with NoESB. Disaster is coming soon. Who’ll have to clean up that mess? Who you gonna call? FatApp Busters! 🙂

Cheers and happy hacking, Gijs