Azure

Microsoft Azure.

Anarchytecture

gindafieldLeave a comment

Cloud. A euphemism for the Internet. Internet. A virtual community without any governance. Anarchy! How can we still make sure that this keeps on working?

For many people this may come as a surprise, but for most of us it won’t: The physical world has been divided in countries with each having their own government / governance in place and the Internet is one big global virtual world. The Internet gurus are all convinced that this should never be tampered with. The Internet is self-governing.

Of course there are some principles to which the Internet adheres and within which’ boundaries it further develops. Global agreement on for example domain architecture was necessary. The right choices that have been made in the beginning turned out to be very valuable. Once in a while we do come to a grinding halt however. Think about the limitations of IP numbering. For once and for all let’s not forget: when you, as a developer, think “uhm, that should be big enough”, just multiply it with at least a 1000 from now on. Just as a side note. 🙂

Architecture has been invented to be able to build a foundation that can keep up with future developments on it. Otherwise, stuff implodes. But especially in IT, architecture means “being able to cope with change”. A good architecture can prove its value for years to come. Until the next possible paradigm shi(f)t.

So, is architecture needed? Yes. Can it work in an anarchistic environment like the Internet? For sure. At least, when you take into consideration that you can always be surprised by rebels who don’t care about your architecture and just spray your artwork with graffiti. Or build a terrace on your flat rooftop. By preparing for the worst, at acceptable cost, an architecture can sustain fine. But, like I said before, a next big invention in IT can for sure start shaking the foundations. For example, a bit that can be 1 and 0 at the same time is something we have to carefully think about before we start deploying solutions built on that paradigm massively.

I’m convinced that with the current methods of developing distributed solutions we are well on our way to build fine, architecturally sound applications that can be deployed in the anarchistic clouds. Microservices architecture is an excellent way to thrive in this chaos. Microservices architecture is anarchytecture! So, what are DevOps then, actually?

Cheers, Gijs

p.s. Thank you Skunk Anansie for the inspiration for this blog post and hence its title.

On the desperate need for (micro)services governance

gindafieldLeave a comment

Am I becoming an old integration cynic (or a Happy Camper as @mikaelsand so sarcastically called me a couple of weeks ago during an #IntegrationMonday)? I don’t think so, but correct me if I’m wrong, please!

My current view on the latest Microservices platform craze is the following: Old wine in new bottles. Yeah, yeah, yeah, we can now run on the latest SQL Server (but not SQL Azure and not AlwaysOn) and run things in different containers. And we have the 3rd or 4th incarnation of a business rules composer that does exactly the same but in a new frame. And yet another way of composing services and building transformations. And we store and exchange things using JSON. Great! 😉 But to me, this all still falls in the category “platform alignment” or “platform modernization”. This is not innovation. This is not adding great features that help our customers solve real life issues in a much better way. This is like adding airbags and anti-lock brakes (that any car has today), but not building a car that’s 60% software and really innovative (like Tesla). This is just making sure that you use the latest technologies to be able to run your basic functions, but even these are not providing the bare minimum.

We are still not fixing what we really crave for: end-to-end design- and runtime governance. And better productivity. But that’s a couple more bridges too far right now. I know that governance is a boring word. But I also know that my customers desperately need it. How else can you manage and monitor your more and more complex integration architecture? When only running on-prem integration, it’s relatively easy. But we lack the real tools here already. We need 3rd party tools to manage and monitor our environment. When adding SaaS, Mobile Apps and IoT to the mix, it becomes more and more complex. Let alone when adding Microservices.

My customers want the following. Today!

  1. An easy way to register and find services
  2. An easy way to compose services (aligned with business processes)
  3. A proper way to handle configuration management and deploy & manage those services (and really understand what’s deployed)
  4. Real end-to-end insight in composed services, including root cause analysis

For enterprises that have a “Microsoft unless” policy, today it’s impossible to use only Microsoft technology for integration purposes. They need 3rd party tools for management and monitoring. Tools that only solve part of the design-time and part of the run-time governance issues we have. Tools that can’t be used end-to-end. Tools that are built by companies we don’t know will still exist in 2 years time. Tools that are not Microsoft tools. Tools of which we even need more than one to provide for end-to-end, fully manageable services.

When, for example, my on-prem BizTalk Server exposes a back-end SOAP or other legacy service as an externally faced REST service (this is called mediation) we can more or less manage and monitor that. In a quite limited way still, because there is no real repository that can register my SOAP and REST services for solution developers to be able to find and start using them. And BAM needs quite some config. But, the major problem is that as soon as the REST service is consumed by API Management which exposes it to Apps and Portals and other consumers, we have no way to find out what happens anymore; no end-to-end tracking & tracing  is possible. Unless we built it ourselves. By coding. Yes, you heard me: Coding; a 20th century concept of building software solutions. So 5 minutes ago!

What we need is platform development (by any vendor by the way, including IBM, SAP, Google, etc.) that takes supportability first seriously. This means: first think about how your customers are going to be able to manage & monitor this when going in production before you start building features. Really cater for management & monitoring. Not build software that cannot or only poorly be managed and monitored. And once we have that, we’ll be more than happy to get better design and development productivity as well. But for now, being able to get rid of the bulk of the cost of any integration environment, namely supporting it on a day-to-day basis (the part of the iceberg that’s under water), would be a good starter indeed.

Thanks for listening. Please spread the word. 🙂

Cheers, Gijs

p.s. I think Microsoft are still way ahead of most other vendors. Pure-play integration vendors don’t have a bright future. It’s a platform game.

IoT as the big driver of Microservices

gindafieldLeave a comment

The Internet of Things (IoT) market will grow tremendously in the coming years. The quantity and diversity of devices that are connected to the cloud and that will spit out data will explode. The generated data will have to be analyzed and these analyses will be used to optimize business processes. New architectures will be needed to accommodate all this and Microservices architecture is probably the way to go and here to stay.

The big hype around IoT has just started and we can expect that the largest bulk of IT investments will be done in this area. Who is not interested in optimizing processes and increasing the margins on delivered products and services? Granular insight in what exactly happens in the complete production- or services chain and being able to predict as reliably as possible what will happen in the (near) future is worth a lot. We’ve got gold in our hands.

In an IoT architecture, the following components are important:

  • The Things; the devices that are connected to the cloud that, without human intervention, can generate (mostly) sensor data and which in some cases can handle instructions to take certain actions.
  • The (cloud) infrastructure and services to execute the following actions:
    • Receiving large amounts of data, generated by the Things.
    • Storing these data.
    • Analyzing these data in real-time, but also by means of predictive analysis.
    • Generating “business events” as a result of these analyses.
    • Orchestrating these business events by means of integration logic.
    • Potentially send back data (instructions) to the Things, so that they can take some kind of action.
  • The (cloud) applications that can handle the business events and can in their turn generate new business events.

The Things are most of the times small machines with their own processor, operating system and sensors and contain some (hard-wired) logic. They can be communicated with by means of lightweight APIs. They are in fact Microservices. Because they are completely autonomous, run in their own process space and deliver a small service. A Thing is the ultimate incarnation of a Microservice!

In order to be able to process the enormous amounts of data at high speed and efficiently, we need an architecture that supports lightweight mediation; capable of distributing the generated events to the right receivers with as little overhead as possible. Such a receiver can for example be a NoSQL database, but also a machine learning service. The NoSQL database can store events for later processing and distribute them asynchronously to other interested parties. This enables offline analysis. The machine learning service can, by means of pre-defined models, determine if certain actions should be taken based on the current event and previous related events. This will then result in a business event. Which in its turn will be published to the lightweight mediation layer which will take care of it. All of this is very time critical and has to happen as soon as possible. Because the next events to be processed are already arriving.

The generated business event will have to process further by the orchestration layer. This may involve multiple business applications and services. This can no longer be called lightweight mediation. We’re talking service bus or application integration middleware here. Time criticality is less important here. What is important in this layer though is guaranteed delivery and transactionality. Traceability is also of high importance in this layer. This orchestration layer is something we most of the times have in place already. Think of traditional integration middleware such as Microsoft BizTalk Server, Tibco or IBM WebSphere Message Broker (and of course many more). The orchestration layer can be hosted in the cloud or, much more still the case, on-premises.

Lightweight mediation all revolves around the following tasks:

  • Lightweight transformation (for example JSON to XML and vice versa).
  • Routing.
  • Applying (security) policies.
  • Lightweight service composition.

In my (humble, ahum) opinion, the lightweight meditation layer is not a layer where you handle (or even need) things like batching, heavy XSLT transformations and long running transactions. In this layer, we should only think about Microservices. Microservices that are indeed lightweight, single task entities that can be composed into solutions by means of lightweight composition.

I foresee a hybrid integration world that is here to stay. Not because EAI can only be hosted on-premises or because we want to run integration solutions in the cloud so desperately, but because cloud integration in an IoT world is about Microservices and lightweight mediation. And this realm needs to be coupled to (often on-premises) infrastructure by means of transactions and orchestrations. Hybrid integration, in my opinion, is about connecting a Microservices architecture to traditional integration middleware, no matter where the latter is hosted. The big challenge with this is to provide for end-to-end run- and design time governance. We’re certainly going to need that.

IoT is the big drivers for Microservices and integration specialists will have to get to know both integration realms and be able to apply the right (combination of) patterns by means of the right (hybrid) technologies. This is a great era to live in, for us integration folks!

Cheers, Gijs

BizTalk Server, Microservices & APIs

gindafield3 Comments

This week I attended Integrate 2014 (a.k.a. The BizTalk Summit). It was great meeting up again with lots of folks I’ve got to know in the past 14 years (veterans like Tom Canter, Brian Loesgen, Bill Chesnut, Richard Broida and the “younger generation” like Mikael Sand, Saravana Kumar, Michael Stephenson, Kent Weare and many more, including lots of Dutchies and Belgians), since starting to work with BizTalk Server in 2000. It was also nice to meet some of the Microsoft product team architects (online and offline, Guru Venkataraman and, offline Evgeny Popov) getting to understand their vision.

What keeps on surprising me however, is how bad Microsoft is at positioning their great stuff and how good they are at confusing us all, including the integration specialists and customers. The community will need to help them with bringing the right message. Customers deserve this. They need to be comforted. After all, they spend hundreds of thousands or even millions of dollars and euros on implementing, supporting and migrating their middleware solutions and their businesses are relying on it.

I think it’s quite simple. In integration, we have a couple of things to take care of, now and in the future:

  • Message based A2A integration
  • Message based B2B integration
  • API based B2B and B2C integration
  • The underlying SOA architecture. Because folks, that’s what it is. At least the SOA as I have always understood and applied it.

Thinking in (orchestrated) task, entity and utility services is crucial. How to expose and compose these services and how granular they are (micro, nano or macro), is basically not that important. But it depends on the consumers of these services.

BizTalk Server is integration middleware that you can use to create an on-prem SOA architecture and use for message based A2A integration. It can also take care of “traditional”, message based B2B integration. We use SOAP, flat files, EDI and XML to exchange stuff in a very reliable and manageable way. We include MDM and other data services to take care of (master) data. And in order to expose (business) interfaces as APIs, we use the REST/JSON adapter. That way, creating hybrid buses using a federated bus architecture.

BizTalk Services (MABS) is the newer, cloud based integration offering. It is basically a B2B (EDI) gateway. A very logical place to have that in the cloud, since the cloud is a big DMZ and B2B is about structured communication with customers and partners outside your own firewalls. It’s a pitty though that this has not built on further since its first release 2 years ago (but we now know why: the team has shifted focus to the new microservices architecture and tooling).

Now we have two more things: APIs and Microservices.

APIs expose your business services to Apps and Portals  in a managed way. Mashing up services. With granular security. And you can even monitize them that way, exposing them to affiliates and other partners. For an indepth (IMO) view of API management and how it relates to this, please check this article.

Microservices are there to easily create solutions by combining functionalities exposed by these services. But again, these services are juse plain old utility, entity and task services. But now exposed as lightweight JSON/REST. And probably more granular. So they can more easily be versioned and deployed independently. Check out Martin Fowler’s vision on microservices here (also read his comment on SOA vs. Microservices). They can also be exposed through a gateway. See this article.

Service (micro, nano or macro, don’t care) composition is something you can do by means of integration patterns, implemented through some sort of pre-defined proces, being an itinerary, orchestration or workflow (but probably in the not to near future, composition is also something that is taken care of by using NoProcess, based on AI: the system just automatically finds out what to do best and which services it will need doing that, and will constantly finetune that automatically).

What’s going to make or break your (hybrid) architecture is design- and runtime governance of these services. In order to be able to compose services into solutions, you’ll have to know what services are available and how you can use them. In order to be able to manage and monitor your solutions, they have to be instrumented in a standard way, feeding management and monitoring solutions in a real-time fashion. API Management (by Azure API Management and Nevatech Sentinet) is doing that for APIs. SOA governance (by Nevatech Sentinet and SOA Software) is doing that for integration middleware. These somehow have to be tied together to provide end-to-end runtime- and design time governance (including ALM), tying in with your development environment. If I were Microsoft, I’d just buy the best solution and integrate it in the platform. Just like they did with Apiphany.

Now let’s hope that Microsoft:

  • Keeps the BizTalk Server brand for on-prem A2A and B2B integration and building a SOA.
  • Keeps the BizTalk Services brand for cloud based B2B integration.
  • Comes up with a new brand for the cloud based (micro)services architecture and tooling.
  • Versions and roadmaps these offerings separately
  • Clearly depicts usage scenarios, including hybrid solutions. A couple of good architecture pictures should suffice.

On a final note, check out Next Generation SOA. Thomas Erl and his co-writers have done a great job on simply explaining it and it all fully applies to modern architectures including APIs and microservices. It’ll really help you think clearly about complex, services based solutions.

Cheers, Gijs