indaField

The Internet of Things (IoT) market will grow tremendously in the coming years. The quantity and diversity of devices that are connected to the cloud and that will spit out data will explode. The generated data will have to be analyzed and these analyses will be used to optimize business processes. New architectures will be needed to accommodate all this and Microservices architecture is probably the way to go and here to stay.

The big hype around IoT has just started and we can expect that the largest bulk of IT investments will be done in this area. Who is not interested in optimizing processes and increasing the margins on delivered products and services? Granular insight in what exactly happens in the complete production- or services chain and being able to predict as reliably as possible what will happen in the (near) future is worth a lot. We’ve got gold in our hands.

In an IoT architecture, the following components are important:

• The Things; the devices that are connected to the cloud that, without human intervention, can generate (mostly) sensor data and which in some cases can handle instructions to take certain actions.
• The (cloud) infrastructure and services to execute the following actions:
  • Receiving large amounts of data, generated by the Things.
  • Storing these data.
  • Analyzing these data in real-time, but also by means of predictive analysis.
  • Generating “business events” as a result of these analyses.
  • Orchestrating these business events by means of integration logic.
  • Potentially send back data (instructions) to the Things, so that they can take some kind of action.
• The (cloud) applications that can handle the business events and can in their turn generate new business events.

The Things are most of the times small machines with their own processor, operating system and sensors and contain some (hard-wired) logic. They can be communicated with by means of lightweight APIs. They are in fact Microservices. Because they are completely autonomous, run in their own process space and deliver a small service. A Thing is the ultimate incarnation of a Microservice!

In order to be able to process the enormous amounts of data at high speed and efficiently, we need an architecture that supports lightweight mediation; capable of distributing the generated events to the right receivers with as little overhead as possible. Such a receiver can for example be a NoSQL database, but also a machine learning service. The NoSQL database can store events for later processing and distribute them asynchronously to other interested parties. This enables offline analysis. The machine learning service can, by means of pre-defined models, determine if certain actions should be taken based on the current event and previous related events. This will then result in a business event. Which in its turn will be published to the lightweight mediation layer which will take care of it. All of this is very time critical and has to happen as soon as possible. Because the next events to be processed are already arriving.

The generated business event will have to process further by the orchestration layer. This may involve multiple business applications and services. This can no longer be called lightweight mediation. We’re talking service bus or application integration middleware here. Time criticality is less important here. What is important in this layer though is guaranteed delivery and transactionality. Traceability is also of high importance in this layer. This orchestration layer is something we most of the times have in place already. Think of traditional integration middleware such as Microsoft BizTalk Server, Tibco or IBM WebSphere Message Broker (and of course many more). The orchestration layer can be hosted in the cloud or, much more still the case, on-premises.

Lightweight mediation all revolves around the following tasks:

• Lightweight transformation (for example JSON to XML and vice versa).
• Routing.
• Applying (security) policies.
• Lightweight service composition.

In my (humble, ahum) opinion, the lightweight meditation layer is not a layer where you handle (or even need) things like batching, heavy XSLT transformations and long running transactions. In this layer, we should only think about Microservices. Microservices that are indeed lightweight, single task entities that can be composed into solutions by means of lightweight composition.

I foresee a hybrid integration world that is here to stay. Not because EAI can only be hosted on-premises or because we want to run integration solutions in the cloud so desperately, but because cloud integration in an IoT world is about Microservices and lightweight mediation. And this realm needs to be coupled to (often on-premises) infrastructure by means of transactions and orchestrations. Hybrid integration, in my opinion, is about connecting a Microservices architecture to traditional integration middleware, no matter where the latter is hosted. The big challenge with this is to provide for end-to-end run- and design time governance. We’re certainly going to need that.

IoT is the big drivers for Microservices and integration specialists will have to get to know both integration realms and be able to apply the right (combination of) patterns by means of the right (hybrid) technologies. This is a great era to live in, for us integration folks!

Cheers, Gijs