Welcome to indaField.com. I use this site to share my views on Microsoft Azure and cloud computing in general.
You can get to know me a little bit by reading the About Me.
If you need some help with cloud strategy & architecture, want to make use of my CTO-as-a-Service, need an advisor on your board, or you have product management challenges in your software development team just contact me here.
Kind regards,
Gijs in ‘t Veld (in-the-field…. indaField)
Afgelopen jaren heb ik veel columns geschreven, met name voor Computable. Ik heb deze opgefrist en in een boekje gegoten. Met als doelgroep stagiairs, studenten, afstudeerders en young professionals in de IT. Om (op een hopelijke vermakelijke manier) te leren van de dingen die ik geleerd heb.
Daarnaast is het natuurlijk ook interessant leesvoer voor alle andere IT’ers, ook die met veel ervaring 🙂
Veel leesplezier!
You’ve probably all heard of Conway’s law. If not, check it here.
I’ve never been involved with a company where his law did not apply. It is shocking to see that in all these organizations the architecture indeed reflects that organization and communication structures.
The result is that you end up with an architecture that gets harder and harder to maintain. Especially when the organization and communication structures change, which often happens. Solutions that have been built a while ago, when waterfall projects were the norm and teams were working in silos, are now hard to grasp and maintain and evolve, because companies have transitioned to DevOps and Scaled Agile.
You first had a middleware team, a front-end team and a back-end team. Creating bottlenecks when developing multiple solutions with cross-dependencies. This is probably the hardest nut to crack. What to do?
First of all, having an architecture with front-end, back-end and middleware is not the issue. It becomes an issue when the middleware team starts to build all kinds of stuff to help get rid of issues in back-end and front-end systems. Building logic where it shouldn’t be. And having ownership of things the middleware team should not have ownership of.
So, make sure that you have teams in a Scaled Agile organization that work according to a number of strict principles. Especially around data ownership, service orientation, service ownership and business domain ownership. You most probably will still have specific teams in your release trains, that are specialized in hard platform stuff, related to data and integration. And keeping the underlying platform(s) up-and-running. But when strictly adhering to the (business-, information- and technology-) architecture principles chances are that better (maintainable) solutions will be built.
The issue also quite often lies in formal structures within organizations and the accompanying strict hierarchy. Instead of relying on specific functions within the organization, we should do our work based on roles. It’s much less worse when an architecture reflects the organization of roles than when it reflects the organization of functions. Transitioning to such a new organization paradigm is not something you can do overnight. So, during the transition, make sure to guide your teams. Both with regard to technology and principles and working together. This way you can keep them working strictly according to the core principles. In the end, we should be able to reverse our old practices and make sure we’re not bothered by Conway’s law anymore.
At the various organizations that I’ve been involved with regarding IT strategy and architecture it always strikes me that they so often “bury” themselves in technology. What started out as a nice architecture has derailed into a Frankenstein solution, as I’d like to call it.
Often, this happens due to a combination of the usage of monolithic (and bought licences of) software products, the phenomenon “sunk cost” and architects who, as a result of that, as a carpenter (have to) treat everything as a nail.
What do I exactly mean by Frankenstein solutions? A couple of examples:
In all three examples, the tools have been used as a hammer. But the solution is oftentimes not a nail. Sometimes you need a screw, glue or a click-system. But, because we can only hold a hammer, we try to make it fit anyway. This, while you have learned as a kid that you cannot fit a triangle in a square hole in your block box. Unless you of course try to fix it in a creative way, by turning the block and making it fit. Or hit it hard with a hammer.
With the arrival of PaaS (cloud platform-as-a-service) services, it has become possible to choose the right service for the right challenge. What seemed to look like an unsorted box of legos of which many users were scared because of the complexity that such a large number of different blocks could bring with it, this turned out to be a blessing in disguise. Together with the arrival of better governance and monitoring tools in the cloud, the solutions you build with these PaaS services become more manageable.
It will make the architect’s life also easier. Instead of making choices based on already acquired, expensive licences of product x or y, the architects can now easily choose new PaaS services if the challenge asks for it. It has also become easier to “fail fast”, because if the solution doesn’t seem to work as expected in a proof-of-concept situation, you can easily turn the cloud service off again.
It does however require a lot of knowledge. The architects not only need to know everything about the various cloud services, but also about the applied architecture patterns. And the operational costs of the services. Luckily, the cloud suppliers have become more an more clear on this. Every supplier provides “open source” best practices and patterns that you can easily use. And a calculator to estimate the operational costs.
Finally: It’s always difficult for any organization to change an already chosen direction. More so, if lots of money has already been invested. But it is very important to also think about the maintenance cost. And the job satisfaction of the people who have to keep the solutions afloat is also very important. It too often is the case that 80% of your IT budget goes to maintenance and thus only 20% to real innovation. And as a result, many IT people leave the company dissatisfied. So, make sure that the architects in your organization have more than a hammer in their toolbox!
As a start-up or scale-up in the technology business, more specifically the software and services business, it can be hard to face all the challenges that you meet every day. From a commercial perspective, but certainly also from an organization and technology perspective. How do you market you products and services? How do you make sure what you do is different from your competitors? How do you attract the right employees. How do you make sure that you deliver in an excellent way? How do you stay relevant for customers and potential new hires? How do you scale?
As I see it, there are challenges in the following areas when it comes to technology:
Every company makes lots of mistakes during their growth and hopefully learns from them. But wouldn’t it be great if you could learn from other companies mistakes as well, so that the price for these learnings will be a lot lower and you will be able to grow quicker and at the same time more sustainably?
I’m available as a member in your advisory board, or you can hire me on a consultancy or project basis to help you with these challenges.
Just let me know if you’re interested, and let’s discuss!
It’s rather amazing what happened the last couple of years when it comes to blockchain. It all started with Bitcoin of course. At the time of writing this, the total market cap of Bitcoin is $1 trillion. When we look at Coinmarketcap today, there’s a total of more than 80 coins that have a market cap of more than $1 billion. and more than 360 coins that have a market cap of more than $100 million. Lots of fiat money is locked into these “projects”. Most of them basically being ponzi schemes, because the technology underneath those “projects” is mostly crap and the only ones making money are the ones that bought during the ICO (initial coin offering) or the ones that initiate “pump-and-dump” events. A new crypto sucker is born every day.
The whole idea behind Bitcoin and other blockchain based cryptos is “decentralization”. There’s no central organization responsible for how things are handled. Except of course when it comes to tokenomics (how many coins are there in circulation, how many can there be max and how many will be printed per period; the inflation) in most cases. When you look at most coins, there is no max number of coins. Which means inflation is potentially infinite. This is different from Bitcoin, which has a max number of 21 million. But what keeps evil people from just changing that? And what can you do about it?
As we all know by now, the “rareness” of coins is based on the difficulty of mining new coins. The way coins fluctuate (unlike the “stablecoins” like $usdc and $usdt) will make it very difficult for them to ever be used as digital money (except of course for buying a Tesla, according to Elon). Basically, people just gather them, hoping they will increase in value. Do you actually own a piece of the company behind the crypto? So, the case of the SEC against Ripple with regard to “is XRP a security?” is something to be watched carefully.
In the meantime bots rule the buying and selling of crypto. It’s such a coincidence that coins go up and down in groups. And lately we can even see that crypto moves along with stock markets. And because of all the derivatives, futures and all kinds of other “smart” products, including the “shorting” as we know it from traditional stock markets there’s really no way that the moving up or down does have anything to do with underlying fundamentals. It’s all FUD or hype.
And also in the meantime lots of cryptos, most notoriously $eth, are used as a platform by other cryptos and tokens and because the price of the “platform” coins goes up, transaction prices go through the roof. Basically making executing of transactions way more expensive than plain old credit card transactions.
The way I look at it is that just like with the gold rush in the last century, the only people really making money on crypto in a predictable and stable way are the “tool supplier” commercial exchanges such as Binance, FTX, Bitmex and quite some others.
But that’s all about crypto. How about the underlying blockchain technology, including the “smart contracts” running on them? When a smart contract fucks things up (because as a user I didn’t really understand it, or the “independent” auditor did a bad job auditing it) you are basically fucked. Nowhere you can turn to. That’s the downside of decentralization. Let’s say I want to borrow some $eth based on my $btc balance. And that I enter “90” in the number of days within which I will pay back. And the smart contract just automatically wires the $btc collateral to the other party after 80 days because I didn’t pay back yet? Nowhere I can turn to.
Of course there are more and more “governance” tokens that are implemented, that are basically used to democratically determine when and how things are implemented and executed. But we all know that democracy doesn’t always work well. If we’d held a referendum right now about getting rid of the Corona lockdown, it would problably pass with great majority. But would it be a good solution? And who guarantees how the things voted on are actually implemented? Lot’s of scams already exited. “Rug pull” has happened quite a lot lately. Gone are the coins, or at least gone down in value to (almost) zero.
And the blockchain itself? It’s just an energy consuming monster that replicates everything to every node. And it will not forget anything. And because they are so inefficient, lots of important data is in fact stored “outside” the blockchain, only the metadata is stored on the blockchain. That does not make sense at all. And how about GDPR / AVG? No one cares.
Apart from that there are issues with tokenization of digital assets, let alone “real-world” assets. The way we put things on the blockchain should also be certified. But how can you certify, without independent external auditors? For digital assets, the whole digital process for getting things on the blockchain should be 100% reliable. What if we log certain access to information on the blockchain, but we cannot guarantee identity? What if we log the the sale of a house or piece of land and we cannot digitally guarantee that the identity of the asset is correct? There are so many things that we still cannot digitally 100% prove. So, the same old “garbage in, garbage out” still applies, even on a blockchain.
Of course this is all about public blockchains. And I just don’t see use cases for consortium- or private blockchains.
Correct me if I’m wrong, but the current ways of thinking by the blockchain gurus is just plain stupid or at least misleading to the dumb folks (you and I) actually using it. I’ve read quite some “white papers” and “roadmaps” on lots of projects, and 99.9% are plain stupid and the other 0.1% forgot some important stuff that still “needs to be taken care of” but in fact they just cannot. We need a new ledger paradigm, including new ways of guaranteeing smart contract behavior AND we need 100% reliable tokenization and oracles. But is that actually possible, without third party (government controlled) institutions? I’m not sure. Maybe open source AI can help? But for now, I just don’t trust it at all. Decentralized or not. They are all just underworld casinos and the house always wins.
A lot is going on in the world today. Through social media and news papers we get to see what happens on a daily or even hourly basis and it does not look good these days. Especially right now in the USA – a what we think rather evolved, western country – things are quickly escalating.
My take on all this is that we as humans take ourselves too seriously.
We’re just born into a family we have not chosen ourselves. Born in a country. With a certain “color”. In a family that lives in a certain “culture”. And which practices a certain “religion”. And has some hate issues because things happened to them or their ancestors in the past. When you are born, you don’t have the chance to choose any of this. It’s only until later, when, let’s say you are 14, 16 or 18, that you can actually start to think for yourself. But by then, the “damage” is already done. You’ve been taught certain morals. But mainly “group think”. Your own (is it, actually?) version of good and bad. Or, more accurate, the morals that the “culture” you have been brought up in thinks are good.
Very young people don’t have prejudice. It’s only when they are subject to peer pressure that they become prejudiced. It’s what is called “cattle behavior”. This cattle behavior can actually lead to millions of people following a certain “role model”. But such a role model can sometimes be quite evil. Presumably sane people can start following evil leaders just by pure peer pressure, which becomes even worse when the circumstances in which they are living are currently not that good. That’s how Hitler got big.
That’s why nurture has a much greater effect on humans than nature. You are not born a racist. Or a Christian. Or a Muslim. Or an anti-semitist. Or Buddhist. Or a white supremacist. There are no known genes that make you any of this. You are born into an environment that cultivates these morals. And it is very difficult to come up with your own ideas within such an environment. Your read the books and other publications your parents read. Or your priest or imam preaches. Or the guy with the biggest mouth in your neighborhood tells you to. Or a left- or right wing teacher teaches you to. How would you be able to sustain the peer pressure if you do not agree with any of this? How would you break out of this?
Taking someone from a certain situation into a completely other situation, let’s say by adoption, will have a great effect on that person. Just by putting a human in a completely different environment, that human will evolve into something completely different. Maybe even quite the opposite of what he or she was born into. Look at the evidence provided by identical twins that were separated after they were born and grew up in completely different environments.
The issue is: there is no right or wrong in culture or religion. If my parents would have been followers of the flying spaghetti monster, I would have had a passport now with a photo in it with me wearing a colander on my head. And it would all be fine. As long a I don’t take myself too seriously and I don’t bother other humans with my beliefs. A fact is that most people are atheists, because they don’t believe in 99% of the other officially recognized world religions.
If only humans would be able to not take themselves too seriously and not try to convert other humans, the world would be a much better place. But how will we accomplish this? I think by reading lots of books and other publications that open up your mind. Have discussions with people with other opinions. Weather you finally agree with them or not. Which makes you look at things from different angles. And by doing so become your own unique self. Just do not make the mistake to just thoughtlessly make other human’s ideas your own. You have to think for yourself! And of course get the opportunity to get secular education that actually teaches you to think for yourself. Education is not about learning other peoples opinions or beliefs, but learning how to learn! And in the end, even when you are 100% convinced of your own beliefs, don’t bother other people with it. Have discussions, of course, but don’t ever use violence or other oppressive methods to “teach” others your beliefs.
One of the biggest problems in the digital world is that people can do stuff anonymously. Especially for large public networks such as Facebook, Twitter and Instagram this is a huge problem. People cannot be held accountable for the sometimes very bad actions they take.
On some networks, you have to provide your true identity. The identity is checked by means of passport, credit card, mugshot and other means. This is mostly only done on financial networks such as banks and brokerage systems.
What if we would just ban all identity providers who can’t prove true identity? Which can’t be traced back to a real person or legal entity?
The technology and legal bodies are available.
If all important digital networks would just only accept users with real identities (backed by either an open source initiative or governments) the world would be a much better place.
With the rise of functionally rich PaaS services delivered by the so called cloud mega vendors such as Microsoft, quite a lot has changed. First of all, it has resulted in confusion in the market, but right now we can see that organizations are really starting to understand the philosophy and embracing it.
Think in services
Especially in the early days it was difficult to understand for both consumers as suppliers of cloud services. For data- and integration products it was the hardest to grasp. In yesteryears, it was quite easy for organizations with a Microsoft-unless policy: if you wanted a solution for data you got SQL Server and if you wanted a solution for integration you got BizTalk Server. Very easy to compare features between those products provided by Microsoft and other vendors such as Oracle or IBM. Work through a checklist, score and choose.
Nowadays, these kind of data- and integration functionalities are no longer delivered by products but by a set of cloud services. This has at first resulted in great confusion. What do we compare with what? Especially when comparing a set of cloud services on the one hand and a monolithic platform on the other.
Say goodbye to platform “products”
Even for Microsoft this was difficult. Because on the one hand there is a very powerful message because Microsoft Azure provides all building blocks to build a mature integration platform, but on the other hand Microsoft has to compete with some pure-play vendors that provide monolithic productds. Products that are very good at one thing. And in which the various components are 1:1 dependent on eachother. Microsoft also provided one of those products: BizTalk Server. A fantastic integration product, but monolothical and not very service oriented. The processes in such a product are designed beforehand and therefore “baked-in”. And therefore not very flexible.
However, we saw that competing with the pure-plays was (and still is) not easy. At one moment in 2018, Microsoft chose to rebrand and bundle the services in Azure that are needed to build an integration layer to “Azure Integration Services”. The same happened to “IoT Suite” for example. Of course this was done to make it easier for potential customers to make product comparisons. But personally, I thought this was an admission of weakness by Microsoft. It basically does not make sense at all. It is simply the power of the platform to provide a set of loosely coupled services with which you can build very flexible integration layers. And besides that, you get the whole Azure platform with it that you can make part of your end-to-end processes, potentially being integration-, data-, app- or collaboration oriented solutions.
The same goes for data solutions. These days we have relational data stores, table stores, document stores, blob stores, data lakes, etc. It has been tried to put that all in SQL Server in that past. In Microsoft Azure, these are all “loose” services that all run in their own scalable containers and that can be connected to eachother through (among others) Azure Data Factory, to create a data platform that suits your organization best. And on top of which you can deliver all kinds of analysis services such as Databricks or PowerBI in order to create information from the data.
Non-functional aspects
What is very interesting, is that Microsoft has not only provided all these great features as services, but at the same time has invested a lot in architecture guidance and non-functional aspects. Where you had a silo application such as BizTalk Server and SQL Server (and comparable products from other vendors) before, in which all the management and monitoring tools where built-in, you can see right now that on top of all cloud services you have tooling for end-to-end management and monitoring. For example, for deploying workloads by means of infrastructure-as-code and to gain insight in the relationships between services. Right now we can deal with loosley coupled platform solutions that at the same time are very manageable and quite easy to monitor. That’s real progress! I predict that monolithic integration-, data-, collaboration- and app platforms will die a slow death. Long live PaaS!
Cheers, Gijs
Chances are that most of the transactional data in your organization are still stored in relational databases like SQL Server, Oracle and DB2. When modernizing your application landscape this will rapidly change. Why? And what does that mean?
Modern software solutions are service oriented and very distributed, where the systems of record are still hardly having a user interface but do have good, mediated APIs that expose business logic as services. And the tailormade solutions are created through low-code and integration on top of these APIs. The composition of these services exposed through APIs is taking place on a higher level. That is where the differentation takes place. That’s where you use small apps that do exactly what you need, fully integrated with your application landscape. The underlying systems of record will in the meantime also be modernized by the suppliers, where these will also move away from the silo approach and relational databases. Mark my words.
There are three great problems with relational databases:
In practice, lots of relational databases are misused to store non-relational data. Just because the thing is already there and operated. And because it easily goes with the regular backups. That’s a wrong strategy. It should change. Now.
Now is the time to slowly say goodbye to relational databases in your tailormade solutions. For every need of data storage, try to find out how relations with other data can be created. These days, there are lots of different storage methods, especially in the cloud. To name a few:
In a (micro)service architecture it is important to cater for eventual consistency. The database cannot take care of that for you anymore, simply because that cannot be done overarching multiple databases. How you organize your services and APIs will become more and more important, including the data architecture belonging with that.
So, what about business intelligence and analytics? These solutions will have to change rapidly as well. More and more data that is needed to show in dashboards or reports will come from non relational databases. Traditional ETL cannot be used anymore. Cubes cannot be created that easily anymore. Artificial intelligence as part of your analytics solutions, even for the traditional hindsight applications (“wat happened?”), cannot be avoided. Let alone if you want predictive (“what will happen?”) or prescriptive (“what do I have to do to realize this?”) solutions. You’ll have to start thinking about these kind of things and adjust your architecture to it. Waiting for “the hype to blow over” is not a good strategy. The relational database is already clinically dead IMHO. You can throw your SQL skills in the bin in no time!
Cheers, Gijs
indaField 