When you first start your career in data or software engineering there is what I like to call a ‘Wizard of Oz moment’. You see behind the curtain and find that behind the shiny exterior of all these bright companies there is the technology equivalent of the great Oz sitting behind the curtain (or, more likely, in an on-premise server room).
It can come in many forms, the SSIS package that has been running since 2008, the on premise VMs that are still running old windows OS and instances of SQL 2012, the PowerShell scripts that you only realise have hardcoded user credentials in them when the aforementioned user leaves (ask me how I know), the business critical bespoke software written in an ancient version of C by a long left developer that won’t compile and no one even has the correct source code for and finally the 3000 line single class .net v1 application that creates bespoke pdf reports for an entire organisation. All of these are real scenarios I have run into (and in some cases created, sorry!) over the years.
The thing is, these systems never exist in isolation. They’re always part of a larger ecosystem. With newer parts often reliant on older parts for data or functionality, this is never more so when it comes to data platforms.
Really a data engineers task is a thankless one. You are often dealing with large complex systems that have grown over years, are hard to test and no longer meeting the business needs (for instance an old ETL based Data Warehouse that cannot handle unstructured data). You are dealing with business critical reporting that has to accurately inform the rest of the business and you have data, lot’s and lot’s of data which is expensive and often rigidly structured. Finally to add insult to injury you are consuming data of very variable quality (often created by other legacy systems inside and outside of the business).
All of this can often make data platforms incredibly hard and costly to change and it is understandable that people approach this with trepidation and can often defer and kick the can down the road.
If you do not change direction, you might end up where you are heading.
Lou Tzu
The first thing to understand is that sticking your head in the sand is not going to make this go away. All businesses face these problems in some form. Every business has to deal with ‘legacy code’ and ‘legacy systems’, whether it’s the new start up that is having to deal with the code the founders threw together over a weekend or the 60 year old company dealing with decades of IT initiatives every business has some form of legacy codebase.
Secondly these things do work (for the most part). It may not be efficiently, it may require manual intervention or some form of terrible process, but they often underpin critical processes and millions of pounds of revenue. If they didn’t work in some manner or form then either the thing or the company would no longer exist.
So why then do we need to deal with them? Firstly because of the risk they bring to the business, dealing with any of these systems is always skirting the point when they will fail irrevocably and that risk rises the longer they are kept. However there are more subtle risks, these systems are often less adaptable, potentially designed to solve problems that have changed or a no longer as relevant. This gradually limits your business choices, making it harder to adapt, constraining your choices over time giving advantage to rivals and newcomers unencumbered by the same restrictions.
The longer you leave a system the harder it will be to move away from it and the shorter the timeframe you will have to do so. Do you want to deal with something now while you have time to plan or do you want to do it at the point you realise that no one is patching key libraries anymore and there is a critical security hole? Or a key rival has moved into a market that you cannot because their software is more adaptable than yours? Or you have missed an opportunity for growth because you your business intelligence is no longer intelligent enough.
If you feel stuck with a plethora of legacy systems, accompanied by grand plans to rescue them, it may point towards a more systemic organisational problem that is not being dealt with.
The way I like to categorise this is what I call ‘Artifact’ vs ‘Process’ thinking. Artifact thinking is often centred around a system or product that must be ‘protected’, all processes and behaviours are centred around minimising change and maintaining a status quo.
This can be periodically upended when the status quo becomes categorically unmaintainable, at which point there is an against the clock desperate rebuilding and disruption followed by a new status quo that has to be maintained with gradually diminishing returns.
The other main issue is not being able to react to change. If these systems are developed to meet a particular need and that need shifts, changes or disappears altogether a company or department that is organised around maintaining a status quo struggles to adapt.
That resistance to change is exactly what Process thinking aims to overcome.
The contrast to this mindset is what I term ‘Process thinking’.’ This sees all software and systems as just an extension of a constantly set of evolving business processes, the object of the exercise is to accept change is always happening and manage it in the most efficient way possible.
This is not to say that you should be constantly binning your frameworks and data systems and rewriting them every week, change will happen at different rates in different parts of the system. Rather it is about managing change at the right pace and responsibly.
For instance with data platforms rate of change can be often slower because of the difficultly and cost of managing vast amounts of data, this is fine, what is more important is that change and evolution is still happening even at a slower cadence.
This brings us to a key consideration when managing large systems: the difference between letting things become static artifacts, versus treating them as living, evolving systems.
This means instead of a cycle of a slowly degrading systems heading towards obsolescence, followed by a massive effort, increased risk, inflated budget and a new system that begins the cycle anew. We instead have a slowly evolving system that is always in some sort of managed flux, with risk and cost spread more broadly.
You could argue that following the evolutionary approach increases complexity and risk, because you are often managing multiple systems as one thing supersedes another. From experience I’ve found that teams that are only exposed to change periodically in high risk situations don’t respond or manage that change well. If you make that complexity constant and familiar, just something you do every day, then you are forced to regularly deal with it and therefore become very comfortable with it and therefore good at managing it.
It also means that if change has to happen rapidly, a new area of opportunity opens up for instance, you are not going from a standing start rather you are just accelerating a process that is already in motion.
Also when dealing with ‘artifacts’ you are still dealing with complexity and risk, just of a more toxic variety. Legacy code is often complex and hard to manage, artifact thinking often devalues refactoring so code becomes an accretion of ‘fixes’ that increase that complexity and fragility, if often ends up with a system that is designed through compromise because people are more and more afraid to make the decisions needed to make it work better.
So how do you move to be more ‘Process’ orientated, what are the techniques you can employ that will allow you to evolve your systems over time instead of sudden bursts? This is what I hope to lay out over the course of my next few articles.
We’re going to look at things like Conway’s law, how to clearly define our outcomes, employing clear testing strategies, using architecture patterns like the delightfully named ‘strangulation pattern’ and release strategies like dark launching and canary/soft launches to to allow you to move from one system to another with minimal disruption to an end user.
I’ll also look at some examples of different change scenarios and use those to discuss how we could use different techniques to approach them.
In the meantime, if you're interested in how we help organisations manage change and evolve their data systems, explore our services at Advancing Analytics.
*Disclaimer: The blog post image was generated by AI and does not depict any real person, place, or event.