I’ve spent three years in the trenches of data mesh implementation. The lesson that took me far too long to learn: this is a photography problem, not a platform problem. Data mesh doesn’t fail because of technology. It fails because teams confuse taking snapshots with owning photographs. A data mesh works when three things happen:

• Perspective widens

• Ownership becomes explicit

• Teams resist the urge to photograph whatever is in front of them

Everyone can take a photo

I am chronically guilty of ‘spray and pray’ photography. When someone hands me a phone to take their picture, I mash the digital shutter button like a video game champion. I give zero thought to the lighting, the background or, if I’m being honest, sometimes even the people in the shot themselves. All we end up with are 20 identical, useless shots clogging up the photo gallery. None are the trusted memory being sought. In the data world, we do the exact same thing. I’ve watched a team rebuild an entire data product from scratch - reimplementing months of gnarly business logic - because they needed one extra column and didn’t trust the team that owned the original. In both cases, we created plenty of data but zero trusted memories.

The contrast became obvious when I hired a professional photographer for my wedding. I watched them work and learned that they didn’t mash buttons but intentionally used lighting, aperture and framing. They knew exactly what they wanted to capture and - more importantly - what they wanted to leave out. Building a data product requires the exact same discipline. A professional knows that a great photo is a deliberate, narrow slice of reality.

Every photo is a slice

A photograph is narrow by design. It captures one subject from one perspective. That’s the entire point. A data product should work the same way. Think of your smartphone’s ‘panorama’ feature where you take multiple overlapping shots to create a wider landscape. Have you ever actually taken a good one?

The same is true of the panoramic data product. Someone tries to capture everything - whether data or scenery - and ends up with a warped monstrosity with someone’s disembodied arm floating in the middle. The lesson? Split data products by subject area. Multi-domain data products are corporate panoramas. Crop ruthlessly.

Which photo goes into the album?

We have all been held hostage in that meeting. The one where three different leaders proudly present three completely different numbers for ‘active customers’. Instead of a strategic discussion about business value, it devolves into a 55 minute ‘robust discussion’ over what the word ‘active’ means.

That meeting doesn’t derail because you chose Snowflake over Databricks. It derails because your organisation confused the ability to snap a photo with the discipline of curating a gallery.

What you witnessed in that room is simply two teams who photographed the exact same scene from slightly different angles. Both datasets are technically “correct” in their own context. Neither is authoritative. To solve this, you need a person or process with the authority to decide which shot actually makes it into the company album. That’s governance, not bureaucracy. But governance only cleans up the mess after the fact. The better question is: what if teams shot with intention from the start?

Shoot with the end in mind

A great photographer doesn’t just take the shot - they understand their audience. Who is this photo for? What story does it tell them? People take photos at concerts just to prove they were there, not to create a masterpiece. Similarly, engineers build data products just to check a box, taking the path of least resistance.

Nobody wants to spend weeks building a data product, only to burn the final Friday manually typing out metadata into a dusty Confluence page. Neither do you pull out your phone at a concert and want to click through endless settings before taking the shot.

People will always choose auto mode. Every time. Our job isn’t to shame people into manual mode. It’s to make auto mode good enough. We must code the rules so the path of least resistance is also the path of compliance.

The EXIF data test

If you’re wondering whether your organisation has photographers or just people with cameras, here’s a quick exposure test:

• Can a new team discover data products without a meeting?

• Do data products exist with clear owners?

• Are duplicates intentional and labelled as so?

• Is there accountability for how it all fits together? (aka the big picture)

Say cheese

Data mesh isn’t a technical or architectural problem - it’s a perspective problem. Like photography, everyone thinks they can do it because they interact with data every day. But it fails because we confuse the ability to snap a photo with the discipline of curating a gallery. The real work of data mesh is widening people’s aperture - getting them to see beyond their narrow frame of reference.

Widening the aperture is uncomfortable. It means caring about things outside your frame. It means accepting that your “photo” exists in a gallery alongside others - and that the gallery needs curation. For most domain teams, this means caring about someone else’s problem. That’s the real ask of data mesh. Not better tooling. Not more autonomy. More perspective.

Are you building a data mesh with skilled photographers... or are you just handing out disposable cameras and hoping for a masterpiece?

My First Encounter

I worked in one place as a consultant and built a pipeline and dashboard - it was beautiful, a work of art that Da Vinci would have been proud of. We deployed it, everyone high fived and then I left. A year later, I caught up with the team and enquired how it was. They said they had stopped using it and went back to Excel. I asked what they did with the dashboard, someone piped up “It’s still there running but no-one uses it”. It was technically alive but practically dead to the business. I realised that I had created a zombie.

Anatomy of a Zombie

Zombieware isn’t abandonware rotting in a repo. It’s the technically-alive-but-business-dead software actively draining your resources while everyone pretends it doesn’t exist. And the only cure? Stop celebrating builders and start demanding owners.

“Zombieware” - software that is technically alive because it consumes resources but is dead to the business because it delivers zero business value

Zombieware is software that is still alive and running - consuming resources and attention. This in itself is not necessarily a bad thing, however, it creates the illusion of value as it devours resources on your server or cloud bill.

What makes it lethal is that it serves no living purpose. There is no business value for the effort to design, create, deploy and maintain the software. This means that it is:

• technically functional and consuming resources (think: CPU, storage, licences)

• it has no active users or its outputs are ignored

• it has no clear owner responsible for its lifecycle

To be clear, this is not some passive form of tech debt. It’s an active, resource-devouring plague on your organisation’s focus, resources and profit margins. The only cure is to shift from a culture that celebrates building to one that demands ownership and return on investment.

The Silent Tax on Everything

Zombieware isn’t some passive liability, it’s a tax on everything you do. It’s paid in compute cycles, cognitive load and opportunity cost. It’s bleeding you dry and no one - not even Alex from Finance - ever approved the expense.

This tax shows up in insidious ways. Consider two exhibits from my own chamber of horrors.

First, the technical drain. We built an internal API to handle customer verification. It just worked - a model of reliability. So reliable, in fact, that when we shipped v2 and moved on, nobody remembered to kill v1. Two years later, we stumbled upon it, faithfully running in production, ready to serve requests that were never made. The direct resource cost for its dedicated compute and storage was bad enough. The real surprise? The indirect cost of shipping and storing of logs no one was reading was even higher. That’s before you account for the security risk of an unpatched, two-year-old service and the cognitive overhead it placed on any engineer trying to understand our architecture.

But the most blood-draining costs aren’t on your cloud bill, they’re inside your head. We spent a whole quarter building a complex ETL pipeline. It pulled data from three tricky systems, cleaned and dutifully loaded it into our data warehouse every night. It ran flawlessly. Six months later, a curious junior engineer asked which dashboards were powered by those tables. The answer? None. The business stakeholder who requested the data had changed roles and their replacement had no idea it existed. The build cost was immense, the ongoing compute cost was in the thousands daily and a total waste, but the real damage was to morale. Nothing kills a builder’s spirit faster than seeing their work become pointless.

These aren’t isolated incidents. They are symptoms of a culture that celebrates the launch over the lifecycle. Let’s be honest: building is the glamour work. Shipping new things gets you promoted. Maintenance is the work nobody sees. The creates an environment that breeds zombieware.

You know how hard it can be to get money to budget or for spending approval. However, zombieware is a cost to your organisation that no one approved. It’s paid in compute cycles, in maintenance overhead, in cognitive load of engineers and in the opportunity cost of what you could be building instead. It’s bleeding you dry, one server instance and one artefact at a time.

The Cure: how to prevent and remove zombies

The first step is instrumentation.

Without the telemetry of what exists and how it’s being used, we are flying blind against the zombie invasion. As we’ve seen in the films, this doesn’t end well.

The second step is clear ownership.

Every new project, from a single script to a microservice, must have a named owner before a single line of code is written. This sounds insultingly simple. It is. Often, it’s the lack of accountability that drives zombies. Ownership isn’t about building - it’s about the entire lifecycle, including decommissioning.

The third and final step is to set a kill switch.

Teams often measure technical metrics like CPU utilisation and service latency. These are important but only half the story. Define metrics up front to measure minimum acceptable usage.

• How do we define usage? e.g. opening a page, clicking a button, making a decision, resource cost

• What time period makes sense? I’ve found that 14, 30 or 90 days are good starting points depending on the situation

• Add the review of the metric to an existing process. If the minimum acceptable usage isn’t hit, undeploy the software

Sound extreme? It’s less extreme than paying for servers to run code that delivers precisely zero value. You already do this with feature flags and A/B tests, you’re just applying the same ‘prove your value or die’ logic to the whole software lifecycle.

Advanced Zombie Hunting

Those three steps will help but zombies are notoriously difficult to kill. It’s often easier to turn a blind eye to the zombie feasting on your cloud bill than rally the team to do something about it. Here are a few ‘gotchas’ that can help stop you from being caught out.

Related systems

It’s not always so simple when many software provide or read information from other systems. If our system is a building block for a downstream system that is important, then we need to consider if everyone is using the same approach. Otherwise, we risk continuing to run our service to feed a downstream system which has no real usage - meaning that our service, despite it’s appearance, is actually zombieware.

Run a Zombie Purge

While the kill switch is great, sometimes we see metrics that are too easy or not measuring the real problem. To solve this, once a year, ideally near October 31st, have the team dress up in Halloween costumes. Spend the day reviewing the usage of all software and doing a purge of any zombieware identified. Every resource or process should be investigated and tagged as either ‘Ok’ or ‘Zombie’. Think of it as a ‘trick or treat’ and reward zombie kills with lollies (candy) or whatever motivates the people at your company.

• Ask a simple question for every system: “Who uses this and what decision did they make with it last week/month?”

• Use monitoring tools to find things with zero usage

• Create a “Zombie of the Year” award as a fun way to celebrate decommissioning and simplifying the stack

Organ donors

When it comes time to kill a zombie, ensure that you harvest any organs that might be useful. These could be features or components you could reuse or queries that add value. The simplest method we’ve seen here is store them in a shared ‘Organ bank’ repository.

The final stake

Look at the last three things you built. Do you know who uses them today? Do you know when they should die? Or did you just launch them and walk away?

True senior engineering isn’t just about building complex things. It’s about having the wisdom and discipline to kill them. Stop measuring your worth by what you launch and start measuring it by what you have the courage to kill.

The “sexiest job of the 21st century” is a lie

Back in 2012, an article came out about the role of a ‘data scientist’ - someone who “has the training and curiosity to make discoveries in the world of big data”. It was noted that this skillset was rare and decreed the role as the “sexiest job of the 21st century”. As people flocked to this career path, the role was declared a “unicorn” and a dreaded Venn diagram was created to capture the overlap of the three key skills:

• software engineering

• statistics

• business acumen

The role took off because it was something different and fun. Who doesn’t love to predict the future? Everyone wanted to believe that data scientists were the mythical hero who could do it all. However, data in the corporate world was their kryptonite - while in theory it was clean like iris flower data - the reality was more like overgrown weeds and fallen trees. This proved to be a stumbling block to getting started but the enthusiasm to hire data scientists didn’t slow. Yet over a decade later, why do 80% of data science projects fail?

The problem isn’t the data scientist, it’s the role - aka it’s not you, it’s me

The answer isn’t the practitioner - it’s the job description. The role itself is a paradox. What started out as the universal skill for a data scientist was “the ability to write code” but it quickly became a catch-all term for three distinct functions.

1. The data cleaner

 The bulk of the work is wrangling data into a usable format.

 This is the job of an Analytics Engineer.

2. The pipeline builder

The task of getting data to a model and getting predictions out.

This is the job of a Data Platform Engineer or ML Engineer.

3. The storyteller

Being able to explain the insights and link it to the business problem or need.

This is the job of a Data Analyst.

Notice what’s missing? “science”...

The science that was really just vibes

At this point, you might be thinking “But what about the ML models? Didn’t the data scientists create those?”. And you’d be right. However, while that was the glamorous side of the job, it generally accounted for just 10-20% of a data scientist’s time.

When you unpack the ‘science’ further, it starts to unravel more:

• Algorithm Choice: This isn’t a mystical quest. It’s a dependency choice, just like a software engineer choosing between React and Vue (or the other thousand Javascript packages). Why do people spend three weeks testing Random Forest vs XGBoost for a 2% accuracy bump that translates to zero business impact?

• Data Drift: This is portrayed as a novel scientific challenge but it’s just a data quality and monitoring problem. Analytics Engineers have been handling schema changes and slowly changing dimensions for years.

• Build vs Buy: The decision to build a custom model is not a scientific one. It’s a classic engineering build vs buy analysis. For most use cases, it doesn’t make sense to build your own, since off-the-shelf models or AutoML solutions will usually suffice. However, if a 0.01 uplift in AUC translates to a $1M return, then the investment and ongoing maintenance might be justified.

All that’s left is a process driven more by intuition than evidence. What we called data science was often just trial and error in a Jupyter notebook - vibe coding at a PhD price.

Will the real model owner please stand up?

Unlike traditional engineering which was delivering in small increments, data science often involved large upfront time investment called discovery or experimentation. This created friction between executives keen for results and the data scientist grinding through experiments.

Things went sideways as the data scientist become too technical-focused on the ‘science’ rather than the actual business problem. Disconnected from the business outcome, it became a negative feedback loop and more time was spent tuning the the model accuracy rather than validating the outcomes.

Thus, the ownership of the model became unclear as the business person asking for the help didn’t understand how the 5 layer neural network predicted customer A would churn. Neither did the data scientist understand the impact of a customer leaving the business. This left the model running in production but unused. And this was a good case scenario.

Often the model hardly ever made production due to hand-offs. At one place I worked, the data scientist wrote a model in python. However, the ML framework was written in Java. So an engineer had to rewrite the model in Java that the data scientist had created in Python. This is crazy! Imagine McDonalds creating two hamburgers for the one you ordered. It’s no wonder the executives were wondering why ML initiatives took so long with situations like these.

The Data Science Unbundling: what comes next

Data scientists won’t exist in five years - not because of AI - but because the work splits cleanly into two real jobs: product analytics and engineering. The rest is a vibes-based middle where notebooks flourish, models drift and nobody owns outcomes.

The roles that actually matter:

• Data Analysts: Own the understanding and explanation of data

• Analytics Engineers: Own the semantic layer and deliver trusted, clean data

• ML Engineers: Own the end-to-end process of deploying, scaling, and monitoring models as a product

• (Optional) Research Scientists: The true scientists, reserved for the 1% of novel problems where off-the-shelf solutions don’t exist

We’re not saying that the people who were Data Scientists aren’t valuable. Quite the opposite. This unbundling is actually a promotion - where there is increased focus, accountability and, most importantly, impact.

If you’re a Data Scientist: Audit your last three months. Where did you create tangible value? Double down on that - whether it’s engineering or analytics - and rebrand yourself.

If you’re a leader: Stop writing unicorn job descriptions. Redefine your roles for clarity and ownership. Your team’s velocity and the business’s profit margins will thank you.

The “Data Scientist” title is a failed experiment born from a decade of hype. While the job title joins Data Entry Clerks gathering dust, the future belongs to those who make data impactful.

In the 2000s, we had the humble role of business intelligence (BI) developer. Business Intelligence was a fancy term that really meant ‘analytics and insights’ but sounded mysterious and cool. I couldn’t resist either and used it myself to refer to the work we did.

A BI developer was tasked with crafting data from a source system into something that could be used to understand the business, answer questions and make decisions. This generally got built in applications or in a data warehouse. BI developers did it all - extract data from source systems, transform it into something useful, load it into a data warehouse, then build reports and dashboards to share with the stakeholder. They were the full-stack data professionals of their time.

This role was often in the IT department and often quite removed from the business domain - meaning lots of back and forth between the BI developer and stakeholder to understand the data and to align on the final output.

While it was one role, it had five completely different skill sets:

• data detective (what does this actually mean?)

• pipeline engineer (how do we load it?)

• data modeller (how should we structure it?)

• governance analyst (can we trust it?)

• storyteller (what does it tell us?)

Fast forward to 2011 and we started having large tech companies, such as Google and Facebook, advertise roles for a ‘data engineer’. This role was eerily similar to a BI developer but sounded much cooler (yes, again). In a simplified sense, it was tasked with moving data from a to b. While it sounded simple, the complexity required to understand both the content of the data as well as the computing resources needed to move and transform the data gave the role a large scope of work. This breadth naturally had many overlaps with existing roles. Suddenly everyone wanted this title. Data modellers, BI analysts, ETL developers - all became ‘data engineers’.

Here’s where it gets interesting... as all these data engineers worked away, we discovered that understanding customer behaviour to build a churn model requires completely different skills than optimising database partitions for performance. One is about business logic, the other is about infrastructure. But we kept hiring for both under one role - data engineer.

Stakeholders put in requests to the data team to build the data asset and associated report or insights. However, due to the mix of skills as well as the domain expertise needed, it became a huge time sink. I was at one organisation where the data team had a 14 month backlog! But when I asked if I (in a different team) could do the work for them, I was met with the reply “no, only we can work on the data tools”.

The chasm was increased as data was declared ‘the new oil’ and became necessary in processes and decisions across the business. Data couldn’t be restricted to IT - it was needed in Sales, Finance, Marketing and Customer Service to drive decisions and personalise experiences.

By the time we hit 2020, the ‘data engineer’ term was locked in. The term may have only been 9 years old but resumes were popping up with 15 years experience as a ‘data engineer’.

So now we have a two-fold problem:

• two distinct kinds of work

• one popular job title

So how do we untangle this mess? We can think of it as a continuum - from pure data work on one end to pure engineering work on the other:

Image: The data-engineering continuum

While these four roles exist on a continuum with blurred lines - especially at smaller companies where people wear multiple hats - they give us a useful framework for thinking about the work:

Data Analyst
Investigates business questions through data exploration such as "Why did sales drop last month?" or "Which customers are most likely to churn?".

Analytics Engineer
Builds reliable data models that multiple teams can trust. Figures out why sales figures don't match between systems.

Data Platform Engineer
Designs the infrastructure that moves and stores data at scale, enabling both speed and safety.

Software Engineer
Builds the APIs, tools and services that make data accessible to applications and end users.

Every company is now a data company, and as data and AI play bigger roles, we need language that matches reality. Expecting a full-stack ‘data engineer’ is madness. It’s like expecting one person to be both the bar technician who maintains the kegs and taps AND the bartender who serves customers. Both work with the same liquid - but one focuses on pressure, flow and keeping the system running - while the other focuses on what the customer actually wants and how to serve it properly.

The data analyst and software engineer roles have been around for a while and are well defined. However, it’s clear to see that what we have been calling ‘data engineer’ is really two roles in disguise:

• Analytics Engineer

• Data Platform Engineer

The cost of this confusion isn’t just semantic. When you hire a data pipeline expert to build business logic, you get brittle data models. When you hire a data modelling expert to scale infrastructure, you get performance bottlenecks. In larger organisations, it’s time we stopped asking one person to be both bartender and bar technician. The next time you’re hiring a data engineer, ask yourself: do you need someone to build the pipes or someone to serve the data flowing through them?