No Surprises on Moving Day: Governing Data Migration for AI Readiness at Scale

Large-scale data migration is one of the most common reasons public sector AI projects stall, as moving to a modern cloud platform does not make data AI-ready on its own. What makes the difference is the governance work done before, during and after the move.

In this session from Digital Leaders AI Week 2026, our data scientist and engineer Maja Strawinska uses moving house as a way to explain how to plan and govern a migration of national-scale, citizen-level data. In a similar session to that delivered earlier in the year at the Public Sector Connect Data and AI Summit, she covers why organisations move off legacy systems, how to choose an approach using the 6 R’s, how to decide which data to leave behind, why metadata matters, the security and approvals needed and how to bring stakeholders with you. She also explains the 1-10-100 rule: fixing data problems at source costs a fraction of correcting them later, and far less than dealing with a failed migration.

The session runs for around 17 minutes and is aimed at digital, data and technology leaders responsible for legacy modernisation or AI readiness. You can watch it below.

‍

For the evidence behind the talk, read our whitepaper, The Hidden Cost of Legacy Data Systems in UK Public Services.
‍

‍

‍

Video transcript:

Hello all, and welcome to No Surprises on Moving Day, which is all about governing data migration at scale, where I take you through the steps to make your migration stress-free in a bite-sized way.

Firstly, let’s get to know me and Butterfly Data. I’m a data scientist and data engineer, experienced across multiple government agencies, with two years at Butterfly Data under my belt. In my own time, I’m a proud Scouts leader and adventurer. At Butterfly Data, we are SMEs in dealing with national-scale, citizen-level data across the public sector, with expertise in data quality, visualisation, insights, migration, engineering, data science, and much, much more, all about data.

Let me introduce the flow of this presentation. First, I’m going to introduce you to the house move analogy, then the migration process as a house move. Next, the new home and the foundations that we need to have there before we move.

So, why use an analogy? Using an analogy to explain complex technical things makes these ideas a lot more digestible and relatable. Just as you wouldn’t move house without a plan, you don’t want that experience of missing that box, do you? You can’t necessarily migrate data at scale without a governed moving list. In the public sector, we manage millions of citizen records. The moving day stakes are so much higher than just a broken lamp. The old home is our legacy data estates, technical debt, the clutter. The new home represents the modern cloud platforms available to us.

You may be thinking, why move now? There may be [many] reasons to move your home, but in terms of migrations, there are three overarching reasons. Our legacy systems tend to be high maintenance. They’re poorly maintained. We spend thousands fixing them. There may be flood risks, increasing structural damage. There may be leaks, water damage. The electricity might need redoing. In terms of migration, external factors tend to force the move. So the legacy hardware could be reaching end of life. There could be security vulnerabilities that are piling on. Compliance regulations could be shifting, meaning we no longer comply with them. And vendor licensing costs could just become too much.

Another reason is that our house could be too small. In our first home, we may have those overflowing closets, stuff from our trips tucked away in the hallways. In reality, the floor plan just doesn’t support our growing family. So thinking about migration, there are operational constraints. Databases may hit physical storage limits. Queries may crawl during peak traffic hours. And the rigid legacy architecture may just not be able to digest the large volumes we’re getting from modern-day systems.

And our final reason is new tech. In the new home, we might have the smart features, the solar panels, the automated lights when we walk through the door. We might have electric charging for our vehicle outside our doorstep. So in the migration, what does this mean? These are the feature upgrades, upgrades into cloud-native perks. We have real-time streaming analytics. We may have automated disaster recovery, serverless scaling, and plug-and-play AI and machine learning integrations, which in reality we just can’t do on our legacy systems.

Now, I’m going to guide you through the house move in the migration process.

When assessing our options, we have to ask ourselves, what do we need? Do we need to build a new home from scratch? Yes, that would provide us a bespoke platform, but places the entire burden of maintenance just on our shoulders. Or can we move to the apartment block, the cloud? In the public sector, the apartment block is often the more governed and scalable choice.

When we discuss migration, we talk about the six R’s. And let’s just quickly run through what each one of those means. A refactor is remodelling the new house. A rebuild is a purpose-built house. If we retire, we demolish the old house. If we retain, we do nothing and just become a squatter. If we rehost, this is our lift and shift approach. And if we replatform, we are lifting and shifting with new furniture.

So how do we decide what to pack? We need to do a check and assess what we currently have. Do we need to take the bust bike tyre in the garage to the new home? Or can we recycle it before we get there? Discovery is a very vital part of migration. During this time, we identify all that we have and create our do-not-pack list of ROT data. So our redundant data, our duplicates of the same citizen across different legacy silos. Our obsolete data is data that might have surpassed its legal retention period. Trivial data is system logs or temporary files that add no value to the future of the digital services.

Now, what do we want to take? Clean and validated, accurate data that is up to date, with information that can be trusted. We want it to be governed, data with clean lineage and ownership, the labelled boxes. And we also want to be able to plug our data into AI models straight away. So it needs to be interoperable.

You may be wondering what some of the common mistakes of discovery might be. The lift and shift approach can cause pain much further down the line. If you move the clutter from your old attic into a brand new smart home, all you have done is pay to store rubbish in a more expensive location. Governance guides our do-not-pack and must-take lists. It’s a filter that ensures only the high-quality, AI-ready data makes it into our new environment.

So let’s talk about labelling our boxes. If we don’t label them correctly and just call it “stuff”, how will we be able to find it in the new home? Will you remember what stuff you put in there a month ago, two months ago? We’re really risking important records ending up in the garage while some of our junk might just take up space in the kitchen. So the label, the metadata, tells us what the data is, where it came from, and who’s allowed to see it.

In order to take this correct action, there are a few things that we need to do. We need to define our schemas: create the floor plan for the new house so every box has a designated spot to end up at. We want to create data dictionaries: the inventory list that ensures the living room in the old house matches the living room in the new house. And finally, we want to ensure strong governance, ensuring our data is not just moved safely but ready to power the next generation of our digital services. We don’t want that nightmare where we have to open 20 boxes to find our kettle, and this is just how we make sure of that within the data world.

So next I’m going to talk you through our building regulations. Are there any rules that we have to abide by in the new world? One of them is soundproofing. We don’t want a home where we can hear the neighbour’s 4am alarm. We want the data to be isolated on private or public cloud, preventing data leakage. We need to ensure encryption at rest and in motion.

We also want access control. Does every member of our family or friend group need a key to our home? Can we provide temporary access to the home to water the plants during summer when we may be on leave? So when talking about this, it’s all about identity and network security: role-based access control, least privilege, blast radius and zero tolerance. Does everyone need a key to every single room?

And finally, let’s talk about our structural integrity. Have we had the house surveyed? Are all the approvals in place? We need approvals from the Design Authority. We need approvals from data governance and other stakeholders, and we need to make sure that we have our data sharing agreement in place. [And to this,] there is a little routine: authorise, approve and iterate. We need to make sure we do all these steps and repeat and repeat in order to get the best outcomes.

Next, I’m going to talk briefly about the family and our stakeholders. In every large-scale public sector project, there is a family of stakeholders who are deeply sentimental about their legacy system. They might have spent 20 years learning the quirks of that specific database. So what is the old house? The old house is comfort. The legacy systems are comfortable, just like your own home. People know their quirks, know that wonky step and know how not to trip walking that way. But the new home feels different. The light switches are in new places. The heating is automated. The spaces feel bigger, possibly empty.

So to do this move correctly, we need to move the family, not just the furniture. We need to address the [new config], our knowledge that the new home feels different, and make sure that they know that. We need to manage the human impact. We don’t just show them around the new home. We want to involve the family in the architectural phase, making sure they feel part of the team at every step. Next is the training. We need to help the workforce understand that the smart home features and automation aren’t there to replace them, but to make the housework, or data processing, easier, smoother and more streamlined.

Now we are in our new home. So we need to talk about preparing the foundations for AI. Just like you can’t install a smart security system if a house doesn’t have electricity, you can’t run a large language model if your data is unstructured and unverified. You don’t just want to move for the sake of it. You move to make room for new things: AI, machine learning, automation.

You need to make sure that the flow of water is there, so the data pipelines must be consistent and clean. If there’s muddy water at the source, the smart fridge might break. We also want to ensure high-quality input. AI is only as good as the data it’s fed. You’ll hear “garbage in, garbage out” referred to quite a lot in the world of AI. We also need to make sure that ethical governance is set. So, are the smart home sensors and AI algorithms unbiased and secure? We also need operational readiness, moving from legacy batch processes into real-time digital services.

So you may come across something called the snagging list, and this is basically prevention over reconstruction. Fixing a foundation after a house is built costs 10 times more than surveying the land before breaking ground. Core data governance works the same way. It isn’t a delay, it’s a 10 times cost-saving measure.

So I’m going to break this down for you. It follows a 1-10-100 rule. £1 is prevention: validating the schemas, labelling the data and enforcing hygiene before moving. £10 represents correction: paying the specialists on inspection day to track down the broken APIs or the unmapped fields. And £100 represents failure: the system downtime, broken AI models and interrupted services post-migration.

In [house] building, a snag may be a loose tile. In data, it’s a corrupted record. Without early governance, your snagging list becomes a structural failure. Fixing one downstream defect causes three more to pop up in analytics and AI models. In government, a failed migration means citizens lose access to vital services. Governance ensures moving day is quiet and non-eventful. Modern cloud platforms allow teams to test and learn on [sample] datasets, validating pipelines in a sandbox before reaching the point of no return. It’s important: spending 1% upfront on pre-migration checks avoids 100% of a rebuild later, protecting public funds from steep premiums and a lot of technical debt.

Fantastic. Now we’re going to discuss moving in. Firstly, I want to talk about some cutover strategies. So when do we exchange contracts? This is the point of no return. The cutover strategy is the digital equivalent of exchanging contracts and officially changing your address. This is the service transition, the point of no return. Formal Design Authority approval and stakeholder sign-offs validate that the new environment is ready, triggering the live cutover with full governance backing.

Redirecting the flow: remapping the users and processes. Changing your house address requires updating every single service and utility provider to maintain continuity. In a data migration, this means identifying all dependencies to seamlessly remap users, access controls and business workflows to the new solution. Mapping these operational pathways before go-live ensures that daily processes transition cleanly without breaking any downstream operations.

I’m going to take you through a quick summary now. Decluttering and data preparation: audit the attic. Use data discovery to leave the ROT data behind, and separate our high-value operational data in the living room from compliance archives in the attic. We need to label the boxes: implement rich metadata up front so the new environment is organised, searchable and governance-ready from day one. We need to check the pipes: validate the APIs, the pipelines and the underlying architecture to guarantee structural integrity before go-live. Then we need to hand over the keys: use active change management and thorough documentation to transition our stakeholders, our family, smoothly into the new system. We need to have an elastic infrastructure. Unlike physical houses, cloud environments leverage automation to instantly build temporary space for spikes, auto-scale on demand, and tear down when no longer needed.

I just want you to be aware, this will be live by the time this presentation is up. It’s key findings from Rewiring the State, DSIT’s State of Digital Government Review and more, all in one paper. It’s all about the hidden cost of legacy data systems in UK public services.

Thank you.

‍