What is data migration?

Data migration is a process that almost every business undertakes, moving data between different storage systems, databases and formats. The data migration market has grown by an estimated compound annual growth rate of nearly 18% from 2017 to the present, hitting $11.5 billion in value. The forces behind this are the need to optimise and manage ever-higher volumes of data, exponential cloud adoption and the growth of the everything-as-a-service model, regulation and a need to maintain legacy data.

Conducted properly, data migration will enhance performance and competitiveness, and may reduce cost. There are many different types of data migration.

Types of data migration:

Let’s consider each type of data migration and what its benefits are.

• Storage migration

Storage migration is moving data from one location to another, from one medium to another. The purpose is to deliver higher and faster performance and reduce costs. Organisations may want to scale more cost-effectively, or they may want improved backup and disaster recovery capabilities including snapshots and cloning. Some data migration projects are just about moving from one storage system to another.

• Application migration

Application migration is more complicated. The data migration process here is about moving to a new environment, which could be to a new data centre, or from on-premises facilities to a colocation centre or public cloud. Each application is likely to have associated databases, folders and installation files that organisations must also relocate. Liaison with software application vendors is usually necessary to ensure optimal performance in the new environment. Getting application migration right delivers better costs, scalability and agility in the face of changing demand.

• Cloud migration

Cloud migration is the movement of data and applications from an on-premises environment into the public cloud or private cloud. Alternatively, it can be the migration of data and applications from one cloud to another. In many cases, moving applications requires data migration as well.

Large organisations make substantial savings in IT operating expenditure by transferring their data centre facilities to a public cloud provider. As well as moving from a CapEx to OpEX model by removing on-premises environments, the main drivers for cloud migration are increased organisational agility and innovation, access to cutting-edge cloud-based applications and services and the ability to scale quickly. Businesses also benefit from more advanced security and higher bandwidth connectivity.

• Database migration

Data migration is at the heart of database migration. This is about upgrading a database engine. When complete, the data in the source databases resides in the new, target database, although the database migration process may have restructured it. The source database is then deleted. The new database may well have a different management system. Organisations use a database migration system to achieve the necessary data migration process, reformatting datasets to work on the target database.

Database migration should make data more useable, enabling more people to extract more value and insight from it within an organisation, leading to better decision-making and greater ROI. After data migration to a new system, applications should work better and become more effective.

• Business process migration

Business process migration is the transfer to a new environment of applications, workloads and data relating to business processes. The project can cover the full gamut of business applications – from customer-facing systems to logistics, supply chain, manufacturing and process optimisation. This type of data migration is frequently triggered by mergers and acquisitions activity, as well as the constant drive to re-energise and optimise processes or to reorganise for greater efficiency.

• Data centre migration

Data centre migration is where an organisation outsources all its data centre hardware to a public cloud provider. The infrastructure includes data storage which maintains critical applications, along with servers, routers, storage devices, switches and networking equipment. A data centre migration will include transfer of all internal and external process software to a public or private cloud. The challenges this gives rise to persuade some organisations to use hybrid or multi-cloud infrastructure, migrating only some applications or databases. They will keep sensitive data, or legacy databases and applications that were not built for the cloud, on-premises.

Data centre migration can also be the transfer of infrastructure to a new geographical location, or to a newly built facility on the same campus.

The data migration process

Organisations must commence their data migration by drafting a strategy. A critical decision is whether their data migration project is to be a single, mass-migration or “big-bang” event or if they will trickle data through in phases. The single event approach is faster and less complicated.

If organisations opt for more than one data migration process, each can be executed sequentially or concurrently depending on requirements.

An organisation with personnel trained in systems administration, networking, programming and web development can draft the strategy internally. Many businesses will, however, engage data migration specialist consultants. They may also choose from the wide range of data migration tools from hyperscalers and third-party software vendors.

The data migration process should comprise:

• The strategy and setting of objectives
• Planning how and when the organisation will execute the strategy
• Data-mapping and inspection to work out the location, quality, format and source of data and to determine who uses it and how
• Deciding which data the organisation must migrate
• Backup provision
• Design of the process including data migration-testing, acceptance criteria and recruitment of specialists such as ETL developers, data engineers, systems and business analysts
• Execution of migration
• Checking and validation
• Deletion or decommissioning
• Continued monitoring

Data migration strategy and best practices

Data migration must be well-planned to run smoothly and deliver ROI swiftly. A data migration process should follow the established ETL (extract, transform, load) methodology. ETL tools handle the complexities attached to large-scale data migration, facilitating the integration of diverse data sources and multiple application platforms.

The following are best practice steps for successful data migration:

• Convene a dedicated data migration team with the right specialists to handle and direct the project.
• Establish business requirements, timelines and explore all potential effects including toleration of data loss or delay
• Divide a large data migration project into segments for ease of management
• Improve data quality before migration to avoid repeating old problems
• Select the right amount of data and no more, using right-sizing tools if necessary
• Assess the source environments and the target for operating systems and compatibility
• Ensure the data has already been cleaned up before transfer
• Profile all data in the scope before writing mapping specifications
• Back up data to insure against any loss during the process
• Test the migration from planning and design through to execution and maintenance to ensure the success of the data migration project
• The source system should only be decommissioned after the data migration process is definitely a success
• In the event of failure, a roll-back will be required without downtime as the previous system will still be operational

Data migration risks

Almost all data migration projects come with risks which teams should be aware of. They include:

• Misconceived data migration projects through poor planning or inadequate access to expertise
• Large datasets overwhelming networks
• Incompatibility of data and workloads with new environment leading to poor performance
• Loss of data
• Exceeding budget
• Delays – some data migration projects last years, rather than months as they encounter problems
• New security vulnerabilities

Data migration challenges

Common challenges that data migration projects encounter include:

• Poor planning, slowing time to ROI and business goals
• Poor communication
• Lack of knowledge in how to specify the data migration process and data modification
• Complexity and delay arising from constant need to synchronise when using trickle data migration approach
• Problems from migrating applications with dependencies
• Drift from business objectives through poor management
• Inadequate choice of data migration software
• Absence of contingency planning

In conclusion

The modern data-driven economy now demands organisations make full use of the information they possess. The need to upgrade databases and to optimise rapidly-expanding volumes of useful data mean data migration is becoming a regular task for all organisations regardless of size or sector. Gartner, for example, predicts end-user spending on public cloud services will increase by a fifth in 2020 and reach $600 billion worldwide in 2023.

Yet without the right approach and the right tools it is easy for poorly-planned data migration processes to exceed budget and take longer than expected to deliver greater efficiency, cost-reduction and access to innovation and new applications.

Database migration tools

Some tools are designed for on-premises alone, while others are specific to the cloud. Among commonly used data migration tools are:

Oracle Data Service Integrator

Informatica PowerCenter

IBM InfoSphere

CloverDX

Microsoft SQL

IRI NextForm

Xplenty

AWS Data Migration

Azure DocumentDB

Talend Open Studio

Cyberduck

Zerto

To find out more about data migration, visit: https://www.telehouse.net/products/migration/