Tidying up as a chance

How to improve data quality when migrating to the cloud

The migration of on-prem data to the cloud offers the opportunity to systematically cleanse and reprocess data and, if necessary, enrich it with external data. Therefore, it is important to plan the data migration at an early stage and to execute it properly. This way, legacy data can be sorted out in a targeted manner and all important data can be completely merged.

In our article, we show five steps from practice with which companies can automate data flow processes and successfully migrate their data to the cloud. With the help of tools that can be integrated into the process, the quality of the data can also be improved during migration.

Organising removal helpers, terminating the tenancy agreement, furnishing the new flat – a move does not only involve logistical issues. Early, targeted planning can help you keep track of everything. At the same time, the move can also be used to get rid of unnecessary ballast, because every item that is no longer needed does not have to be transported to the new home.

Just as with moving house, migrating data can kill two birds with one stone: With strategic planning, data can be cleansed in such a way that duplicates and unnecessary data ballast are disposed of before the transfer. The steps described below help to plan the data migration process in advance.

There are many reasons why software systems have to be renewed and data transferred. If the technological basis is outdated, new functions are desired or systems are merged, then the data must be transferred from the old to a new IT system. Merging different companies or divisions can also involve merging data. With the current trend towards cloud computing, transfers of local on-premise data to internet-based cloud servers have been increasing recently.

When preparing for data migration, it helps to draw up a schedule in advance. In practice, a five-step process has proven successful. If these are followed, the data can not only be migrated cleanly from A to B, but also optimised at the same time.

Picture: Pixabay/CC0 adapted by TOLERANT Software

Step 1: Inventory of current data (data profiling)
Regardless of the reason for the data transfer, an early inventory should be made in the first step of the migration process. This clarifies what data is involved, what weaknesses and data quality problems exist and where they are located. The diagnosis of the weak points shows, for example, where there are errors or duplicate content and which data needs to be cleaned, improved and merged.

Step 2: Determine migration requirements
The second step is to plan the scope of the data migration. The migration requirements provide information about what data will be needed in the future, by whom and where, and how and for what purpose it will be used in the future. During this planning phase, rules and standards must also be defined to guide the different stakeholders of the data migration project. This includes data standards as well as current and future business rules for data use. Finally, responsibilities are also defined in the planning phase so that all project stakeholders know who manages the information, who is responsible for data quality, access and use, and who has the final say in case of ambiguities. Since the entire company is often affected by the data migration, it should be accompanied by a professional change management process. Ideally, the responsible change agent gets all stakeholders on board and involves them in the data migration process from the beginning. In the planning phase, precautions are also taken for risk management so that nothing goes wrong during the transfer process. For example, it must be ensured that all data remain accessible during the data transfer in case they are needed for an audit.

Step 3: Create migration flow and data quality clean-up
The third step is to automate the migration processes. Here, so-called ETL tools for data flow modelling (ETL stands for Extraction Transformation Load) and tools for job automation are used. The toolbox can be expanded to include useful data quality tools that can be integrated into the process. These special tools improve the quality of migration data by identifying and cleaning up errors. Suitable tools can be used, for example, to check whether duplicates exist in the master data. The data quality software enriches and completes data with external sources. It also merges data from different sources (e.g. also accounting and CRM). The tools standardise and check names, bank and postal time numbers as well as addresses and match them with relocation addresses. With the help of the tools, data entry errors are successfully detected and eliminated even before the actual data move, so that only clean data is transferred.

Step 4: Create the target format
Quality optimisation is followed by the fourth step of data migration. Now the input data is transferred into the structure of the target system and prepared for import into the target system.

Step 5: Test and carry out data migration
In the fifth and final step of the data migration, the migration process is tested to detect errors and possible problems. Only after this test does the actual data transfer take place. Thanks to the preceding meticulous planning, the data can now be moved from A to B fully or at least partially automatically.

Would you like to learn more? Then we look forward to receiving your e-mail at info@tolerant-software.de or your call at +49 711 400 4250.