Dell Boomi Data Integration and Management Services

During this phase, your team will be prepared to spend ample time in defining the migration design strategy prior to beginning the execution. Since 80% of Data Integration projects run over time, budget, or both, this is a crucial step to your project’s success.

• Analyze and define the environment of the legacy infrastructure
• Determine the connectivity for the migration
• Legacy System Data Analysis (LSDA) is performed
• Define the data model, database reference architecture or data mapping rules
• Develop an extraction strategy for different types of source data
• A cutover strategy is defined 
• Roles and responsibilities amongst the team are determined and assigned
• Communication protocols are established
• Necessary migration tools are identified and allocated 
• The migration strategy and procedures are developed
• Acceptance criteria is defined and documented

In this phase, the existing (legacy) data is extracted from its current source, to be further processed for transformation for import into the new repository or system.

Legacy, or source data, can come from many different places, such as a your current application files, DCS extracts, databases, documents (flat files), web pages (HTML or XML) or emails.   These types of sources are known as unstructured data, since they typically all house data in different formats.

Once extracted into an extract file, the data is parsed, or checked for integrity, exposing any similar patterns, which will aid in the transformation.  The mapping document or functional specs document will also come into play during this step.  Lookup tables from the target destination also aid in the parsing of the data to be ready for transformation, the next step.

Data that’s been extracted from the source is then analyzed and cleansed in the Transformation phase.

In this phase, your data that was extracted from its original source, is analyzed and “cleansed” for accurate loading into the new destination.  Data is checked for integrity and any necessary business rules are applied as the data is processed.  Data Quality plays a huge part in this phase.

There are six parts to Data Quality:

• Completeness – ensuring all data fields are populated that need to be
• Conformity – checking that the data is in the standard form
• Consistency – all data is consistent and non-conflicting
• Accuracy – all data is correct
• Duplication – data is in the same formal
• Integrity – data retains relationship link information

Other areas to look for when performing a Data Quality check:

• Duplicate records
• Irregularities and discrepancies
• Integrity constraint errors
• Missing or invalid data
• Obsolete information

Data Transformation is made up of two elements, data mapping and code generation.  First, the data is mapped from the source to the destination, which determines any transformation of data which must be done.  The code generation piece is an executable piece of software that is run and performs the actual transformation.  The timespan for this phase varies with the size and complexity of your data.  Once your data is through the Transformation phase, it’s ready for the Load phase.

In this phase your transformed data is loaded into your new repository or database, and also the final step in the ETL process (Extract, Transform, Load).  The ETL tool chosen by your migration team to perform the load will ensure consistency as the data is loaded.  Any indexing is typically disabled until all the data is imported, then re-enabled once complete.

As with any major cutover project, it’s important to have a contingency plan, in case any part of the process fails.  Your migration team will have planned for any issues in advance, and will be prepared to take steps necessary to back out and restart the process if necessary.

Once the data is loaded, the Verification step is next, the final phase in the migration process, also known as migration testing.  Data integrity is checked during the Transformation process, but final verification is necessary upon completion to ensure accuracy and a successful migration.

During the verification,checks the data against preloaded data for business logic verification. It compares both the source and the target data, checking for any discrepancies.  If any are found, they are reported, and if possible, repaired.  The tool also provides a dashboard displaying errors, enabling the team to reconcile any issues early on. This is part of the overall test plan, and can encompass several other testing methods that are chosen by your migration team.