Today’s organizations are dealing with growing volumes of data. They not only need to manage their own data, but also the data of customers and business partners. Many organizations are turning to data warehouses or data hubs in order to better manage their vast amounts of data and get better insights into their business operations. Generally, this will involve a great deal of complexity to reconcile conflicting data between systems of record and to reformat data to common formats. As a result, the lineage of data can become lost through several of these complex data integration processes.
With the ever-growing focus on data privacy practices, you cannot risk being left in the dark when it comes to how and where your data is being used. Laws such as Sarbanes-Oxley and HIPAA can leave your organization exposed to massive fines and criminal prosecution if data is lost, reported incorrectly or released to unauthorized parties. Additionally, incorrect or lost data can jeopardize relationships with business partners or customers who have entrusted the use of their data to you. A well-designed data integration auditing framework can be a key component to ensuring data integrity and security as data moves across your IT systems.
The following are some questions you should ask when designing an auditing framework:
- What regulations on data retention and personal information affect your organization? Chances are your organization is required to keep track of how your customers’ and partners’ data is used, and that requires knowing where the data in each of your systems comes from and which process moved that data.
- Are there any sanity checks that should be performed after moving data? Often, business owners want to make sure any moved data is valid and will want to compare record counts or sums against a trusted source system to verify the numbers look correct. An auditing framework should support running these checks automatically.
- How much flexibility is needed for restarting aborted job flows or running partial job flows? Operational data flows can abort due to bad data or hardware errors, and complex job flows need to be easily restarted from the point of error by utilizing job statuses. Additionally, running partial job flows should be supported as business needs dictate.
A solid data integration auditing framework should be a key component to every organization’s data stewardship and security strategies. It not only helps IT staff with maintaining data feeds by providing job statuses and an easy way to restart jobs, but also provides transparency to business owners on where specific records were pulled from and the results of post-execution validation checks.
Having quick access to this information is a good idea for every organization, but it should be a key requirement for organizations in industries where there is a high amount of scrutiny on data security and integrity practices. Healthcare organizations can utilize record-level auditing to verify where data in reporting systems is sourced from and prevent HIPAA disclosure violations before they happen. Additionally, public companies can use a combination of job auditing metadata and validation check results to ensure that data in their financial reporting systems are complete and correct, avoiding Sarbanes-Oxley violations from publishing inaccurate financial statements.
If your organization is currently building out its data architecture or does not currently implement some form of auditing, consider including an auditing framework for your data integration processes. It will give you more transparency on where and how your data is being used, and can prevent you and your organization from having to deal with legal and contractual violations resulting from improper use of data.