“I'm not dumb. I just have a command of thoroughly useless information,”
Calvin from the comic strip Calvin and Hobbes once famously said. This is a situation that most IT users will identify with as they try to make sense out of the reams of data they have to sift through on a daily basis whilst adding updates to the system.
What is needed is a proper handling of this information using metadata management. Forrester defines metadata management as information that describes or provides context for data, content, business processes, services, business rules and policies that support an organisation’s information systems. Metadata management ensures visibility and delivers the control that is required to manage change in a complex, enterprise data integration environment.
Why don’t you see what I see?
It is a given that most enterprise data warehouse (EDW) projects typically extend from 18 to 36 months. EDW projects are complex and supported by numerous, meticulously prepared documents, such as the traceability matrix, ETL specs and design documents. These documents are crucial and require regular updating at every phase of the project. Quite often, users must collaborate across different time zones, on different areas of the project.
Potential Problems
What most IT project managers fail to consider, are the imminent changes that can occur over the duration of the project. This includes alterations in project requirements, modifications to the data architecture during the design phase, to replacement of resources such as business analysts moving over to different projects. In such a situation, IT users tend to get a distributed view of information, which conveys a lopsided picture.
Even worse, if these changes are mismanaged, it could give rise to several issues including:
- Rework or the project going haywire
- Multiple versions of the truth
- Redundant data across numerous systems
- Source systems out of sync
- Difficulties with delivering analytics
No single user has end-to-end visibility and ownership into the data flow of the project. Therefore, it is essential to maintain a consistent understanding within the team, as well as with the wider company stakeholders such as management and analysts. Tracking and evaluating changes and understanding their impact during the various phases is critical to the delivery of the project.
Impact on Business and IT
From an IT and business perspective, there is a strong case for building a metadata management system.
Business Drivers
- Data/ Regulatory governance
- Support mergers and acquisitions
- Improve risk management
- Process optimisation
- Address rapid changing business needs
- Ensure information quality
Technical Drivers
- Lower development and testing costs
- Reduce risk of changes and updates
- Reduce support costs for business users
Gaining a 360 Degree View
Mastek’s metadata management solution offers business and technical users the ability to quickly find, understand and manage information that supports their daily business operations. It provides answers to the difficult questions that users want to resolve, as detailed in Figure 1.
Figure 1: What Users Really Want to Know
Users benefit from a single view of the data.
Full Audit Trail
Tracking data creators, including edits, approvals or deletions to the information, at any point in time
Data Lineage
Origination of the dataset, date and schedule, and period covered, contents, change impact, etc.
Wiki-style Functionality
Easy visualisation of data flow and management
Search and Storage
- Ability to carry out fast and flexible advanced searches
- Store each version of a data element using relevant timestamps
Data Enrichment and Connectivity
- Ability by authorised users to enrich and expose data to EDW
- Link to technical metadata
- Management of reference data
- Common vocabulary of business terms
Usage History
Identifies teams that have used the dataset, the purpose and the level of success achieved
Figure 2: View of Mastek’s High-level Metadata Management Matrix
Depending upon the unique IT and business requirements, customers can decide to opt for an off-the-shelf or bespoke, metadata management solution.