What challenges does your organization face in addressing Data Quality issues? Last Friday at our IT Leadership Group meeting Divakar Khanna, Director at Optum Data Solutions and Ansh Sarkari, Principal Engineer at Optum Data Solutions lead an informative discussion on the topic, Open Source Technology Solution for Addressing Data Quality. A huge thank you to Optum for hosting us!
We kicked off the meeting have a few members share how they are currently dealing with data quality issues. While we heard various answers to that question, we felt a common theme when sharing what some of their challenges were with their current model. A few that were discussed were the inability to monitor it, unable to control it, and owning the data.
As Optum started to evaluate their options for DQ they knew they wanted to be proactive instead of reactive, operational, and high performing. While each theme had risks and needs, they found that they needed to provide a pluggable DQ Rules Engine compatible with all data formats. Some of these offerings would include:
- Best performance (comparable tor better than industry tools)
- Global DQ rules repository
- Inbuilt results in dashboards/repository
- Open source hence no vendor lock-ins or cost
- DQ rules scheduling flexibility
- Easy user interface for business users/analysts
- OSFI infrastructure
After evaluation, they found their DQAF Rules Engineer was their best option because it was multi-tenancy, reusable – meaning less rework. Not to mention, because of the open source it meant there were no vendor lock-ins.
As their DQAF Rules Engine has been live they are currently in the process of updating their features and capabilities including; user administration via UI, multi-column business rules, automated ServiceNow Tickets, embedded schema browser in UI, and standard results dashboard.
So, what’s next for Optum? In the near future, Optum is looking to add capabilities to reconcile between sources/tables, integrate with ServiceNow ticketing systems to increase automation, Kafka streaming compatibility, integrate DQ job scheduling, rules results, and rules management in one UI.
Again, a huge thank you to Divikar Khanna and Ansh Sarkari for sharing their experiences with Open Source Technology. If you were unable to attend and are interested in learning more, please check out the slide deck, here.