Automated CSV Data Comparison & Advanced Field-Level Validation Utility

To satisfy the validation requirements and ensure processed data accuracy, QA Test Engineers within an organization often face the challenge of completing major regression tasks that require detailed comparison of multiple pairs of CSV files. These files can be very large at times (thousands of rows & hundreds of columns per file). Testers also need to detect the presence of any duplicate records within those files along with the deltas (extra or missing records on both sides of the CSV pairs). Beyond those challenges the automation utility needed has to be reusable & exposable to approved external automation tools within the ecosystem. This session will discuss a solution that was developed to solve these issues by performing advanced records segregation and field-level data validations for various data types, such as numbers and strings. This automation utility is usable as a library or API, therefore allowing the organization to cut testing costs for regressions. Overall hundreds of thousands validations can be performed within few minutes along with multiple comparisons reports. The future enhancements will focus on extra-large CSV files handling and parallel processing. One takeaway that will be learned is that a gigantic problem can be solved by breaking it into layers and flows.