Opening up the Black Box
Challenge
Nearly a decade ago, a client asked us to build a data management system to process multiple datasets acquired via automated and manual feeds from third-party sources. The system was for internal-use-only and the processing was designed around a “black box” of analytical requirements based upon the client’s proprietary methodologies.
In time, as regulatory obligations around transparency and auditing grew more stringent, it became apparent that the “black box” methodologies would need to be revised; not least because incoming third-party data was now changed, having been processed and reconciled according to new methods.
The internal-use system had provided the client’s research team with powerful analytical capabilities for many years but its outputs were presented in a legacy ‘table’ format mirroring their XLSX files. They now wished to develop more user-friendly data visualisations and a branded interface for charts to use in their published reports.
It was time for an overhaul, and the client asked 542 to lead the way.
Action
The legacy code was approaching end-of-life in terms of functionality, and a comprehensive reworking of the existing architecture was in order:
- New code to replace the legacy application
- Additional tables and data storage areas to support and enhance extensibility
- A complete restructuring of the API
- A new data validation toolkit
- Integration and platform unification
This aligned with the client’s progress towards a data warehouse structure for all of its data, updating and integrating the old datasets into a new tool for analysis and reporting.
Solution
We are adept at multi-disciplinary projects like these; our Data, Development and Design teams collaborated closely and a project plan was soon in place. Weekly check-ins meant that all stakeholders were kept abreast of the project’s progress whilst, internally, our Agile processes ensured that working priorities remained aligned with the client’s needs. Our mission was to transform an end-of-life product into a dynamic solution designed to accommodate constantly evolving analytical needs.
New code to replace the legacy application
The “black box” calculations would have to be dismantled. While there were many core functionalities that could be retained, several new stages were required to provide transparency to the processes of data ingestion, reconciliation, and publication. It was no longer appropriate for this to be a standalone application with siloed data and methodologies.
In consultation with the client’s SMEs, 542’s analysts and developers re-developed the codebase to include data validation and audit logs, reducing the system’s reliance on developers to identify and mitigate data issues. While this increased the complexity of the system, it maximised transparency for the client and helped meet regulatory obligations.
Additional tables and data storage areas to support and enhance extensibility
As the feeds, calculations, and reporting structures were being reworked, we were able to improve the storage and analytical capability of the system. In restructuring the data flow, we made it highly scalable and accessible. Designed for integration with the client’s wider data library it enabled unified indexing, standardised lookups, and more efficient data maintenance.
The creation of new outputs displaying more granular data, (geographical locations and flexible periodicities), as well as the addition of comparison tables (to show data before and after it had been transformed) greatly expanded the analytical potential of what had been a siloed black box.
A complete restructuring of the API
To support the increased capacity of the dataset, and its integration into the data library and our analytical dashboard, the API had to be rebuilt. We optimised both the infrastructure and interface to connect the newly-configured dataset to a suite of analytical and visualisation tools.
A new data validation toolkit
The original version of the app did not allow users to interrogate the proprietary calculations - data went into the black box, something happened, and it came out transformed. It was a fixed, isolated system with only a single stakeholder capable of validating the data. We saw the need to build a data validation toolkit that would automatically check in at key points in the ingestion and calculation process and identify any anomalies.
Integration and platform unification
Having worked with the application, unchanged, for the best part of a decade, our client saw the opportunity for a comprehensive update to meet their evolving business objectives. As the project progressed new opportunities became apparent. The potential of the new system gave rise to many requests for additional features and functionalities that extended the scope of both this application and the wider database. Our Agile processes allowed us to keep pace with our client’s initiatives, adapting priorities and new features on a rolling basis while maintaining a tight grasp on deliverables, timeline, and cost.
Conclusion
No matter how carefully you plan, sometimes a client’s needs change during discovery and even into development. In this case, as soon as we started delivering test environments, our client identified new possibilities and opportunities and we were asked to extend the project scope.
This type of scenario at any stage can present problems but we have found that transparency and openness to continuous adaptation pays dividends.
Allowing the client to customise the software over and above the initial requirements results in an extensible solution that not only reflects their changing needs but also saves them time and money.