Implementing a Champion-Challenger Machine Learning System in the Financial Services Sector

Background

Our team developed a machine learning (ML) framework for a major player in the South African financial services industry. They faced challenges with their existing actuarial models, which utilised traditional modelling techniques. Our objective was to introduce a more dynamic Champion-Challenger system to allow newer, more advanced models to be tested against the established "champion" models in a live environment.

Problem 

Our client needed a system that could facilitate the experimentation and deployment of multiple ML models to continuously validate and improve decision-making accuracy in real-time. This need was identified due to the limitations of the client's older systems and the lack of technical flexibility, which made it challenging to implement advanced data-driven strategies.

The Solution

Our solution comprised three key components:

1. Data Infrastructure Development on AWS

Our engagement with the client coincided with their strategic shift to the cloud, specifically to Amazon Web Services (AWS). This transition was pivotal, allowing us to leverage advanced AWS technologies to establish a robust data infrastructure for ML model deployment. The key components utilised included:

• AWS EMR (Elastic MapReduce): We used EMR to manage Spark clusters efficiently, facilitating large-scale data processing and analysis.

• AWS S3 (Simple Storage Service): As a scalable object storage service, S3 served as the backbone for data storage, ensuring data integrity and accessibility across different stages of the ML lifecycle.

These technologies were selected not only for their performance but also for their compatibility with the client’s existing data systems. The integration allowed for seamless data flow and management, thereby supporting the complex requirements of deploying ML models in a dynamic environment.

2. Machine Learning Model Development

The centrepiece of our solution was a custom-developed data science pipeline orchestration and deployment management tool, designed specifically for the client's computing environment. This package facilitated the experimentation, orchestration, deployment and operation of ML models and pipelines within a structured Champion-Challenger framework. Key features of this development included:

• Model Experimentation and Deployment: We implemented tools for rigorous testing of new challenger models against established champions, utilising live data to ensure real-time validation and comparison.

• Continuous Learning and Adaptation: The system was equipped with features for auto-retraining and monitoring input data drift, which are crucial for maintaining the accuracy and relevance of ML models over time.

• AB Testing Framework: Our solution utilised statistical AB testing to conclusively determine the superior model, allowing them to make data-driven decisions with high confidence.

This framework was not only about deploying new models but also about establishing a continuous improvement process where models could be updated, retrained, or replaced based on ongoing performance data.

3. Operational Integration and Continuous Support

Integrating this ML framework into the client's operations involved several strategic steps to ensure minimal disruption and maximal utility:

• Seamless Integration: The ML models and the Python package were designed to integrate smoothly with the client’s existing systems, allowing their team to adopt the new tools without needing significant changes to their operational processes. The system can be used on both their AWS cloud environment as well as their on-prem big data platform.

• Training and Documentation: Comprehensive training sessions and detailed documentation were provided to empower their staff. This initiative aimed to enable them to manage the ML system independently, ensuring sustainability and scalability.

• Ongoing Support and Iteration: Anticipating future needs, the system was designed to accommodate ongoing adjustments and expansions. Our commitment to the client included continuous support and iterative enhancements to the system, adapting to new challenges and opportunities as they arise.

The goal was to not only provide a robust ML solution but also to transform the client’s internal capabilities, fostering a culture of innovation and continuous improvement within their teams.

Outcomes and Benefits

The new Champion-Challenger system provided the client with several advantages:

• Enhanced Decision-Making: By continuously testing new models against current ones, they could adopt more accurate and efficient predictive models, thereby improving their decision-making processes in critical business areas.

  
  

• Scalability and Flexibility: The cloud-based infrastructure allowed for scalable and flexible model training and deployment, accommodating the client’s growth and the increasing complexity of data.

  
  

• Operational Efficiency: The centralised Python package reduced redundancy and streamlined the development process, drastically decreasing the time from model development to deployment.

Challenges and Considerations

Despite the successes, the project encountered challenges primarily related to the integration of new technologies with the client’s legacy systems. The initial lack of technical specifications and the rigid older systems presented significant hurdles in adapting new solutions. However, these were progressively overcome through strategic adjustments and enhanced collaborations between our team and their internal technology providers.

Conclusion

By adopting this advanced ML approach, the client is well-positioned to enhance its operational capabilities and maintain its competitive edge in a rapidly evolving market. This project underscores Spatialedge’s expertise in navigating complex technical landscapes and delivering innovative solutions that drive tangible business outcomes.