The Challenge 

Confused.com is the first and longest-running UK comparison site, making it easy to compare a wide choice of insurance types and financial services from a range of trusted providers.

As an online, data-driven business, Confused.com had invested significantly in its data science capabilities and platform. However, the company needed help implementing the processes and technical infrastructure required to produce machine learning models and derive value.

“Blending the teams at BJSS and Confused.com has led to incredible results being achieved in just four months. The combined efforts of the teams allowed them to build a new capability in such a short space of time, in which Confused.com can carry out data science. At Confused.com, we always want to ensure we are evolving with the demand of our customers as helping them is our priority. It’s amazing how much we have all learnt, and I think I can say that for both sides it's been a brilliant experience."

Nick Sharp - Head of Data, Analytics & Data Science, Confused.com

The Approach 

BJSS was engaged to develop a service in Confused.com's data platform and create features that could be passed to a model for inference and used to train new versions of the model.

Drawing on our AI/ML expertise and our experience providing managed services for data and analytics, we performed a maturity assessment of Confused.com's data science and MLOps capabilities to develop an understanding of the client's existing strategic environment, priorities and vision.

This assessment identified the need to include a Data Scientist in our team profile for the best outcomes. BJSS' Data Scientist then worked with Confused.com to assess the existing model, features, and validation techniques and help define the approach for a new model. This resulted in identifying a new set of features for model training, using a more appropriate algorithm, and considering different metrics for model evaluation.

The solution was then developed using Azure Databricks (a new solution for Confused.com), delivering it into production. The solution processes data from various sources, including SQL databases, API calls, and XML data. The service first creates features that can be passed to the model for inference and training of new versions of the model. The model then generates the inferences, which are saved to the data lake before being provided to the business to use.

The team also accounted for data labelling and developed this into a new Feature Store that fed into the data pipeline. The pipelines use horizontally scalable services and generate the features required by the model to update the labels. We monitored these for their performance and to ensure services such as Databricks were performing within agreed NFRs.

For Confused.com, issue management and resolution processes were designed from the beginning of the project. As it is expected that models will change over time, we ensured that the service could detect drift, retrain models, and redeploy models with minimal intervention.

“Impressive results, and it looks like we've learned a lot from you…Bravo!"

Andy Brockway – Chief Technology Officer, Confused.com

The Outcome 

BJSS delivered the capabilities to facilitate the building and productionising of new models for Confused.com. The solution utilises the Confused.com approved technology stack (Azure, Databricks, Data Factory, Azure DevOps and Delta Lake). The target system for inferences generated by the model is a SQL database used by Adobe Campaign. All these systems were identified by BJSS during discovery to de-risk the delivery of the model.

As the solution is AI/ML, BJSS monitored the model in production, ensuring the data provided to the model and the inferences generated by the model were within the expected operating range. To support this, BJSS developed a framework enabling detection of data drift for features being passed to the model and detection of model drift based on the accuracy of inferences generated. Should either of these checks indicate that drift has occurred, log messages are created and events raised with the appropriate teams to address the issues; this could include retraining and redeploying a new model version.

With these capabilities, Confused.com can now scalably create and implement new machine learning models, as well as increase maturity within the Data Science team to work collaboratively on models to solve future business problems.

“Collaborating with BJSS has allowed us to enhance our ability to use innovative technology to better serve our customers. What we have learnt and achieved with them over the past few months is incredible. At Confused.com, our customers are at the forefront of everything we do and this work will change the way we use data models to solve customers’ problems in the future. Huge thanks all and it is great to have a powerful new capability as a result."

Steve Dukes – Chief Operating Officer, Confused.com