Blog

The Challenges of Predictive Maintenance – Key Metrics and Considerations

In the previous instalments, we have looked at what predictive maintenance has to offer and the challenges surrounding the data. In this final part, we will look into how to assess the success of implementing Predictive Maintenance, exploring key metrics for the evaluation of models, gaining business buy in and briefly look at other factors to consider.

The Challenges of Predictive Maintenance – Key Metrics and Considerations
28 February 2025

To assess the effectiveness of predictive maintenance models, we need to go beyond a simple definition of accuracy. This is because how we evaluate the models has a direct impact on maintenance and engineering teams if we start to generate too many false positives in our predictions.

Precision – This looks at measuring the ratio of the true positive predictions (the actual failures) to all positive predictions (both true and false). This metric is critical because a high number of false positives can quickly start reducing trust in the model. If engineers are sent to fix something unnecessarily, they will lose confidence in the system which can reduce the overall effectiveness of the efforts.

For example, in refrigeration systems, if the model incorrectly predicts pack failures too often, engineers may attend to check and replace components that are actually functioning correctly, not only wasting resources but also disrupts operations unnecessarily.

Recall – Recall looks at the models ability to pick up on actual failures. It calculates the ratio of true positive predictions to all actual failures (both detected and undetected by the model). So whilst precision minimises attendances to fix issues that don’t exist, recall ensures that legitimate failures are not missed.

For example, in a HVAC system, a low recall rate could mean that the model fails to detect a failure in time, leading to unexpected downtime and potential discomfort for building occupants, so ensuring a good recall rate helps present costly and disruptive breakdowns.

Accuracy – This is a term widely used for assessing how well a model performs. It looks to represent the percentage of correct predictions out of all predictions made by the model. It’s great to use as a broad measure of performance but can be very misleading if the dataset is imbalanced as is likely the case seen in part 2 of this blog series. For example, if the failures are rare, a predictive maintenance model that always predicts no failure will have a very high accuracy but will fail to deliver any value for predictive maintenance. For example, in industrial refrigeration, if failures occur only once every few months, a model that predicts “no failure” every time might show 99% accuracy, however this would be ineffective because it fails to provide useful insights into potential issues that need attention.

Assigning Relative Costs to Predictions – One of the most important but overlooked aspects is the cost associated with each type of prediction. We can assign costs to true positives, false positives, true negatives and false negatives in our predictions. By assigning relative costs to each, we can tune models to provide the best minimised cost outcome for the use case.

These are just some of the metrics we can use to assess the outcomes of predictive maintenance, however the challenge lies in that to build and train a model, we need to choose one of them. Choosing which metric will ultimately come down to the specific use case.

How to Convince Stakeholders?

One of the biggest challenges in implementing predictive maintenance is gaining the buy in from internal and external stakeholders. Its extremely natural for decision makers to be sceptical about carrying out maintenance activities when everything looks fine and it requires a bit of a mindset shift. A few things can be done to overcome this challenge.

Data Transparency – Stakeholders are more likely to trust a system they can understand. By offering them visibility into the data that is driving the predictions and explaining how the model identifies failures can go a long way. If you have any historical success of predictive maintenance, showcase its success.

Focus on Cost-Benefit – Showing clear examples of how predictive maintenance reduces cost resonates well with decision makers, especially when accompanied by any financial projections.

Proactive Education – Equipping stakeholders with the knowledge about predictive technologies and their benefits will help them in understanding the basic principles behind predictive maintenance and they will be more likely to trust the model, take proactive actions even in the absence of any visible issues.

Legal Aspects

Predictive maintenance may involve some legal considerations, especially surrounding liability and any regulatory compliance.

Asset Warranties and Service Level Agreements – Predictive maintenance could potentially void warranties if activities are performed outside the prescribed schedules defined by manufacturers.

Accountability for Missed Predictions – If a model fails to predict a breakdown and causes financial losses or safety risks, questions of accountability will arise. Ensuring that contracts are in place that outlines who is responsible in such cases is key.

Additional Considerations

Continuous Improvement – The telemetry and operating conditions are always evolving. You need to ensure that the predictive maintenance model undergoes regular retraining and updating to remain accurate.

Human in the Loop Collaboration – No matter how sophisticated and accurate the models can become, humans should look at samples to validate the predictions when applying them in a practical context.

Explainability – This is a tricky one as once you go past basic models and into the realms of using neural networks, explainability becomes complex. Always try to build the simplest model that satisfies the outcome.

Summary

Assessing the success of any predictive maintenance implementation involves more than just evaluating the technical accuracy. By focussing on specific metrics and accounting for the business costs of different prediction outcomes, organisations can fine tune the modelling. Achieving stakeholder buy in and navigating legal obligations are additional crucial steps to fully realising and embedding the benefits the predictive maintenance has to offer. With the right approach and all steps taken, predictive maintenance can be an invaluable asset and investment for organisations, optimising maintenance activities and reducing downtime.