Evaluating digital technology in mental health care

Project aims

This study, conducted by NIHR ARC West and the University of Bristol, evaluated the introduction and use of these two tools, whether they were effective, helpful to staff, and what impact they had when working with service users or planning care, and what challenges they faced when trying to use the tools in practice.

What we did

We interviewed 20 staff, including support staff, registered mental health professionals and managers, to explore:

• how the care pathway tool changed their support work with service users
• what factors influenced their use of the tool in working practice
• how the risk profiling tool was used to decide which staff could best support service users according to their needs

We assessed the accuracy of the risk tool by asking clinical staff to assess which service users were more likely to have a crisis within 28 days, comparing this with the tool’s predicted risk levels. We then analysed these results and compared this with the number of service users who actually did experience a crisis in that period. We also used statistics to determine whether staff use of the risk tool made a difference to:

• whether service users with more complex needs were allocated to appropriately qualified staff
• the time it takes for a service user to be assessed and treated after referral

What we found and what this means

Results from the risk profiling tool evaluation

Comparing the risk profiling tool’s assessment with clinical staff judgement

The results from the risk profiling tool didn’t always match clinical judgement of a service user’s risk of a crisis. From the group of 93 service users who were assessed, four service users had a crisis in the following 28 days. The clinicians identified one of them, while the risk profiling tool identified three. But because the sample of 93 is small, there is no statistical evidence for a true difference in the predictions of clinicians and the risk profiling tool. It could just be a chance finding. All interviewees who used the risk tool said that it was an initial step in a much broader clinical decision-making process.

Improvements in allocating service users to care coordinators

All interviewees who had used the risk profiling tool said that it helped with decision-making when allocating service users to appropriately qualified staff. When the risk profiling tool was being used there was a greater chance of a service user being allocated an appropriately qualified care coordinator. Because of the small numbers using the tool, it was not clear whether this was just a small improvement (5% higher odds) or a big improvement (51% higher odds). Also because there were many other changes in the hospital and patient management during the period of study, we can’t be sure whether any improvement was because of the risk profiling tool.

Improvements in referral to assessment and treatment time

We compared the waiting time for a patient to be assessed and treated, from before versus after the risk profiling tool was in use. In the period where the tool was being used, the amount of time between referral to a service, and being assessed and treated was reduced by about 14 days.  We can’t be sure that this was because of the introduction of the risk profiling tool, because other work and changes happening in the organisation may also have led to the reduction in waiting times.

Overall, staff valued the risk profiling tool, saying that they would recommend it to a colleague, although firm conclusions cannot be reached because of the small sample size. However, it was seen to be helpful to support:

• Clinical decision-making when allocating appropriately qualified staff to service users
• Deciding how many qualified staff were needed through the different services
• Assessing risk and having an overview of staff caseloads

Results from the care pathway tool evaluation

Several factors were identified that could make it easier or more difficult for staff to use the tool in routine practice. Some of these factors were linked to the tool itself.  The care pathway tool was visual and interactive, engaged staff and service users in different recovery-focussed conversations that they might not otherwise have had, and could help service users contribute to writing their care records. The tool wasn’t always easy to use because of technical issues with the tablet computer. On the other hand, staff thought the digital content of the tool was easy to navigate. .

Usual practice was for service users’ health records to be kept in offices, and for staff to update these records after they’d visited service users in the community. Some service users were surprised to hear that these records existed and were keen to take part in writing them. The tool enabled this.

This could also save staff time as they didn’t have to write up their notes on a computer after visiting service users, and the care pathway tool helped staff work away from the office. But the tool didn’t automatically update the organisation’s electronic health records.

The tool wasn’t always easy to use because of technical issues with the tablet computer, and it depended on whether service users felt well enough and were willing to engage with it.

Features that worked particularly well included:

• ‘People in my life’ – this interactive feature helped service users to identify and appreciate all the different people who could provide them with support and help
• ‘Managing my warning signs’ – this helped service users discuss the warning signs that might indicate that they need more support. This also helped staff discuss with service users how to better manage their symptoms, and what to do if they thought they might be at risk of a crisis
• ‘Planning for my future’ – this helped service users to set their own goals.

Other factors were related to the wider mental health care delivery context, for example policies, and professional and organisational understandings of how care should be provided. Service providers felt positive towards the tool because they were involved in its design and development, and these priorities informed the tool’s design.

For staff, the needs of service users were central to decisions and influenced the use of the tool. Using the tool depended on whether service users felt well enough and were willing to engage with it. Concerns about whether service users were ready or able to use the tool, and how the tool might change relationships between service users and providers, drove decisions about whether to use the tool, how, and with which service users.

Staff workload, and how much time staff had to prepare for using the tool with service users, could make it difficult to use the tool. Not all staff had positive attitudes towards the use of technology in their practice, which influenced how proactive they were in using the tool. Some staff felt they didn’t have sufficient knowledge of how to use the tool in practice, although training provided by peers – which was part of the development and testing phase – made it easier for them. Staff thought that more training on the interactive aspects of the tool might help them feel more comfortable using the tool.