Evaluating the Impact of Digital Health Communities

Overview

This study examined the impact of new forms of digital health peer-support groups in partnership with Shift.ms, a digital health community for people recently diagnosed with multiple sclerosis (MS) based in Leeds. It has 40,000 registered users and receives 60,000 monthly visits.

The study was led by Dr Ana Manzano (School of Sociology & Social Policy), in partnership with Dr Edward Webb (Academic Unit of Health Economics), Dr Albert Varela (School of Sociology & Social Policy) and the Shift.ms team. The research assistant for the study was Johanna Hentschel (MA in Social Research, School of Sociology & Social Policy, University of Leeds).

Digital health communities are new platforms where people with the same health condition from across the world communicate with each other to gain support. Despite their undeniable impact on society, fast moving digital health communities often struggle to demonstrate how they impact people’s lives and their relationship with their illness.  

We have filled this gap by co-producing a self-evaluation toolkit – with Shift.ms – specially designed to support the methodological needs and special context of digital health communities. This toolkit provides a set of evidence-based principles and resources to support evaluation activities, illustrated with real case studies from Shift.ms.

Download the toolkit

 

The process

The co-production process consisted of two embedded methods:

1. Dealing with Big Data and Digital Health Communities

Quantitative analysis of data gathered by Shift.ms using a longitudinal survey distributed to all new Shift.ms users, linked to customer relationship management (CRM) for websites data as a means to learn lessons about the complexities of evaluating the impact of digital health communities. Global digital health communities (GDHC) are complex entities and demonstrating the long-term impact that they have in their users is also a complex process. A number of initial recommendations were identified to act as a good foundation for more focused and local investigations in the implementation of impact studies in global digital health communities

2. Interactive Online Co-Production Workshops

Here academics and the Shift.ms team worked together to design a self-evaluation tool.

Co-Production Workshop 1 (July 2020) – What is a Self-evaluation Toolkit: Mission & Vision

In this workshop, Shift.ms and University of Leeds teams discussed what a self-evaluation toolkit is and how this could be useful to digital health communities. The workshop had two co-production exercises:

• Mission and Vision: From Prototype to a Useful Tool

The teams drafted together a mission statement of what they wanted the toolkit to do.

• Prioritization: Key Sections

The teams prioritized the necessary toolkit sections to accomplish the agreed mission statement.

Co-Production Workshop 2 (November 2020) – Focus on Sections

In this workshop, the content and structure of each of the toolkit sections were discussed and agreed by the teams.

Co-Production Workshop 3 (January 2021) – Learning from Big Data

In this workshop, the results of the quantitative analysis of the Shift.ms survey and big data were presented to Shift.ms. This exercise served as further reflection on how the self-evaluation toolkit could be used by other digital health communities to support them when they collect data to demonstrate their impact.

Co-production Workshop 4 (March 2021) – Toolkit Concept Development: Meet the Designers

Shift.ms and University of Leeds teams met with the graphic designers and fed back their ideas on the visual style of the toolkit. After this meeting, proofs were designed and five drafts were reviewed by the whole team, providing further feedback and final tweaks.

Download the toolkit

 

Further information

This study was funded by an ESRC IMPACT Acceleration Account Grant at the Leeds Social Sciences Institute. The toolkit was co-produced with Shift.ms and was also supported by the School of Sociology & Social Policy and the Leeds Institute of Health Sciences at the University of Leeds (United Kingdom).