Evaluation

‘Fairness’ as a discussion point

The starting point for using ‘fairness’ as the focus for the work was:

  • Fairness is a word increasingly being invoked in debates around public services, but perhaps without a consensus on what it means.
  • As a general principle, fairness is a good starting point as it applies to everything the NHS does.
  • It avoids getting bogged down at an early stage in the details of which areas should be prioritised.
  • GPs themselves are very interested in the subject and aware of the need to be seen to be acting fairly.
  • The public will be able to engage with the subject on different levels.

Although at times the group wandered a little ‘off topic’, fairness worked well as a frame for the discussions and I would recommend it as an opening topic for conversations by other emerging GPCCs.

 

Methodology

Some concerns had been expressed about this ‘citizens jury’ deliberative approach. Specifically - that members of the public would struggle to grasp the issues sufficiently within an amount of their time we could reasonably ask for; and that a group of 12 people could not reflect the views of the wider population.

On the first point I maintained that if a group of ‘twelve good men and true’ could be trusted in a court of law with a decision that could send someone to prison for life, we should have faith in their ability to tell us what fairness means. I think the report, its reception by the GPs and the group’s feedback vindicates this view.

On the second point, which was echoed by one of the participants, I accept that the findings of the group should not be presented as representing the views of the population. Apart from anything else, it would be unfair to the participants to put the weight of responsibility on their shoulders, they were only there to represent their own views. For this reason, their report was marked ‘provisional’ and the wider public have been invited to add their comments via the website for a six month period which will be incorporated into a ‘final’ report.

The reason why we badged this work as a ‘Health Commission’ rather than a ‘citizens jury’ was to avoid the suggestion that something or someone was on trial. Initially we referred to the participants as ‘Commissioners’ but dropped this when it started to create confusion between the Commissioners on the Commission and Commissioners who do Commissioning!

My own major reservation about the methodology is sustainability. This has proved to be a very time intensive piece of work. It may be that having learned the lessons of having done it once, less work would be involved in a similar exercise but it would still be a considerable amount of work. It perhaps represents a ‘gold standard’ of public involvement which we would use only sparingly as needed and as resources allow. It is also perhaps more suitable when discussing “umbrella” concepts than individual issues.

 

The sessions

To recap, the format was: a midweek evening preparation session followed by two consecutive Saturday’s of evidence and discussion and a further weekday evening to finalise the report and debrief.

This was a lot to ask of people – a total of 21 hours of commitment over a 2 week period. Clearly there is a tension between wanting to cram in as much useful information and discussion as possible and not asking too much of people.

However, apart from 1 person missing the initial preparatory session due to a communications glitch, everybody attended all the sessions.

We did the sessions on Saturdays specifically to accommodate people who work during the week. However, this did make it more difficult to get speakers, most of whom work full time Monday-Friday. In the main we did get our first choice of speakers but good will might start to thin out if we did this more often!

Once the thoughts had been gathered on flipcharts from the discussions we had to convert this into a draft report. To do this, a core group of LINk and PCT staff met with the patient chair to pull the points together on the Monday morning. The patient chair then went away and wrote a commentary reflecting these points. These were combined into a draft report which was edited in real-time by the whole group using a laptop and data projector on the final Wednesday evening.

This was a compromise between the need to give the whole group ownership of the process but not have too many people trying to work on the same task. It might be the least worst way of balancing the need to be inclusive with the need to make things happen quickly.

At the final session there was considerable debate on each recommendation and a number of changes were made. Each bullet was voted on. On the occasions where votes were split, this was recorded. With hindsight we should not have allowed people to ‘abstain’, they should have voted for or against.

However, the fact that one or two people voted differently acted as a check on the process, and shows that participants had understood and assimilated the information.

 

The subjects

The thinking behind the sessions was as follows:

  • Keynote Speech – to demonstrate that the GPs were fully on board and saw the work as important
  • How The NHS Works – because the group had no prior knowledge it was important to cover the basics about how the NHS is structured and the commissioner/provider split
  • Where The Money Goes and The Financial Challenge – because if the group had failed to grasp that choices have to be made, it would have been less likely that they would have produced recommendations which were of practical use.
  • Prioritisation – to give the group an idea of how the PCT currently prioritises
  • Health Inequalities – to get across the idea that different localities within the borough have very different health outcomes with implications for fairness
  • Fairness – this was an opinion piece on fairness designed to be thought provoking.
  • Ethics – to give a theoretical underpinning
  • Health Economics – to help the group think about practical ways of allocating money and measuring effectiveness
  • Public Involvement – to encourage the group to think about their expectations on this subject
  • What Really Matters To Patients – to help the Group think beyond their own experience as patients about the wider evidence.

We kept one session on the second Saturday clear for the participants to decide what they would like to hear more about. Initially I had thought we could have a second opinion piece to balance the views of the Socialist Health Association on ‘fairness’. However, given that the that presentation was not quite what I had expected (though I enjoyed it the most!) and we felt the group would benefit from more time for reflection and discussion, we agreed with the group to use the time for discussion.

There were no sessions which the group thought were not useful (all scoring over 90%). The group struggled somewhat with the ‘how decisions are made’ session but I am not sure they would have felt any more enlightened if we had spent much more time on the subject.

 

The locations

We decided that we did not want to use NHS premises for the main Saturday sessions. I felt a clinical environment would have been less than ideal and wanted people to look at things afresh without focusing on the day to day of operational delivery.

We used an out of town venue in a countryside location on the edge of our beautiful Saddleworth area. Although people liked the venue, it was rather out of the way for some of the participants. The venue for the two evening venues was more central but perhaps less pretty. It was however received as being an excellent space for the task. Had we not had funding for this project we would probably have been forced to use NHS premises, or other local venues we could have used free of charge.

 

Sample

In my naivety on matters statistical I had originally envisaged a ‘representative’ sample of the population. However, my public health colleagues informed me we would need a sample of several hundred people to do this and instead I was advised to refer to a ‘reflective’ sample. We agreed a back of envelope description of what such a sample should look like.

As well as being reflective, we knew we wanted a group without any prior baggage, who were not ‘known’ to either the PCT or the LINk and were not community leaders or other high profile people. In practice, looking for a reflective sample of people you don’t know proved very difficult!

We decided not to publicise our search because we felt that we would get inundated with people who were known to us or who were from a limited demographic (typically white, middle class, retired women with long term conditions). Of course this group are fantastic and their views are at least as valid as anyone else’s, but we did not want to be in the position of turning away lots of people.

However, I now feel we could have done this differently. We should probably have publicised the recruitment but set out clearly why we were looking for the sample we were. We may have had to explain to interested people that places were filled, but could have asked to keep their details (data protection applies) for future events. We also should have allowed more time to recruit the sample, as this proved very time consuming as well as developing new contacts as sources of recruitment (e.g. college, university).

Both the PCT and LINk drew on our networks, formal and informal to put the word out and we did get a sample that broadly reflected our plan, with the exception that men were still under-represented. In order to recruit a panel that fits the panel profile, we would suggest recruitment should start at least 2 months before the start of the event.

A definition of the target sample and the routes used for recruitment can be found here.

Roles

The project was run jointly with Oldham LINk. In Oldham, the PCT and LINk have a very constructive relationship with a track record of working together. I feel we worked well on this project and it was certainly useful to tap into their networks and experience in supporting people to participate in engagement. However, it may be that in other localities, where relationships are not as good or the LINk lacks capacity or capability, this joint approach would not be practicable.

The LINk staff team reports that this is a two-way process. They recognise the value of partnership approaches from NHS Oldham andd the necessity of personnel being proactive in seeking to work collaboratively with the LINk in the best interests of patients.

The core team running the Commission was 2 PCT and 2 LINk staff plus the patient chair.

We decided to have an independant patient chair in addition to the 12 jurors. Unlike the jurors, Peter has considerable prior experience in health service engagement and governance. Peter’s role was to support the jurors in participating and throw in the odd informed question if the group were losing their way. In practice, I think there was some overlap between the Chair’s role and that of the LINk facilitator and if we did the event again I would want to differentiate these roles more.

 

Cost effectiveness

The total direct financial outlay was as follows:

Participation Fees £1,618.89 (£5.93x21 hoursx13 people)
Participants’ Expenses fee £676.13 (mostly childcare)
Venue hire/catering £1,394
Website £52.89

TOTAL £3,741.91

The indirect costs of PCT and LINk staff time, plus that of the witnesses however is considerable. It is more likely time than money that would limit the use of this methodology going forward.

 

Participants

The key points I take from the participants own evaluation were that overall the methodology was vindicated, with all participants awarding us 100% for providing a positive experience and 94% for feeling well informed on the subject.

The group appeared to have struggled most with the presentations on finance and the mechanics of commissioning. This is not their fault, it is hard to understand! It is a difficult line between ‘dumbing down’ the subject and giving a meaningful primer.

However, it was clear to me through listening to the discussion that the group had taken on the key points.

The ‘evil commissioner’ session was rated relatively poorly. This involved making a rather ‘over the top’ presentation by a supposed Commissioner whose plans were obviously perverse to tease out from the participants their objections. However, we felt that this session provided the most insights which were subsequently incorporated in the report.

The other telling feedback was that, despite being broadly happy with the process, there was only a 78.2% confidence that GPs would take on board the report and use it. This suggests there is still much work to do in building confidence.

Finally, 10 of 11 participants said that would still have participated if they had not received a participation fee, but only out-of-pocket expenses. Generally the PCT has not paid people such a participation fee over and above expenses for taking part in such events, but on this occasion we decided we should because:

  • of the amount of time and commitment we were asking for
  • we would pay other experts for their expertise
  • the wanted to attract people of working age who may need to forego income
  • we had a budget to make payment

This would suggest it would still be possible, albeit more difficult, to attract people to participate with only paying out of pocket expenses.

 

Technology

The website was done very cheaply using free open source software (Joomla 1.6 with the Beez 5 template). I chose Joomla because it is modular and flexible, and will allow us to add functionality in future should we wish to (e.g. a chatroom, forum, or even a shop!).

I chose Beez 5 because of its accessibility. The Web Accessibility Evaluation Tool (WAVE) detects no accessibility errors. The template required minimal customisation. I did not need to edit the stylesheets but did create the ‘COG’ torpedo and juror’s heads in Adobe Photoshop and replaced the original images with these.

I added several plugins to the site to enable external content. These were: Youtube (videos), Iframe (for the Slideshare presentation), Flickr Slideshow (for photos) and iComments (to add commenting).

The domain name was registered with 123.reg for 2 years at a cost of £7 and hosting for one year was with 5quidHost for £55, so the cost was mainly in time to devise the site, set it up and create the content.

I had set up sites using Joomla before and had used the services of the same domain and hosting companies so had a head start. I would not recommend others to do this without some prior experience of running a content management system and knowledge of how websites work.

With hindsight, I would have wanted to perhaps spend more on making the videos. These were done for nothing using my own equipment but the sound was disappointing.

Mark Drury
Associate Director, Engagement and Stakeholder Management
NHS Oldham
May 2011

 

Participants' Feedback is here.