Outcome measures

Defining and using appropriate outcome measures are the hardest parts of undertaking an audit that is going to achieve a worthwhile end. In general, there has to be a decision as to whether the outcome is doctor or patient orientated. Doctor-based outcomes look at such outcomes as mortality, complications and the cost of consumables. A patient orientated outcome may measure patient satisfaction. In some situations the operation may be a technical failure but the patient is happy with the result, producing a poor result for doctor orientated audit, but a good result for patient based audit.

Whatever method of assessing outcome is to be used the surgeon must be certain that the instrument is measuring something relevant to the audit being undertaken. Moreover, the outcome measure must be recordable within a time frame that will be relevant but also achievable. For example, following laparascopic herniorraphy it would be reasonable to assess patient satisfaction with the process of the operation within a short, perhaps 3-month, time frame. The success of the operation in terms of recurrence of the hernia cannot be fully assessed for many months or perhaps several years after the procedure.

In order for outcome scores to be useful, they must be rigorously validated. The specific ‘tests’ that need to be used are as follows.

  Internal consistency (Cronbach’s alpha) determines whether the questionnaire or score is measuring a single essential concept.

       Reproducibility (test—retest reliability) determines that the score will give a roughly similar result if repeated.

Validity: does the score actually measure what the developers are interested in? This is divided into content valid­ity (the questions deal with all aspects that they are intended to cover) and construct validity (whether the questionnaire supports the hypothesis).

Sensitivity to change: will the score reveal small changes which are of clinical importance during the period of study?

To develop a score/questionnaire there are several ways of establishing the types of question and the specific questions to be asked. It is possible to ask a panel of experts what they think are the important issues to be assessed in a disease or operation or anatomical region (face validity). The list that is created is then collated to come up with groups of questions and specific questions within each group. Once the draft questionnaire has been created the validation tests will be applied. An alternative method in order to create a patient orientated questionnaire is to ask the patients what they think is important, especially functional aspects. Again the responses are collated and can be refined by posing the questions to another group of patients. The questions should also be exposed to a panel of experts at an appropriate stage. Once again the validation tests will have to be applied before the questionnaire can be used in a study.