Outcome measures
Defining and using appropriate outcome
measures are the hardest parts of undertaking an audit that is going to achieve
a worthwhile end. In general, there has to be a decision as to whether the
outcome is doctor or patient orientated. Doctor-based outcomes look at such
outcomes as mortality, complications and the cost of consumables. A patient
orientated outcome may measure patient
satisfaction. In some situations the operation may be a technical failure but
the patient is happy with the result, producing a poor result for doctor
orientated audit, but a good result for patient based audit.
Whatever
method of assessing outcome is to be used the surgeon must be certain that the
instrument is measuring something relevant to the audit being undertaken.
Moreover, the outcome measure must be recordable within a time frame that will
be relevant but also achievable. For example, following laparascopic
herniorraphy it would be reasonable to assess patient satisfaction with the
process of the operation within a short, perhaps 3-month, time frame. The
success of the operation in terms of recurrence of the hernia cannot be fully
assessed for many months or perhaps several years after the procedure.
In
order for outcome scores to be useful, they must be rigorously validated. The
specific ‘tests’ that need to be used are as follows.
•
Internal consistency (Cronbach’s alpha) determines whether the
questionnaire or score is measuring a single essential concept.
•
Reproducibility (test—retest reliability) determines that the score
will give a roughly similar result if repeated.
•
Validity: does the score actually measure what the developers are
interested in? This is divided into content validity (the questions deal with all aspects that they
are intended to cover) and construct
validity (whether the questionnaire supports the hypothesis).
•
Sensitivity to change: will the score reveal small changes which are of
clinical importance during the period of study?
To
develop a score/questionnaire there are several ways of establishing the types
of question and the specific questions to be asked. It is possible to ask a
panel of experts what they think are the important issues to be assessed in a
disease or operation or anatomical region (face validity). The list that is
created is then collated to come up with groups of questions and specific
questions within each group. Once the draft questionnaire has been created the
validation tests will be applied. An alternative method in order to create a
patient orientated questionnaire is to ask the patients what they think is
important, especially functional aspects. Again the responses are collated and
can be refined by posing the questions to another group of patients. The
questions should also be exposed to a panel of experts at an appropriate stage.
Once again the validation tests will have to be applied before the questionnaire
can be used in a study.