Some of this document is reasonably polished, but other parts
are rough notes only.
I have also put up a webpage on "How to study statistics",
aimed particularly at those taking their first statistics course:
My name is Paul Hutchinson, and I work in the
Psychology, Macquarie University. However, this document is
not really specific to psychology --- in previous existences I've worked
in Statistics and Civil Engineering departments, and I think
most of what is said below is quite widely true.
Anyway, here is my advice on
How to do research
The first thing to say is that if it has to be done, it's you who's
got to do it.
You. There's no-one else to do the project for you.
Very likely, there's no-one who will give you any help. You're the one who
cares, the drive has got to come from within you. You're the one who knows
how to answer the question, no-one else knows as much.
The above is not the whole truth --- as your project evolves, you'll find
yourself receiving technical help, wisdom, and emotional support from a
great many people --- but you need to be aware that self-reliance is vital.
There may be other types of research, e.g., in which a neophyte is fitted
into a slot in the God Professor's program. I don't have experience of them.
Do. It is important that you start the main activities
of the research --- experimentation, or survey, or simulation, or data
analysis --- sooner rather than later. A common mistake is to think and plan
and read, and never get round to the main game. When you start, you may find
that the real difficulties and issues are quite different from what you
thought in the abstract.
Research has several components, for example: thinking of the question,
answering the question, communicating the answer.
It may be worth mentioning that not
all of these are essential.
Even without much of a question, one can collect some data, summarise
it, and report the results. Certainly if you're the first to think of this
kind of data, that's all you need to do. Certainly if the data keeps
changing from year to year (e.g., accidents, and many other society-level
phenomena), this type of study is always useful to a greater or lesser
Even without an answer, the question is sometimes such a good one
that it is worth telling others about. (It might be that you don't
have time to answer it, or are uninterested, or not competent in the
Sometimes you don't want to communicate your question and answer.
They might have turned out to be less interesting than you thought when
you started. Or you might want to wait until you've done something
Nevertheless, most research has all three components. The answering
and the communicating can be dismissed quite briefly.
The fact that you're doing research is strong evidence that you
have the skills necessary to answer the question. Not all of them, perhaps,
but enough to work out a way of attacking the research and to serve as a
foundation for additional skills to be learnt along the way.
Incidentally, as a rule of thumb, if there is a course
on something, people think of it as a low-level skill
--- it is not necessarily below your dignity as a researcher,
but you don't get any merit badges for mastering it. (I am a little
concerned at the increasing amount of structured studies that
universities are prescribing for their first-year research students.
To my mind, the overwhelming priority at that stage is to do
a doable piece of research, not attend lectures on bibliographic
tools and the institution's policy on intellectual property.)
As to non-technical aspects of answering the question,
there is some advice below.
There are lots of textbooks and courses to aid you in your scientific
writing. Essentially, though, it is all a matter of practice.
One of the reasons I recommend psychology to students is that you can't
escape either the use of numbers or the use of words. (Economics is
another good subject from this point of view, but it doesn't have much
of an experimental component.)
As soon as you've got a message to communicate, you should write a report
on it, not wait until the whole project is finished. That probably means
writing 1000 words per week, every week.
Just occasionally, supervisors forget to tell
new writers the obvious --- the easiest sections to write are the
descriptions of the results, and of what you did (because these sections are
highly constrained by the facts), and you should write these first; what
the results mean, and why you did what you did, are altogether more
problematic, so the Discussion and Introduction sections should be written
later. If you're desperate, imagine you're speaking to a friend about
your work, and write down what you would say. (Then, as always,
revise and edit, revise and edit, revise and edit.)
Some research has to be done to a tight deadline --- for example, the
final-year projects carried out by undergraduates. The big difference
this makes is that you don't have time (which you do in most situations)
to get things wrong and then do them again. The consequence is that
there's a good deal of luck involved at this stage, so I think that if
your final-year project falls flat on its face, it doesn't mean you
won't be good at research in a more normal environment.
It is only natural for a student to ask "What will get me good marks?"
In assessing theses, some universities have marking schemes --- u marks
are allocated for the project design, v marks are allocated for the
conduct of the experiment or survey, w marks are allocated for the
general writing of the thesis, x marks are allocated for the review of
previous literature, y marks are allocated for the presentation of the
results, z marks are allocated for the interpretation of the findings,
and so on. Other universities have no such marking scheme. (Even
where one exists, it is very often impracticable to use it strictly.)
My advice in such a situation may be different from that of others. But,
for what it's worth, here it is. My impression is that students
typically spend too much time reviewing past literature, and not
enough time thinking about it and criticising it. The student's ideas
need to be firmly grounded in established wisdom, not plucked out of
cloud cuckoo land. But the thing that will most impress the marker of a
thesis is a focus on the central ideas, a willingness to criticise
imprecise thinking by previous authors, the dissection of what is
essential from what is less relevant, and the demonstration of how the
student's ideas sharpen what has previously been blunt. In other words,
An important issue is that of what to study. If you have ideas, there's
no problem. But, is there anything you can do to prompt ideas to come to you?
Or, in the absence of ideas, how can you do something worthwhile? I don't
suppose there's a complete answer to these questions, but the following
comments are intended to be helpful.
To get ideas, expose yourself to them.
Talk to someone about your research. Your mum or dad,
for example. Or (though
I've never tried this) the top stream of 13-year-olds at your local
selective high school. Brainstorming with your colleagues is better
than nothing, but you're not really forced to decode your talk sufficiently.
Anyway, there are other uses for your colleagues, such as being rude to you.
It is tremendously valuable if there's someone in your circle who can
bring off the trick of being rude without causing offence. Suppose you
are a cognitive psychologist, and in a seminar you claim that "activation
flows through the levels" of your model. You benefit if someone from
another tradition confronts you with the claim that "activation",
"flows", and "levels" are all meaningless. It is generally up to the
young to do this.
Informal seminars, where you listen to others, and talk yourself,
are an important part of this.
Read widely. (But remember what I said earlier: reading is not
what research is about. Indeed, I would say there is a type of
personality for whom too much reading is a major danger to their
ever doing anything worthwhile.)
How to read: read the title, skim the abstract, look at the pictures
and maybe the tables, and if there's anything interesting, then consult
the text, looking for that specific point. (No-one starts at the beginning
of a paper and works their way through to the end.)
What to read: many of the "prestigious" journals are best avoided, as
they have got into the habit of attending to a lot of details that everyone
knows are unimportant, thus diluting the real message. The papers are polished
and bland, and the reader is sucked into taking them on their own terms.
You need something rougher, that you can get to grips with.
Comments and letters in any journal. Whatever the point is, it is made
quickly. Furthermore, an area of controversy may be highlighted for you.
Journals from outside your main discipline that are relevant
to your topic. For example, if your own discipline is psychology,
see what the management, sociology, medicine, and engineering
journals are saying about your topic. They may very well take
sufficiently different an attitude to provoke thought. And interdisciplinary
journals, e.g., on the borders of music and psychology, or law and
For the same reason, journals from countries that Americans have
never heard of (such as Germany, India, and Japan).
Low prestige journals sometimes publish simple data on unusual questions
--- if the topic attracts you, ideas for improving the method or generalising
the results will soon come.
For the same reason, you can often find mental stimulation in
journals from 60 or 80 years ago.
Your attitude: sympathetic criticism. So much junk is published that a
degree of sceptical hostility is appropriate when approaching most papers
(unless the author is on the Approved List). But temper this with
sympathy: a paper may be useless to you, and it may even be clear that
by any reasonable standard the research was a waste of time, but yet
the method may be useful to one reader, a detail in the results may be
just what a second reader is looking for, and a third reader may find her
thoughts clarified by a point in the discussion.
If someone says your paper is trash, retort that anyone who writes
a decent paper can get it published, but it takes real talent to get rubbish
Forget harsh realities for a moment, and think seriously about
what question really interests you, gets you passionate.
Even though you're trying to keep your mind on the ideal, you
will find practicalities obtrude themselves. Some of these
practical problems may turn into research projects.
Look to a leisure interest of yours for inspiration. Many students
are interested in at least one of music, sport, and politics; well, if
their academic subject is psychology, surely this has some interesting
things to say about all of these? The same goes for many other
Many of us are lucky enough to have easy access to computerised
databases. Try searching them with an unusual combination of words,
e.g., statistics and music, or laughter and Wagner, or measurement and
Ask successful researchers in your department whether they have
any particular methods for getting ideas to come.
I find making lists, and then organising them, often useful.
Lists of what? Of questions, possible experiments, possible surveys,
sources of data, ways of operationalising a concept, ways in which your
supervisor could be more helpful,...
Draw an analogy. (Theoretical, or method of analysis.)
Thought experiments. If the results were to be ..., then
we would conclude .... On the other hand, if they were to be ...,
our conclusion would be ....
Suggestions for what to do if you don't have any ideas.
Criticise a paper by someone else.
Can I improve the experiment?
Can I improve the theory? In statistical modelling, one might
replace a discrete variable with a continuous variable, or vice
versa --- e.g., if "blue" and "white" types have been
proposed, replace them with a trait ("blueness").
Can I refine the definition?
Can I broaden the circumstances in which the effect occurs?
Can I simulate this on the computer?
Can I improve the statistical analysis? ("Two of a trade never
agree", it is said, and certainly no two statisticians have agreed
with each other this century.) Improvement does not necessarily mean
additional complication. Sometimes, simple descriptive
results are overlooked in the rush to perform a complicated hypothesis test.
If nothing better comes to mind, try confidence intervals in place of
hypothesis tests, or nonparametric methods in place of parametric, or
Bayesian weight of evidence in place of a p-value.
If you do comment on a published paper, you're actually paying it quite
a high compliment. (If you come across 24-carat dross, you
should follow the convention of politely ignoring it.)
You can always write a paper on:
The useability of university Calendars.
The comprehensibility of ergonomics textbooks.
The rise and fall of the use of particular cliches in the titles
of learned articles.
Do readers want more raw data in journal articles?
Bayesian weight of evidence in psychology journals, 1970--1996.
How first-year students view research participation.
Comparison of several fields in respect of their tolerance of
low survey response rates.
Why do students hate statistics?
Get the idea?
Think of an experiment that no-one else has done, and do it.
(The emphasis here is on doing something because it can be done,
not because you actually understand the implications.)
The different packaging of something unoriginal.
Go to a journal that is less quantitative than you are, and
teach the readers something (be tactful, now).
If you have a rigid deadline to work to, it is vital that you
are not reliant on someone else for anything important. Among the
possible areas of difficulty are: administrative approval from an
outside body, construction of equipment, computing. You must have a
workable plan that can be implemented if approval is refused/the
equipment isn't delivered/the computing expertise is unavailable.
Most people beginning research underestimate the importance of
the physical aspects --- going through 500 files a second time because
you didn't extract a particular item of information the first time,
coming in to the lab on Sunday afternoon to try out an idea you've
just had, driving at 4am to another city because the evening news told you
of a "natural experiment" there that day, taking 60 journal volumes off
the shelf to skim through for the key paragraph you wish you had made
a note of, and so on.
Write down an idea when you have it, because you may easily have
forgotten it by the time you get to the office. Some people even keep an
ideas book for this purpose; but the trouble with this is that it really
need to be physically very small, so that it can be always with you.
Work around the problem, if you can't solve it.
Sometimes a problem is best handled by defining it away.
There is some research that simply isn't worth doing. It may be
obvious to the drover's dog roughly what the problem is... and that
the institutional barriers to solving it are insurmountable. Surveys,
especially, can be a substitute for actually doing anything about the
Importance of flexibility.
Luck. Have several projects going --- diversification --- they
won't all be disasters.
Don't be afraid to admit your ignorance. The nature of
academic research is that one is continually venturing into
areas where one is an ignoramus. It is unpleasant
to admit this. But the earlier you do, the sooner you get an
explanation and the quicker you can put that little bit of
extra learning to use.
There can be a danger in overmuch personal involvement. Some
people love people with Alzheimer's disease, or autism, or depression,
and want to research the condition. For a beginning researcher, the
problem of separating scientific evidence about the mass of people
with the condition from the loved one's personal experience is one
problem too many --- it is better to select another subject for
Perfectionism and attention to detail is a virtue. Taken to
excess, it is a vice.
Some researchers seem to suffer from a "fear of success", which I
think is different from obsessive perfectionism. I think what
happens is that they perceive the successful completion of
their research as the end of one stage in their life and the
beginning of another, and they are reluctant to move on to
the new stage.
Universities have become very bureaucratic about documenting the
progress of research students. One can appreciate the reasons for
this, and yet there is a sense in which it is all a waste of time
--- if the student is successful, there's no need to have a file
of routine documentation; if he or she isn't, no amount of
paperwork will rescue the situation.
The supervisor may feel the student is
wasting his or her time, and yet be reluctant to say this (or say it
forcefully enough) because it may actually be wrong and because
in any case it will be discouraging. The supervisor may proffer
clear advice, and yet the student be stubborn. If I had an answer,
I'd give it here...
All sorts of stuff has been written about the role of the supervisor.
Much of it is sensible. But it's all a long-winded way of saying
that it is reasonable for the student to expect the supervisor to
listen to him or her for something like 20--50 hours per year.
Anything more than that is taking a responsibility away from where it
properly belongs (i.e., with the student).
Don't miss any deadlines that have been agreed with your supervisor.
If you do, then (depending on the nature of your excuse) you'll go
on to either the list of people who probably won't make it, or the
list of people who probably won't make it but who deserve sympathy.
You do not want to be on either of these lists. And don't be late for
meetings with your supervisor.
Don't expect much praise from your supervisor. If your supervisor
thought about it, he or she would realise that a few words of
praise were both justified (by the brilliance of your work) and
desirable (for the benefits of positive reinforcement). Sad to say,
he or she doesn't think about it: from about your second week of
research, you've been accepted as a grown-up and
been judged by the same standards as the professors.
You might find it worthwhile reading How to get a PhD.
Managing the peaks and troughs of research by E M Phillips
and D S Pugh (Milton Keynes: Open University Press, 1987).
Here are three points that Phillips and Pugh make, all of which
I mildly disagree with. (i) Intelligence-gathering (i.e., description
of the facts) is not research. (ii) Research of the testing-out type
(as contrasted with exploratory research and problem-solving research)
is much the most appropriate for a PhD student. (iii) It is necessary
for your thesis to have a "thesis", in the sense of a connected
argument and message.
It may be that Phillips and Pugh are substantially correct in the
context of relatively mature fields of study; and that my opinion
that (i) intelligence-gathering may be research, (ii) exploratory research
is useful and appropriate, and (iii) the research may be too messy to have
much of a "thesis" to it, is formed by my experience being mostly in
relatively immature fields of study.
Taste. Is it permissible to use phrases like "glass ceiling" or
"evidence-based medicine" or "gold standard"
in reporting your research? Fastidious
writers avoid cliches most of the time, but not every use of them
should be condemned. (In some allegedly scholarly fields of study,
they seem to be compulsory.)
Don't worry about the possibility of someone else duplicating your
research. It is rare for this to happen, and, when it does, it is
stimulating to examine the similarities and the differences.
Don't despair if it becomes necessary to tear up two years' work.
This is quite common, and often means you are so expert and can see
things so clearly that you can finish the whole project in only
six more months.
If you do happen to discover something important, be sure to do it
at the right time and place. Perhaps you remember the inventor
of the Infinite Improbability Drive: "just after he was awarded
the Galactic Institute's Prize for Extreme Cleverness he got lynched
by a rampaging mob of respectable physicists who had finally realized
that the one thing they really couldn't stand was a smartass"
(The Hitch Hiker's Guide to the Galaxy, Chapter 10).
about an earlier savant who suffered an equally dire fate.
Applying to do research.
There is quite a lot of advice available on the WWW about how to
choose which departments to apply to (for a research degree), and
how to increase your chances of success. I'm a bit sceptical
about the need for this --- I think
that by the end of their undergraduate careers, most students
know in what sub-field of the subject they want to work, or what
style of approach they want to use. Furthermore, they know which
departments at which universities study that sub-field or take that
style of approach. Consequently, their list of desirable departments
is already a short one; if they have made personal contacts, the
list will be very short indeed. However, if you think you need
advice, here is mine.
You should appreciate that there are an enormous number of
specialisations available in the scholarly world. Flick through
one of the directories of scholarly societies if you don't
believe me. For example, there is a Colour Society of Australia,
an International Association for Cross-Cultural Psychology,
and a Stress and Anxiety Research Society. If some specialisation
appeals to you (surely you'll like something in all
your undergraduate studies!), then sometime about the middle of
your undergraduate career, you should stop thinking of yourself
as (for example) an embryo psychologist, and start thinking of
yourself as (for example) an embryo colour psychologist. You will
then only be interested in departments that are strong in
If you are in this position, do not limit yourself to departments
of psychology. You might find the most suitable place is
in a group within a chemistry or physiology department.
Make personal contacts. If colour psychology really does fascinate
you, make contact with people in the field. Academics are
enthusiasts for their own little patch, and will be delighted
to find someone who shares that enthusiasm. If possible, get some
lowly job in their department during the vacation. But be warned
--- I would say it is impossible to successfully fake this.
Alleged enthusiasm without an appropriate level of specialist
knowledge and native intelligence will get you nowhere.
If you really are unsure --- if you just feel you'd like to do
research in "psychology" or "chemistry" --- you may do better
to delay your entry into research until your ideas have
matured a little further. (You cannot be expected to have
a fully-thought-out research project at this stage; what I mean is
that you should know (for example) that your interest is in
cognitive psychology studied by computer-controlled experiments
on normal humans.)
A little advice of a statistical or technical nature.
This is not the real theme of the present document, but a few
comments of this type may be worthwhile. You might also like
to read Pitfalls
of data analysis by Helberg.
There is often tension between the respective
advantages of standardising on one particular method of answering a
question, and of using a variety of different methods.
Think of measuring unemployment, or television viewing, for instance. On
the one hand, one wants a standard, reproducible, method --- comparisons
from month to month are important, and one has reason to think that
whatever defects are present remain pretty constant over time. On the
other hand, any single practicable method is bound to have peculiarities
and biases, and the "real, true" answer should probably be synthesised
from several different methods, even if each of them makes use of
only a small sample size.
When controversies break out, one factor is often the intrusion of a
new player into the system: the existing interests have learned to
co-exist with each other and with the defects of a question-answering method,
but this doesn't suit someone new.
The following is very much a broad-brush statement, to which
there must be numerous exceptions: too often, a standard method
is adopted too early in the development of a subject. In research,
one usually wants the "real, true" answer, whereas for
administration, one may prefer a standardised, reproducible, answer.
"Figures often beguile me, particularly when I have the arranging
of them myself" (Mark Twain). Presumably sociologists have studied
the use of statistics in the wielding of power?
Meta-analysis: this term refers to quantitatively combining the
results of previous studies of a subject, in order to arrive at the
right answer. I'm not terribly keen on the idea, but
this may be because I have not worked in fields where
there have been lots of previous studies of the same question.
There are some difficult principles and practicalities involved,
and inadequate handling of these led critics to refer to
meta-analysis as being "meta-silliness". I think there have been
substantial improvements in the methodology, and these were winning me over,
but then I read a paper by Sohn (Clinical Psychology Review,
16, 1996, 147--156), which argues that publication bias is so serious as to
vitiate all conclusions from literature reviews in the field he examines,
effectiveness of psychotherapy (and I think the implication of the paper
is that publication bias has a more serious impact on a meta-analysis
than on a review in traditional narrative format).
Contrast between exploring data, and using data for hypothesis
Maybe you want to do both. One easy technical solution
is to randomly split your data
into two portions, explore one half and generate hypotheses from it,
and then test the hypotheses using the other half.
Importance of a random sample, or random allocation.
I have the feeling that often it is foolish to prefer a
rubbishy sample of 200 to a random sample of 20, yet often
this is done.
It might be possible to argue that a rubbishy sample of 200 is the
better for hypothesis generation, whereas a random sample of 20 is
the better for hypothesis testing.
My impression is that the subject which is leading the way
as regards strictness of methodology is medicine. Many people will be
familiar with the basics. In order to measure the effect of something,
it is necessary to compare it with something else. That is, we have a
treatment group and a control group. Patients are
assigned at random to the one group or the other. It is
desirable that neither the patient nor the doctor who evaluates the
patient's condition know whether the patient received the drug or not.
(This is termed a double-blind experiment.) But did you know
this idea is being taken so far that Gotzsche
(Controlled Clinical Trials, 17, 1996, 285--293) seriously suggests that
the data should be blinded during statistical analysis? That is,
two reports are written. The first assumes A to be the treatment and B
to be the control, and the second is based on the reverse assumption.
Both are completed before the code is broken. This is to avoid bias creeping
in during the data analysis and writing.
Moderate-sized effects imply large studies.
What may turn out to be the most important statistical paper of 1996
was published in the Oxford Textbook of Medicine, of all places:
the convincingly-written article by Collins et al (1996) will surely
encourage tens of thousands of medical professionals towards
good statistical practice in their research.
One of the messages is that there may be a number of treatments
that are only moderately beneficial, but yet would be very worthwhile
because the disease is common or the treatment is cheap or both; and
which, because the treatment vs. control difference is not great,
require large sample sizes in order to establish statistical
significance of the difference. These sample sizes are so large
that they imply collaboration between many investigators.
Collins et al give clear and interesting examples of both large single
clinical trials (e.g., aspirin in treating heart attacks), and of
meta-analyses of many small trials (e.g., adjuvant therapy for early
breast cancer treated surgically).
The effect sizes in these studies were large enough to be well worth having
--- a reduction in 35-day mortality from 11.8% to 9.4% among patients
with acute myocardial infarction who were treated with aspirin, and a
reduction in 10-year mortality from 48% to 38%
among women aged 50+ with stage II breast
cancer who had tamoxifen included in their treatment --- yet were also
small enough that clinical trials of the usual sizes were too small to
It would not surprise me if there were to be a trend towards
simple, large, collaborative studies in many areas of
scientific enquiry besides medicine. Now, consider that very
many students conduct some sort of small research project
as part of their studies. I suggest that if different students in
different universities together decided on a common topic
and worked out a common methodology
(and stuck to it), then (i) the results would be
of much greater value than results from a single institution,
and (ii) the experience of collaborative work would itself
be valuable training, if indeed we are entering an era when
this will be increasingly common.
Collins R, Peto R, Gray, R, Parish S (1996). Large-scale randomized
evidence: Trials and overviews. In Weatherall D J, Ledingham J G G,
Warrell D A (Editors), Oxford Textbook of Medicine. 3rd Edition,
pp. 21--32. Oxford: Oxford University Press.
Don't get carried away with the statistical analysis.
Tell the story of your
research using the tools from your first course in statistics,
not your last --- that is, using nothing more complicated than
well-chosen descriptive techniques plus the concept of standard
error. As much as is practicable, present your results in such a way
that the reader can see that the conclusions from your more
complicated calculations are probably correct.
Suppose your research is experimental; and involves obtaining
a measurement (e.g., an accuracy score) from each subject in
each of four conditions; that the four conditions consist of each
combination of two levels (e.g., big and small) of one factor
with two levels (e.g., light and dark) of another factor; that the
subjects are in two groups (e.g., males and females); and that interest
centres on whether the average interaction effect is different
for males from what it is for females. Such an experimental question
would be recognised by some people as being about the three way
interaction in a three-factor experiment, one factor being a
between-subjects factor and two of the factors being within-subjects
factors. Such people would know that a concise way of analysing
the data and presenting the results would be via Analysis of Variance.
I recommend: (a) Yes, do that. (b) But also, present your results
in a way that enables them to be roughly checked. (Otherwise, your
readers will presume you've got them wrong.) In the example,
the interaction can be calculated for each subject. It is the
difference between two differences, [(big, light) - (big, dark)] -
[(small, light) - (small, dark)]. So calculate this for each subject,
and summarise the values for the males by their mean and s.d., and
the values for the females likewise. It will then be clear how similar
or different are the interactions for the two genders.
It might be worth going to quite some trouble to make sure you,
your supervisor, and your examiners agree on this one. Having gone
to the trouble of really understanding your data and then explaining
it comprehensibly, it would be vexing to come across an examiner who
wants to see 20 pages of ANOVA tables. Or vice versa.
Just occasionally, it may happen that careful attention to
notation in writing an equation will reveal that you really
did not understand something you thought you did (or even that
everyone else did not understand it).
This can happen with any type of mathematical equation, not
necessarily one involving random variables. But, to give
a simple example of this latter type, there might be an averaging process
somewhere, and one can sometimes get confused about what domain it is
that one is averaging over.
T P Hutchinson, 21.July.97: email@example.com