DB #3 Qualitative Data Collection

1

Zina, O. (2005).  Researching real-world problems: A guide to methods of inquiry. Sage.

CH. 6

Selecting Appropriate Data Collection Methods

 

Chapter Preview

Data Collection: Working through the Options

Reaching out to the Masses: Surveying

The Question and Answer Process: Interviewing

Taking It All In: Observation

Collecting Data without Intervention: Unobtrusive Methods

Manipulating the Environment: Experimentation

Comparing Methods of Data Collection

 

‘It is a capital mistake to theorize before one has data.’

– Sir Arthur Conan Doyle as Sherlock Holmes

 

DATA COLLECTION: WORKING THROUGH THE OPTIONS

 

Data: Factual information, especially information organized for analysis, reasoning or decision making.

I think Sherlock Holmes was right; the key to the whole research thing is ‘data’. Your ability to answer a research question is highly reliant on getting your hands on, and make sense of, data. Now for researchers working with real-world problems there are a range of possibilities for data collection, and this chapter will work you through the fundamentals. Being familiar with the basic process of data collection and having a critical understanding of the pros and cons of various collection strategies, puts you in a strong position to direct further readings (offered at the end of the chapter) and confidently and appropriately design and implement your study.

Options and possibilities

Trying to get your hands on data, and even deciding on appropriate methods of data collection, is something we actually do in the real world more often than you might think. Take, for example, the following scenario. You suspect your partner is cheating on you (bastard! – oh sorry, I’m being a bit sexist … wench!). But you’re not 100% sure, so you need confirmation. You run through your options. As you see it, you can:

 

Get on the phone and ask every single one of your friends if they think it could possibly be true. Now you might get a wide range of opinions here – but some of your friends might not want to go there.

Ask directly – grill both him and his friends. While this might get you to the source, the challenge will be trying to get an honest answer out of the lying son of a #*^%.… And as for his friends, do you really think they’re going to go behind his back?

Spy on him or hire a private investigator. Now you might actually feel a bit guilty about this, particularly if you’re wrong. And he finds out what you’ve been up to, you’ll be the one in the doghouse. On the other hand, he certainly can’t lie if caught in the act.

Rifle through (I’ll change gender for a moment just to make it fair) her personal belongings to see if the cheating hussy left any evidence behind. Not a bad strategy – and one you won’t be able to stop yourself from doing anyway.

Set him up and see if he takes the bait. You’re in control here – but it’s risky, you could be asking for trouble.

Well, you see, these options (along with their associated opportunities and challenges), actually represent the five methods of data collection we’re going to work through in this chapter. The first option, ‘get on the telephone’, equates to a survey. The second option, ‘ask directly’, relates to interviewing – both a respondent (him) and key informants (his friends). The third option, ‘to spy’, is a form of observation, while the fourth, ‘rifle through belongings’, is a method of unobtrusive data collection. Finally, the fifth option, ‘the set-up’, is actually a type of experimental design.

So, in our little scenario, which approach or approaches should you go for? Well, this is the dilemma. Each option has its advantages and disadvantages and each will bring something a bit different to the table. So you need to weigh it up. You need to consider: (1) which option will give you the most accurate information; (2) which will give you the most satisfaction; (3) which you find morally, ethically and legally acceptable; and (4) which are most practical. In short, the answer is, it depends; and it depends on any number of factors.

And this is precisely the case in any data collection scenario. One method of collection is not inherently better than any other; each has pros and cons that must be weighed up in view of a rich and complex context, for this is precisely where real-world problems sit. In fact, the circumstances of the problem you are researching are likely to dictate that you look to more than just one method to get a full picture of a problem situation.

The data collection process

No matter what method or methods of data collection you decide to use in your research study, it is important to recognize that, unlike in the scenario above, the collection of research data needs to be rigorous. In fact, it is the systematic and rigorous nature of your approach that will help define your data as more than anecdotal evidence, and act to give credibility to your eventual findings. Data collection is a complicated process that needs to be tackled in a thoughtful and methodical manner.

Now this chapter does cover five distinct data collection strategies – but in terms of process, the methods actually have quite a bit in common. As covered in Box 6.1, each requires: thorough and thoughtful planning; meticulous and well-considered development; effective and sufficient piloting; reflective and weighed modification; deliberate implementation and execution; and appropriate management and analysis.

 

Box 6.1 Data Collection Basics

 

Step 1: Planning – consideration of ‘who’, ‘where’, ‘when’, ‘how’ and ‘what’

A crucial part of the process. The success of your approach will hinge upon the forethought you’ve put into the planning process. You will need to consider:

 

Population and sample/respondent/participants – this is who (or what) you plan to speak about (population) – and gather data from (sample). It can include people, documents, communities, artefacts, etc.

Access – you’ll need to think about how you will locate and access facilities, people, places, documents, etc. You’ll also need to consider any language or cultural barriers that might keep you from fully accessing your sample.

Your role – the challenge here is presentation of self. Will you be an objective scientist, change agent, confidante?

Your biases – this involves recognizing and controlling for your own subjectivities in ways that can best ensure the credibility of any data you may collect.

Ethics/ethics approval – this involves considering any ethical dilemmas inherent in your project, and asks if you’ve sought and received appropriate ethics approval.

Method – involves thoughtfully considering potential methods of data collection and analysis. Also involves conducting an inventory of the skills/resources needed to carry out your project.

Data – this means knowing exactly what it is you are looking for or trying to find. You will also need to consider if the shape and form of the data you will collect will be compatible with intended modes of analysis.

Details – this refers to thinking through all the details, including tools, timing, location, recording methods, etc.

Contingencies – you need to be ready for the unexpected, the unplanned and the unfortunate. This means having a back-up plan ready to go.

Step 2: Developing – preparing the ‘tools’

The tools you will need to develop will depend on method. They may involve interview schedules, observation checklists, questionnaires, experimental checklist etc. In developing tools, you will need to:

 

Explore existing possibilities – have you sought out existing tools? Remember you don’t need to reinvent the wheel. If a relevant tool exists, such as a somewhat applicable questionnaire, you can adopt, adapt and modify.

Prepare data collection tools – this might involve generating questions for questionnaires and interviews, or creating checklists for observation. You will need to generate themes, work through questions/categories, engage in critical review, and offer logical and systematic presentation.

Prepare any data recording tools – if note taking, consider/develop a form that can aid this process. If audio or video taping, be sure to acquire and become familiar with the equipment.

Step 3: Piloting – conducting a ‘trial’

Giving thoughtful consideration to planning and development is essential – but not sufficient. The only way to really know if something is going to work is to give it a try.

 

Have a run-through – if interviewing or surveying, try piloting your process with a few respondents whose background is similar to those in your ‘sample’. If observing, a trial run will allow you to see if there are any problems in a real-world context. The same is true in working with documents or artefacts. A pilot will allow you to assess the effectiveness of your proposed method.

Reflect – regardless of method, reflect on the piloting process and note any difficulties you encounter. Depending on method, this might include problems with access, response rates, cultural ‘ignorance’, bias, comfort zones, recording/note taking, objectivity, conversational flow, ambiguities, credibility etc. Also review your data and note any difficulties you might encounter in making sense of your record.

Seek feedback – attempt to gather feedback from your pilot group. For surveys and interviews this will involve asking about question clarity, structure, introductory information, instructions/prompts, time taken, or anything else you want to know or your group wants to discuss. In the case of observation, you can attempt to confirm your record by checking with an insider, asking another observer to compare notes, or triangulating your observational data with other data types.

Step 4: Modifying – refining your approach

Remember, collecting data is not a skill you are born with. Review and refine until you are comfortable with the process and data collected.

 

Make modifications – this will be based on the feedback from your pilot group, as well as the quality of the data generated.

Back to the start? – if the need for modification is substantial, you may need to revisit your planning, development and piloting process. This may even involve a return to the ethics committee.

Step 5: Implementing – the actual process of data collection

As you can see, this is step five. So you should’ve gone through four steps before getting here. Only now are you ready to go and collect some data. Now there are many researchers who want to jump in at this point without fully engaging with the steps above. But trust me, short-cutting the process is a sure-fire way to get into trouble.

 

Administration – depending on methods, this might involve mailing/distributing surveys, setting up appointments, arranging site access, organizing experimental logistics, or gathering data/artefacts.

Data collection – this can be done by a number of approaches, including surveys, interviews, observation, unobtrusive methods and experimentation. Each of these approaches is taken up in detail later in this chapter.

Step 6: Managing and analysing – keeping track and making sense

I will cover this more fully in Chapter 11, but suffice it to say that unless your data is effectively managed and thoughtfully analysed, all the hard work above will be wasted.

 

Organize/collate your data as soon as possible – when the time comes to work with your data, nothing is worse than a partially forgotten conversation or illegible notes. Be systematic and organized – use a database if appropriate.

Statistical and/or thematic analysis – it’s time to see what your data yields. Methods of analysis will vary according to data type. But all analysis should work towards addressing your research questions in insightful ways.

REACHING OUT TO THE MASSES: SURVEYING

 

‘The real now talks constantly. News reports, information, statistics, and surveys are everywhere.’

– Michel de Certeau

You’re probably all too familiar with surveys and surveying. I hate to admit it, but when I was an undergraduate at Rutgers University, I actually worked for a market research company. Yes, I was one of those highly annoying people who called at dinner time and asked if you’d mind ‘answering just a few short questions that should only a take a couple of moments of your time’. As the French author Michel de Certeau said, ‘surveys are everywhere’. Market research, political polling, customer service feedback, evaluations, opinion polls, social science research – when we want to know what the masses are thinking we tend to survey.

 

Surveying: The process of collecting data by asking a range of individuals the same questions related to their characteristics, attributes, how they live, or their opinions.

Survey pros and cons

There really are no easy answers when it comes to selecting methods of data collection. There will always be tradeoffs between opportunities and challenges, and this is certainly true when thinking about conducting a survey. While surveys can offer much to the production of knowledge, their reputation for being a relatively simple, straightforward and inexpensive approach is not really deserved – they can actually be a somewhat thorny and exasperating process, particularly if done well.

Now on the plus side, surveys can:

reach a large number of respondents

represent an even larger population

allow for comparisons

generate standardized, quantifiable, empirical data

generate qualitative data through the use of open-ended questions

be confidential and even anonymous

They do, however, have their downside. Constructing and administering a survey that has the potential to generate credible and generalizable data is a truly difficult task. It is not something you can do off the top of your head. Challenges associated with surveying include:

capturing the quantifiable data you require

gathering in-depth data

getting a representative sample to respond

getting anyone at all to respond!

needing proficiency in statistical analysis

only getting answers to the questions you’ve thought to ask

going back to your respondents if more data is required

Survey options

You may think that all surveys are the same, but there are actually quite a number of possibilities. You will need to work through a few key issues before you can determine which will best suit you and your research agenda. Table 6.1 covers fundamental survey types by the issues you will need to consider.

Survey construction and administration

As a research supervisor, I have certainly found survey construction to be the activity most underestimated in terms of difficulty. And that’s not just by students – professional researchers can also have a hard time getting this right. Yet it is essential because the data that even a poor survey generates can be reported as truth and used in all kinds of decision making processes.

The best advice I can give is to take it in steps and get lots of feedback from your peers and other researchers. You’ll also need to have a trial run. Piloting might be important in all data collection methods – but in surveying, I’d say it is absolutely crucial. It is almost impossible to get a questionnaire just right the first time around.

TABLE 6.1 SURVEY TYPES

Will your survey simply describe or attempt to explain?

Descriptive surveys: the goal is to get a snapshot – or to describe your ‘respondents’ by gathering: demographic information i.e. age, socio-economic status and gender; personal information i.e. political opinion or use of illegal drugs; and attitudinal information i.e. attitudes towards multinational corporations, greenhouse gases, or health care costs

A classic example here is political polling, which attempts to describe voter intentions

Explanatory surveys: the goal is to gather descriptive data, but also establish cause and effect. In other words, to figure out why things might be the way they are

A recent Australian survey collected data describing attitudes to the Iraq conflict – as well as data used to establish what might shape and form those attitudes, e.g. personal experience, familial attitudes, and political leanings

Do you plan to sample or ask everyone in your population?

Census: a survey that does not rely on a sample. In other words, a survey that covers every single person in a defined population

The US census is a classic example. A smaller-scale census might be all the students in a particular school

Cross-sectional surveys: surveys that use a sample or cross-section of respondents to represent a target population. The goal to be able to ‘generalize’ findings

Most surveys fall under this category, e.g. a community survey that targets only 1 in 10 households – but aims to represent the entire community

Will you survey over a period of time – and if so, do you want to explore changing times or changing people?

Trend surveys: a trend survey asks the same cross-section (similar groups of respondents) the same questions at two or more points in time. The goal here is to see if classifications of individuals change over time

An example here is a three-phase survey conducted over a 20-year period (1986, 1996, 2006) that asks newlyweds their attitudes towards marriage. The goal is to assess if attitudes of newlyweds in the new millennium are the same as attitudes of newlyweds in the 1980s and 1990s

Panel study: a panel study involves asking the same (not similar) sample of respondents the same questions at two or more points in time. The goal here is to see if individuals themselves change over time

Using the example above, if you had surveyed newlyweds in 1986, you would survey these same individuals in 1996 – 10 years after their marriage – and again in 2006 in order to assess attitudinal shifts as individuals get older

How do you plan to administer your survey?

Face to face surveys:

Pros: good response rate, allows rapport and trust to be established, can motivate respondents, allows for clarification, prompting, probing, and the reading of nonverbal cues

 

Cons: can be lengthy and expensive, can limit geographical range, does not assure anonymity or confidentiality, and requires surveyor training

One example here is the ‘mall’ or ‘supermarket’ survey where you are stopped by someone with a clipboard ready to ask you a series of questions

Telephone surveys:

Pros: relatively inexpensive, allows wide geographic coverage, offers some assurance of anonymity and confidentiality and allows for some clarification, prompting, probing

 

Cons: response rate can be low – it’s easy to catch people at a bad time, respondents can hang up on you if they have had enough, and you are limited to surveying only those with a telephone

In market research the telephone tends to be the mode of choice

Self-administered surveys:

Pros: can offer confidentiality/anonymity, allows wide geographic coverage, and gives respondents the opportunity to answer in their own time

 

Cons: response rates can be very low, does not allow for clarification, and can end up being costly

This can include both snail mail and email. Email can save you thousands in printing and postage costs, but you are limited to surveying within ‘online’ populations. Additionally, the proliferation of ‘spam’ mail means that unless your respondents know you, your survey may not even get looked at

As discussed more fully below, developing a questionnaire will entail: (1) formulating your questions; (2) deciding on response categories; (3) providing background information and clear instructions; (4) making determinations about organization and length: (5) working on aesthetically pleasing layout and design; and finally (6) administrating the survey. Which, of course, all needs to be done in conjunction with several stages of seeking feedback, piloting and redevelopment.

Formulating questions

What is it that you want to know? Who are your respondents and what do you think they can tell you about what you want to know? What do you think is the best way to go about asking them? In order to formulate your questions, you need to be able to answer all of the above. Only then will you be able to search for relevant questions or attempt an initial drafting of original questions. Now there is certainly more than one way to ask the same question. In fact, the possibilities are almost endless. And the dilemma here is that subtle (or not so subtle) differences can affect the data you generate. So what do you do?

Well survey construction is a very well researched topic. In fact, there are volumes written about it, and at the end of this chapter I provide a few key references to which you can refer. Within this text, however, Box 6.2 offers a distillation of the most fundamental ‘rules’ related to question wording that you can apply. The aim here is to help you avoid the pitfalls of leading, offending or confusing your respondents.

Box 6.2 Questions to Avoid

 

Good questions should be unambiguous, inoffensive and unbiased. But this is actually easier said than done. It’s not difficult to fall into the trap of constructing question that are:

Poorly worded

 

Complex terms and language – big words can offend and confuse. If they’re not necessary, don’t use them. This is my favourite example:‘Polysyllabic linguistic terminology can act to obscure connotations,’ vs. ‘Big words can be confusing.’

Ambiguous questions – frames of reference can be highly divergent, so writing an ambiguous question is easy to do. Take, for example, the question ‘Do you use drugs?’ ‘Drugs’ is actually an ambiguous term. Some respondents will only consider illegal drugs, while others may include prescription drugs. Others might use a frame of reference that includes alcohol and/or cigarettes.

Double negatives – like many people, I have a hard time with double negatives. Take the following Yes/No question, ‘Do you disapprove of the Government’s new policy on Aged Care?’ To state that you do approve, you’d have to choose ‘No’, which can be quite confusing.

Double-barrelled questions – this is when you ask for only one response to a question with more than one issue. For example, ‘Do you consider the President to be an honest and effective leader?’ Respondents may think yes, effective – but definitely not honest.

Biased/leading/or loaded

 

‘Ring true’ statements – these are statements that are easy to agree with simply because they tend to ‘ring true’. Some good examples here are agree/disagree statements,’ like ‘You really can’t rely on people these days,’ or ‘Times may be tough, but there are generally people around you can count on.’ Both of these somewhat opposite statements are likely to get high percentage of ‘agrees’ because they tend to sound reasonable.

Hard to disagree with statements – these are statements where your respondents are likely to think ‘Yes that’s true, BUT …’ They are not, however, given a chance to elaborate and are forced to either agree or disagree. For example, ‘It is good for young children if their mothers can stay at home through the week.’

Leading questions – leading respondents in a particular direction can be done unintentionally, or can be done intentionally for political purposes. Consider how the wording of these agree/disagree statements might affect responses. ‘Protecting defenceless endangered species from inhumane slaughter is something the government should take seriously,’ vs. ‘The protection of biodiversity should be a government priority.’

Problematic for the respondent

 

Recall dependent questions – these are questions that rely on memory. For example, ‘How many relationships have you had?’ Without boundaries such as level of significance or timeframe, this question can be easy to answer ‘incorrectly’.

Offensive questions – if respondents take offence to a question or a series of questions, not only are they likely to skip them, they may just throw out the entire survey. Offensive questions can range from ‘What do you think you did that caused you to gain so much weight?’ to‘How much money do you earn?’

Questions with assumed knowledge – try not to assume that your respondents know about, or are familiar with, the same things as you. Take for example the agree/disagree statement ‘Marxist theory has no place in 21st century politics.’ You shouldn’t be surprised to find out that a common response here is, ‘What kind of academic crap is this!’ – followed by a quick trip to the trashcan.

Questions with unwarranted assumptions – respondents are likely to be at a loss when it comes to answering a question that contains an assumption they do not agree with. For example the question ‘What was the most enjoyable part of your hospital stay?’ assumes that the respondent enjoyed something about their hospitalization.

Questions with socially desirable responses – this is more likely to be an issue in face to face surveying. For example, a respondent may be uncomfortable disagreeing with the statement ‘Do you think women serving in the armed forces should have the same rights and responsibilities as their male colleagues?’

Now working within these guidelines is a start, but is unlikely to be enough. Once you have drafted your questions run them past an experienced researcher or two. They are likely to pick up things you have missed. You can also trial your questions with your peers. They will certainly be able to tell you if you managed to confuse, offend, or lead them in any way. Finally, once you’ve made modifications based on feedback received, you will need to run a pilot study. The idea here is to distribute your survey to a small group of individuals whose characteristics match that of your sample and then thoroughly debrief with them. Remember – in the end it is not what you think, or even what your supervisor or peers think that counts. The only opinion that really matters will be that of your eventual respondents.

Response categories

As if getting your questions as precise and non-problematic as possible wasn’t enough, a good survey, and good survey data, is equally dependent on the response categories you decide to use. And there are a lot of things to consider here. For one thing, response categories will influence the data you collect. For example, if you add an ‘I’m not sure’ option to a controversial Yes/No question, it will affect your findings. Secondly, different types of response categories generate data with different types of measurement scale; and data with different measurement scales demand quite distinct statistical treatment. In fact, understanding the difference between nominal, ordinal, interval and ratio data (as discussed in Chapter 11) will definitely facilitate the process of survey construction, particularly determining response categories. But until you actually have some data to play with, understanding the relationship between data types and survey construction is quite abstract.

This makes conducting your first survey a real challenge. So again, I’ll turn to the need for a good pilot study. Not only will a pilot study allow you to assess your questions and response categories from the perspective of your respondents, it will also allow you to generate a mini data set that you can enter into a database, and work with statistically. This really is the best way to see how your data collection protocols, including response category determination, will impact on your analysis.

So what are the options when it comes to response categories? Well, as highlighted in Box 6.3, there are quite a few:

Box 6.3 Response Categories

 

Responses to survey questions can be either open or closed:

Open responses

Respondents are asked to provide answers using their own words. They can offer any information/express any opinion they wish, although the amount of space provided for an answer will generally limit the response. The data provided can be rich and candid, but can also be difficult to code and analyse.

Closed responses

Respondents are asked to choose from a range of predetermined responses. The data here is generally easy to code and statistically analyse. Closed response categories come in many forms, each with their associated issues.

 

Yes/No – Agree/Disagree:

Do you help your child with homework? Yes/No

Do you think children in first grade should be given homework? Agree/Disagree

While it can be easy to work with ‘binomial’ data (or data with only two potential responses), you need to consider whether respondents will be comfortable with only two choices. For example, in the first question, a respondent might be thinking ‘Does only two or three times a year count?’, or for the second question, ‘It depends on how much you’re talking about.’ A potential strategy is to offer a Don’t know/No opinion option – but this allows for a lot of ‘fence sitting’.

 

Fill in the blank:

How much do you weigh? ____________

Even a simple question like this (assuming your respondents know the answer and are willing to tell you) can lead to messy data. Will respondents write 90 kgs, 198 lbs, or 14 stone? Of course you can convert these answers to one system, but that isn’t going to be possible if they just put 90.

 

Choosing from a list:

Who do you enjoy working with the most?

There is an assumption here that there will not be any ‘ties’; you need to consider what you will do if more than one option is circled. You also need to make sure all options are covered (are exhaustive) and don’t overlap (are mutually exclusive). A potential strategy is to offer an ‘Other’ or ‘Other:_____’ option.

 

Ordering options:

Please rank the following according to how you think Government spending should be prioritized:

Health care  Education  Environment  Defence  Family allowance  Pensions

These questions tend to be quite difficult for respondents, particularly if lists are long. It’s worth remembering that if respondents get frustrated trying to answer, they are likely to leave the question blank, leave it half finished, or just write anything at all.

 

Likert-type scaling:

Likert scales offer a range of responses, generally ranging from something like ‘Strongly disagree’ to ‘Strongly agree’. In Likert scaling, you need to consider: the number of points you will use; whether you will force a side by using an even number of responses; and whether you think your respondents are likely to ‘get on a roll’ and keep circling a particular number.

Information and instructions

A survey instrument is not complete without some level of background information. This information is included: (1) to give credibility to the study; and (2) to make your respondents feel like they’re a part of something. In your background information it’s a good idea to include: the sponsoring organization/university; the survey’s purpose; assurances of anonymity/confidentiality; return information, including deadlines and return address; and a ‘thank you’ for time/assistance. This information can be included at the start of the survey, or as a cover letter.

Also crucial are your instructions. What might be self-evident to you may not be so obvious to your respondents. Instructions should introduce each section of the survey; give clear and specific instructions for each question type; provide examples; and be easy to distinguish from actual survey questions. In fact, I’d suggest using a distinct font – try changing the style, size, boldness, italics, underlining etc. It may take a couple of drafts to get your instructions as clear and helpful as possible. Be sure you seek advice and feedback from other researchers, peers and your pilot group.

Organization and length

Once you are comfortable with all the various elements of your survey, you will need to put it together in a logical format that is neither too long nor too short. Too short – and you won’t get all the data you need. Too long – and your survey might be tossed away, returned incomplete, or filled in at random. People might not mind spending a few minutes answering your questions, but ask for much more and they may not be bothered to help you out. Now appropriate length is another aspect of your survey you can assess in your pilot run. Be sure to ask your trial respondents what they thought of the overall length and the time it took to complete the survey.

In terms of logical organization, there are a few schools of thought. Some suggest that you start with demographics in order to ‘warm up’ your respondents. Others, however, suggest that you start with your topical questions and finish off with questions related to demographic information. What’s right for your survey will depend a lot on the nature of both your questions and your respondents. In fact, you may want to pilot two different versions of your questionnaire if you are unsure how it should be laid out.

There is one consistent piece of advice, however, and that is to avoid starting your survey with questions that might be considered threatening, awkward, insulting, difficult etc. It’s really important to ease your respondents into your survey and save sensitive questions for near the end.

Layout and design

All done! Well almost. You’ve written clear and unambiguous questions with appropriate, well thought out response categories that are accompanied by clear instruction and organized into a sensitive, logical and manageable form. And you’ve done this by going through multiple iterations taking into account as much feedback as possible. There’s only one thing left – aesthetics.

Aesthetics is important. Your survey needs to look professional – no poor quality photocopying, faint printing, messy and uninteresting layout etc. Respondents are more likely to complete a survey that is professionally presented. It is also worth keeping in mind that the potential for mistakes increases dramatically if surveys are cluttered, cramped, or messy. So the effort here is well worth while.

Administration

Now you’re done! You just need to get that survey out there and get people to answer it. To execute your survey you will need to:

 

Distribute your questionnaires, by mail, email, telephone, door to door, or face to face.

Collect your completed questionnaires.

Send out reminder letters if response rates are low.

Put a low response rate plan into action if not enough data has been gathered by your deadline.

Record and manage responses so they are ready for analysis.

THE QUESTION AND ANSWER PROCESS: INTERVIEWING

 

‘I like to listen. I have learned a great deal from listening carefully. Most people never listen.’

– Ernest Hemingway

Interviewing: the ‘Art of Asking’ or the ‘Art of Listening’? Well both are crucial to the interview process – but while we tend to spend plenty of time discussing the questioning side, I don’t think we spend nearly enough time on the listening end of things. According to Hemingway, ‘Most people never listen’. And unfortunately, there are too many researchers and interviewers out there who would rather talk than listen. Remember, your job is to talk only enough to facilitate someone else’s ability to answer. It is your interviewee’s voice that you are seeking, and it is their voice that needs to be drawn out.

 

Interviewing: A method of data collection that involves researchers seeking open-ended answers related to a number of questions, topic areas, or themes.

Interview pros and cons

What could be better than getting out there and actually talking to real people … asking them what they really think … finding out first-hand how they genuinely feel? Well, when you conduct an interview you’re able to put yourself in a position to see, hear and get a sense of your respondents. Sounds good, doesn’t it? And yes – interviewing has the potential to do all of the above. But like any other data collection method, its opportunities are balanced by a series of challenges. As they say … there’s no such thing as a free lunch.

The interview process can:

allow you to develop rapport and trust

provide you with rich, in-depth qualitative data

allow for nonverbal as well as verbal data

be flexible enough to allow you to explore tangents

be structured enough to generate standardized, quantifiable data

Now while many of these ‘pros’ are the result of the human element in interviewing – so too are the ‘cons’. The closer you become to your respondents, and the closer they become to you, the bigger the challenge you will face in managing the process. Such challenges include:

resisting the urge to lead your respondents

facilitating honest and open responses even though your interviewees may want to ‘impress’

figuring out how attributes such as race, gender, ethnicity, class and age of interviewer and interviewee alike might affect the interview process – and employing effective strategies for ensuring credibility.

Additional interview ‘cons’ worth considering include:

the potential for communication miscues

difficulties of working with a large or geographically dispersed sample

a lack of anonymity

Interview options

Words can conjure up images. What pops into your mind when you think ‘interview’? Well I think most would probably conjure up the image of a job interview. You know, that formal scenario where the interviewee has to get all dressed up, do the firm handshake, use a formal presentation of self – all while feeling quite nervous. Meanwhile, the interviewer sits behind a big desk or on the opposite side of a conference table, holds all the cards and is definitely the power person. So I guess I shouldn’t be surprised when new researchers subconsciously take this image with them into the research world. Without necessarily articulating this image to even themselves, they tend to think that this is how research interviews should unfold. But they don’t have to. Yes, research interviews can be formal – but as covered in Table 6.2, there are actually lots of options that might better suit your research agenda.

Conducting your interview

Intimidating. I don’t think there is a better word to describe what it feels like to conduct your first interview. No matter how well prepared you think you might be – you are still likely to feel nervous at the beginning, and wish you did things differently at the end. Now we tend to spend a lot of time in interview preparation getting our questions together or thinking about how we can cover our topics and themes. But conducting an interview is a much more complicated management task. You actually need to do three things at once. In addition to questioning, prompting and probing in ways that will help you gather the richest possible data, you also need to actively listen to, and make sense of, what your interviewee is saying – while at the same time managing the overall process so that you know how much time has passed, how much time is left, how much you still need to cover, and how you might move it all forward.

Well here’s a cliché for you – practice makes perfect! I don’t think anyone is born with an innate ability to conduct a good interview, but if you are aware of the key steps and can reflect on your experiences, it is a skill that will develop with time. In order to conduct a good interview you will need to do the following:

 

Prepare questions and/or themes – for a structured interview this will involve drafting and redrafting your questions, and making sure that you have not included questions that will be confusing, leading, offensive, or problematic for your interviewees. For a less structured interview, you will need to think about the themes you want to cover and whether you will put any boundaries on potential conversation. You may not get this just right on your first attempt – so it is essential to conduct a pilot before you construct your final draft.

Consider any translation needs – if you require a translator, you will need to consider the following:

will your translator translate for you and the interviewee on the spot?

will someone conduct the interview in the interviewee’s native language to be translated at a later time?

are you after a literal translation or do you want your translator to use some discretion and judgement in conveying meaning?

how will you manage the overall process, including gaining rapport, keeping on time, exploring tangents, keeping respondents focused etc.?

TABLE 6.2 INTERVIEW TYPES

Will you conduct your interview in a formal manner – or will it be more relaxed?

Formal: the interviewer attempts to be somewhat removed from the interviewee, and maintains distance and neutrality/objectivity. This is often done within a formal setting

This is the classic job interview. While formality can allow interviewers a high level of control – it can limit interviewee comfort, and possibly the free flow of information

Informal: bends or ignores rules and roles associated with formal interviewing in order to establish rapport, gain trust, and open up lines of communication. The style is causal and relaxed in order to minimize any gulf between the interviewer and the interviewee

Settings are not limited to an office and might occur over a beer at a bar, or while having a cup of coffee at the local preschool. The idea is to do whatyou can to get your interviewee chatting comfortably

Will your interviews be highly structured or more free flowing?

Structured: use of pre-established questions, in a pre-determined order, with a standard mode of delivery. Prompts and probes are also pre-determined and used under defined circumstances. Interviewers often call on a formal style to help them stay on track

Best suited for interviews where standardized data is a goal. Inexperienced interviewers generally feel most comfortable with this high level of structure

Semi-structured: use of a ‘flexible’ structure. Interviewers can start with a defined questioning plan, but will shift in order to follow the natural flow of conversation. Interviewers may also deviate from the ‘plan’ to pursue interesting tangents

The advantage here is being able to come away with all the data you intended – but also interesting and unexpected data that emerges. This style of interviewing can take a bit of practice

Unstructured: attempts to draw out information, attitudes, opinions and beliefs around particular themes, ideas and issues without predetermined questions. The goal is to draw out rich and informative ‘conversation’. Often used in conjunction with an informal structure

Most interviewees enjoy this type of interview because it allows them to ‘talk’ and really express their ideas in a way not dictated by the interviewer. Interviewer challenges here are to avoid leading the conversation and to keep it focused enough to get the data needed

Will you interview one person at a time – or will you attempt to tackle a group?

One on one: an interaction between an interviewer and a single interviewee. It is thought that ‘one on one’ allows the researcher control over the process and the interviewee the freedom to express their thoughts. ‘One on one’ can also involve an additional person such as a translator or note taker

One on one interviews are generally face to face, but can also be done over the telephone in order to increase geographical range or capture a ‘difficult to get hold of’ respondent. The lack of nonverbal cues in telephone interviews, however, can be a challenge

Group: interviewing more than one person at a time. Can be done in a formal structured way, or may involve a more open process where the researcher acts as a moderator or facilitator. In this less structured approach, interviewees are often referred to as a ‘focus group’

Not only can a group interview save time and money – it can really get people talking. Some, however, might feel unheard or marginalized. Group interviews can be difficult to follow, so most interviewers attempt to preserve raw data by tape recording

There are no rights and wrongs here. It is the context of your particular research question that will determine the best course of action, and you may need to trial a couple of processes before you know which way to go.

Decide how you will record responses – recording responses can be done in a number of ways; you may need to trial a couple of recording methods in order to assess what is best for you and your research process. Options include:

 

note taking – this can range from highly structured, e.g. filling in a form as the interviewee speaks, to unstructured, e.g. mind mapping concepts or jotting down thoughts, analogies, metaphors, etc. during or after an interview. Keep in mind that note taking is actually a preliminary form of analysis – you have thought through and made decisions about your data, e.g. what to note and how to note it. You will need to consider whether there is value in also capturing raw data.

audio recording – this allows you to preserve raw data for review at a later date. Interviewers are therefore free to focus on the question/answer process at hand. Disadvantages, however, include the unease it can cause for the interviewee, an inability to capture nonverbal cues, potential equipment failure, and the cost of data transcription.

video taping – offers the added bonus of being able to record visual cues, but is more intrusive, is prone to more technical difficulties and can generate data that is difficult to analyse.

In most situations you will be responsible for both conducting an interview and capturing responses, but under some circumstances you may use a note taker. Using a person to take notes or record your interview can allow you to focus and engage more fully in listening and directing your interview. But as well as considering resource implications, you need to carefully consider whether a third party is likely to have an affect on the respondent and the interview process.

Decide on presentation of self – how will you present yourself? How will you strike a balance between formality and rapport? Is your interview style/research goal better suited to officiousness or informality? What tone of voice will you use? Will you joke around? Also consider body language. Reading nonverbal cues (while your interviewee is reading yours) is worth thinking about. Are you both making eye contact, looking down, looking around, picking your nails, coming across aggressively, looking relaxed?

As the interviewer, you are in a position of power. Attempting to negotiate this position of power in order to facilitate an interviewee’s ability to answer questions with honesty and openness should be a central consideration in interview planning.

Take care of preliminaries – quite a few crucial things need to come together before you are in a position to ask your first question. You will need to:

 

 

make appointments – allow for travel time, interview time and wait-around time

arrive on time – building rapport can be a real challenge if you keep someone waiting; and if you miss an appointment altogether you may not get a second chance

set up and check any recording equipment – you can do this in advance or, if done efficiently, when you first arrive for your interview

establish rapport – this includes introductions, handshakes, small talk and expressions of appreciation

introduce the study – this includes reviewing who you are, the purpose of the study, why involvement is important, and approximately how long the interview will take

explain ethics – this can involve assurances of confidentiality, the right to decline to answer any particular questions and the right to end the interview upon request.

Ease your respondents into the interview – finally you can get down to business. As with surveying, it is important to ease your way into main questions and themes. If you start off with a ‘sensitive’ question or one that might be considered threatening, you may find yourself facing an uphill battle for the remainder of the interview. In fact, it can be easy to get an interviewee off-side, so it’s well worth considering how you might handle such a situation.

Ask questions that facilitate answers – if you ask a Yes/No question, expect a Yes/No answer. Try to ask questions that open up conversations and draw out rich responses. Questions should create possibilities, open up options, dig below the surface and lower defences.

Keep it flowing – this involves the use of prompts, that is, giving the interviewee some ideas that might jog a response, and probes, which are comments and questions that help you dig for more, i.e. ‘tell me more’, ‘really’, or ‘why?’. Sometimes probes can be an inquisitive look or a few moments of silence.

Keep on track/explore tangents – if you are conducting a structured interview and have a limited amount of time, you will want to make sure you are keeping your interviewee on track and moving at a good pace. If your interview is less structured, you may find yourself wanting to explore interesting tangents as they develop. The trick here is to be mindful of the time, and be sure you end the interview with the full range of data you aimed to gather.

Be true to your role – if you are using a formal process and are attempting an objective stance, you will want to consider how you can manage the process without directing responses. If, however, you accept that your own ‘subjectivities’ will be part of the interviewing process, you will need to consider and openly report on how your engagement might influence the conversation.

Wind down/close – winding down involves questions that ‘round off’ an interview and asks respondents if there is anything else they would like to cover, contribute, or clarify. The interview then ends by thanking your interviewee for their contribution and their time, and asking them if it might be possible to contact them again if you need to ask any further questions, or need to clarify any points. It’s also good practice to offer something back, for example, a copy of your completed report.

TAKING IT ALL IN: OBSERVATION

 

‘He plies the slow, unhonored, and unpaid task of observation … He is the world’s eye.’

– Ralph Waldo Emerson

It’s easy to overlook observation as a potential data collection method – surveying and interviewing can tend to corner the social science research market. But I can give you three good reasons for thinking about conducting an observational study. The first is that there are times when you need to ‘see it for yourself’ – having it explained to you just isn’t the same. The second reason is that the gulf between what people ‘say they do’ and what they ‘actually do’ can be far and wide. And finally, data collection through observation generally takes place in the real world, not a constructed research world. You are out there in the field, right at the heart of where your research problem sits.

Observation allows you entry to this real world, and invites you to take it all in; to see, hear, smell, feel and even taste your environment. It allows you to get a sense of a reality and work through the complexities of social interactions. So in the words of Emerson, why not think about being the ‘world’s eye’.

 

Observation: A systematic method of data collection that relies on a researcher’s ability to gather data through their senses within real-world contexts.

Observation pros and cons

Who knows, maybe I’ve already convinced you into thinking ‘yes, a method that can

explore what people actually do – and not just what they say they do

allow you to take it in for yourself and

get you out there in the field

sounds exciting.’ And you may become even more convinced when you find out that observation is also a method that can:

allow you to develop rapport and trust

be flexible enough to let you explore tangents

provide both rich, in-depth qualitative data and standardized, quantifiable data

allow for nonverbal as well as verbal data

But because we’re familiar with the general concept of observation, there’s a tendency to think that using this technique as a research tool will be pretty straightforward. There is a real challenge, however, in taking something we do on a daily basis and converting it into a rigorous research method. In relation to observation, these challenges include:

designing a protocol that can credibly capture the data you require

making sure your biases do not colour your observations

avoiding the dilemma of having an impact on the researched

building trust and getting people to act naturally

protecting confidentiality and/or anonymity

Observation options

When talking about options in observation studies, the two ends of the spectrum could not be further apart. At one end, you might have a psychologist who is holding a clipboard and watching a series of interactions from behind a one-way mirror. At the other end is your anthropologist who has lived in a remote Papua New Guinea village for the past 15 years and is dedicated to understanding the reality of this village from the perspective of the observed.

Now, on the surface, similarities may seem few and far between. In fact, these extremes are often treated as two distinct methods of data collection derived from diverse paradigms and disciplines. But when you get down to the brass tacks of researching real-world problems, I think you’ll find that these extremes do sit on a continuum. Table 6.3 covers the key issues you will need to negotiate in order to determine how your own observation processes can best unfold.

TABLE 6.3 OBSERVATION TYPES

As an observer will you attempt to be removed or immersed? In other words, will you become a participant in the environment you are studying?

Non-participant: researchers do not become, nor aim to become, an integral part of the system or community they are observing. The observer is physically present but attempts to be unobtrusive

Non-participant observations tend to ccur over a fixed time period and are ften highly structured

Participant: researchers are, or become, part of the team, community, or cultural group they are observing. The goal is to preserve a natural setting and to gain cultural empathy by experiencing phenomena and events from the perspective of the observed

Participant observation can involve irge emotional and time commitments. Observers may be outsiders who ttempt to become insiders, or they an be insiders who decide to study ieir own, e.g. a member of workforce, community, or church

Will you conduct your observations in a covert fashion, or will you offer full disclosure?

Candid: researchers offer full disclosure of the nature of their study; the role the observations will play in their research; and what they might expect to find through the observation process

While being candid allows observers to take notes on site, the observed can feel under surveillance, and may ot act ‘natural’

Covert: researchers do not disclose the nature of their study to those they are observing; they may not even disclose that they’re undertaking a study at all

It can be difficult to get ethics approval for covert studies since they breach the core ethical principal of informed consent

Will you use highly structured or unstructured observation techniques?

Structured: predetermined criteria related to people, events, practices, issues, behaviours, actions, situations and phenomena are used to collect data in a highly systematic fashion

Checklists or observation schedules are prepared in advance, and researchers attempt to be objective, neutral and removed in order to minimize personal interactions

Semi-structured: observers use, but are not limited to, predetermined criteria

Observation schedules or checklists are used to organize observations, but observers also attempt to record the unplanned and/or the unexpected

Unstructured: observers attempt to observe and record data without predetermined criteria

Observers can record all observations and later search for emergent patterns, or they make judgement calls on the relevance of initial observations and attempt to focus any subsequent observations and reflections

Figure 6.1 actually takes this a step further and combines candid and covert strategies with varying levels of participation to offer four major strands of observation studies. Keep in mind that while these four options cover the basic possibilities, the subtleties of managing the process sit squarely with the researcher. It won’t always go smoothly and you need to be ready for the unexpected. For example, you will need to have a plan you can put into place if your covert study suddenly becomes exposed. And what if, as a non-participant, you can’t help yourself and start to participate? Or what if you get too immersed in the culture you are studying and begin to have second thoughts about your research role? These can be huge challenges and may require you to rethink your methodological design.

FIGURE 6.1 FOUR MAJOR STRANDS OF OBSERVATION STUDIES

Now there is a bit of a paradox here. The more entwined you become with the researched, the richer and more meaningful the data you might generate. But this entwining is also likely to make it a more difficult process to navigate. The key to gathering credible data through observation is your ability to think through such issues, to plan with care, and to exercise considered flexibility.

The observation process

Okay, a common feature of all observational studies is that they attempt to document what people actually do, rather than what they say they do; observational studies rely on actual behaviour. That means there are no tools that you can use to generate particular responses from the observed. There are no ‘questions’; it is simply the observed doing what they do, and observers taking that in, noting it and making sense of it.

Now the perceived advantage here is ‘genuineness’, but people don’t always act the same when they know they’re being observed. How genuine any behaviours might be can depend on both the role of the observer and the nature of the study. So as discussed below, you will really need to think this through, weigh all the pros and cons, and understand the intricacies of the process.

Planning

As in any data collection method, observation begins with planning. The following questions may assist you in preparing to ‘do’ observation.

 

Do your goals and context lend themselves to an observational study that is candid or covert; participant/non-participant; structured/unstructured?

Do you have/can you get required access? Can you arrange admittance and acceptance into the groups/situation/activities you wish to observe? Are there any potential language and/or cultural issues likely to affect the process? Can you get past ‘gatekeepers’? Will you be welcomed? Will you be able to build trust?

Have you considered details? What timeframe will you be working towards? Will you observe on one occasion, multiple occasions, or will your study involve prolonged engagement? How will you record your data? Do you need to prepare an observation schedule/checklist? Or if unstructured, do you need to articulate any relevant themes you wish to capture?

Will you need to seek ethics approval? The potential for ethical irresponsibility exists in any data collection method, and observation is no exception. But the potential covert and/or participatory nature of an observation study brings certain issues directly to the fore. For participant studies, you will need to consider whether immersion will have a physical, mental or emotional toll on the observed and/or you as the observer. For example, observers may find themselves immersed in a culture that may be dangerous; they may feel pressured to become involved in immoral/illegal activities; and they may feel stressed when they need to leave the setting and report findings.

Issues related to covert studies include justifying and getting approval for a study where there is a lack of informed consent. While some ethics committees are loath to do this under any circumstances, others will consider such studies if the researcher can give convincing assurances related to the physical, mental and emotional welfare of the observed and observer, protection of confidentiality, and perceived societal benefits.

Observing

When it comes time to begin your observations, the exact protocol you will use will be highly dependent on the type of observational study you plan to conduct. Most candid observations, however, begin by attempting to build rapport and gain trust. The idea is to try to make the observed feel as comfortable as possible; in fact, comfortable enough to carry on as if you weren’t even there.

The next step is opening your eyes, ears and mind to all that is going on around you. What do you see, what do you hear, what do you sense? We tend to be a visual society, so it’s important to make sure you’re taking it in through your full range of senses. And this can take time. Because you’re not directing the process, you need to be prepared to make a significant investment in order to get the data you need. In fact, unless your design sees you observing for a predetermined period of time, it pays to look for saturation (your observations no longer yield new knowledge) before ending the process.

Now keep in mind that we don’t all take in or perceive the world in the same way. Some of us are tuned into the bigger picture, while some of us concentrate on separate components. Some like to take in the world by looking around, some like to listen. Others understand best by moving, doing and touching. So when it comes to observation, it’s not only possible, but in fact probable, that two observers in the same situation will take things on board in quite different ways. Attempting to control for this is important. If your observations are structured, an observation schedule that requires information to be gathered through a variety of senses can ensure you don’t miss any potential sources of data. In a less structured study, the key will be your ability to critically reflect on your data collection processes and make any necessary modifications.

Recording

There are actually two quite different strategies for recording observations. The first involves the capture of raw data by things like photography, audio and/or video recording. This approach allows observations to be ‘preserved’ in a raw form so that they can be reviewed and used at a later date. These methods, however, demand the use of ‘equipment’ and can be considered intrusive. The second strategy is note taking or journaling. These methods can capture anything from descriptive and formal accounts of space, actors, acts and events – to much more interpretive narrative accounts that include goals, feelings and underlying ‘stories’. The form also varies and can range from coded schedules and quantitative tallying – to pictures, concept maps and jotted ideas.

The recording method (or methods) you will need to adopt will vary depending on the level of participation, openness and structure in your observational processes. For candid studies, the use of an observation schedule that you fill in as observations occur might be appropriate, as would the use of photography, audio and video recording. For studies that involve high levels of immersion and are perhaps covert, you might want to note, journal, doodle, or map your observations when you are removed from your observational setting and have a level of privacy. Your circumstances may also see you looking to employ a combination of the above strategies. Now whatever recording methods you choose to adopt, it will be important to record your data in as systematic a fashion as possible. After all, this is data you will need to analyse in the future.

Reflecting

While not always conducted as a ‘pilot’, there is still a need to review, reflect on and modify your observational methods. Such modifications are generally based on difficulties you encountered in your initial observation work, for example difficulties with access, timing, cultural ‘ignorance’, comfort zones, recording/note taking, roles, objectivity, etc. It also pays to review your observation records and assess if they make sense, and are logical, rich and complete.

You should also look for ‘bias’. It’s very easy to see the things you expect to see and hear the things you want to hear. That’s why doctors conduct double-blind studies in drug trials where neither the doctor nor the patient knows whether they have been given a placebo or the real thing. It is the only way to control for expectations.

Before you go out in the field, it’s well worth consciously brainstorming your own expectations. You can then brainstorm a range of alternatives, so that you’re less likely to observe and reflect on your observations in ways that confirm what you already suspect.

Authenticating

It can be hard to assess whether you’ve been able to control for your biases and generate credible data by reflection alone. There are, however, a number of strategies that can be used to ensure thoroughness in data collection, and confirm the authenticity of reflections.

Thoroughness of observational methods can be facilitated by working towards:

 

well-designed method – rigorous protocols are essential

broad representation – observing multiple events and various people

prolonged and persistent engagement – getting credible observational data takes commitment

crystallization and saturation – observing until a ‘story’ comes together or until new data ceases to emerge

Strategies for authenticating data include:

 

triangulating your observational data with other data sources – assessing whether data generated from interviews, surveys or documentary review ‘gels’ with your observations

checking in with an insider – this is particularly useful if observing within an unfamiliar culture or environment

asking another observer to compare notes – while not always possible, this is an interesting exercise that can allow you to asses your ability to ‘control for’ your own worldview

COLLECTING DATA WITHOUT INTERVENTION: UNOBTRUSIVE METHODS

 

‘The secret of my influence has always been that it remained secret.’

– Salvador Dali

I can tell you – it is no easy feat to find a quote related to unobtrusive methods. So I was pretty excited when I came across Dali’s words on the power of secret influences. You see, most forms of data collection, for example, interviewing, surveying, experimentation and most forms of observation, actually introduce foreign entities into real-world situations. And these entities, namely the researcher and the research process, cannot help but have some impact on the reality of the situations, events, or people being explored; researchers and researching have an influence on social environments. Yet the influences implicit in traditional research strategies often go unrecognized, unacknowledged and/or ignored. In other words, or in Dali’s words, they have ‘remained secret’.

The main advantage of unobtrusive methods is that researchers, and the research processes they adopt, are removed from those who provide the data. There is no disruption to the social environment. The influence of researchers is controlled for.

 

Unobtrusive methods: Methods of data collection in which researchers and research processes are removed from the researched; direct interaction is avoided. As well as some forms of observation, unobtrusive methods include the exploration and review of pre-existing government data and records, corporate data, personal records, the media, the arts, as well as social artefacts.

Unobtrusive research pros and cons

We’ve pretty much covered the main advantage of unobtrusive methods; that is, they are ‘non-reactive’. The potential for research participants to be influenced by the research process or the presence of the researcher is taken out of play. But there are other advantages as well. Unobtrusive research methods:

explore evidence of what people have actually done

allow you to acquire real-world data from the real world

allow you to capitalize on the vast amount of data already out there

allow for nonverbal as well as verbal data collection

can provide both rich, in-depth qualitative data and standardized, quantifiable data

can be time- and cost-efficient

can eliminate the need for physical access to research subjects

can minimize stress for both researchers and research subjects

can eliminate worries related to: building trust; getting people to act naturally; role playing; and figuring out how attributes such as race, gender, ethnicity, class and age of researcher and researched might confound data collection

allow you to be neutral and not a force for change

Well, that’s a pretty long list, and hopefully the discussion so far has piqued your interest in exploring what I think are highly underutilized methods. But there are some challenges associated in working with ‘pre-existing’ data. In conducting unobtrusive methods, you will need to:

work through data not expressly generated to answer your particular research question(s)

overcome any shortcomings that might exist in the original records you are exploring, for example, bias, inaccuracies, incompleteness

make sure your own biases do not colour your interpretations and understandings

avoid taking records out of context

protect the needs of an uninformed researched, that is, protection of privacy, anonymity and/or confidentiality

overcome the expectation that ‘real’ research demands interviews and surveys

Unobtrusive research options

There is so much data already out there: records abound; research is done on a daily basis all around the globe; and the day-to-day evidence (the social artefacts) of what people think, do, believe and feel surrounds us in art, poetry, music and even our rubbish. And I, for one, don’t think we do enough with this ‘data’.

Now before primary data collection begins, most lecturers (including me) expect their students to have at least started a literature review. We do this because we know that to be in a position to contribute to a body of knowledge you need to be conversant with it. But before my students go down the interview/survey path, I also ask them to look for data sources that might be out there already.

Sure, some students come back and say there isn’t very much, or what’s out there isn’t very useful – and these students continue on with their original methodological plan. But others get quite excited by the possibility of ‘data mining’ or working with different forms of ‘non-elicited’ data, and they go on to design fully unobtrusive methods. Others still go on to use some form of unobtrusive data to triangulate their main interview or survey data – adding much to the credibility of their study.

The range of unobtrusive data types available for exploration is almost endless. Table 6.4 attempts to capture the possibilities.

Gathering relevant data by unobtrusive means

Ask a question, get an answer. That’s the advantage of surveys and interviews. The answers, whether tainted by the process or not, are right there on the table for you. You get to direct the process and collect only the data relevant to your research question. Not so with data collected by unobtrusive methods. With unobtrusive methods, you are working with data not expressly generated for your particular research purposes. So this means you will really need to: (1) know what you are looking for; (2) know where it is likely to be found; (3) know whether or not you can trust it; and (4) have some sense of what you can do with it.

Knowing what you’re looking for

It might sound a bit strange, but clearly and succinctly articulating what you are after in terms of ‘data’ is something that can actually be overlooked when using unobtrusive methods. When you conduct a survey, the process of designing and developing your questionnaire demands you work through your data needs. The same is true of interviewing. Question development is actually an exercise in clarifying what you think your respondents can offer. But with unobtrusive methods, because you are working with documents, records, data sets and artefacts not produced expressly for your purposes, it’s easy to skip this articulation step and take a somewhat haphazard approach to data collection.

Unobtrusive methods demand the same consideration as any other data collection method. You need to assess your research question and/or hypothesis and ask yourself, ‘What data am I after, and how will it contribute to my understanding?’ You need to know what you are looking for – and think through it with the same rigour you would put into developing a questionnaire, interview schedule or observation checklist.

TABLE 6.4 UNOBTRUSIVE DATA TYPES

What type of data will you be exploring through your unobtrusive methods?

‘Official’ data and records – while you may have to work at getting access, it’s worth exploring:

 

International data held by organizations such as the United Nations, World Bank, or World Health Organization

National data held by many federal or national governments and government departments, e.g. national census data

Local government data such as State of Environment reports, community surveys, water quality data, land registry information, etc.

Non-governmental organization data collected through commissioned or self-conducted research studies

University data, which is abundant and covers just about every research problem ever studied

Archival data such as records of births, deaths, marriages etc.

Organizational communication, documents and records – generally official communication that includes, but is not limited to:

 

Press releases

Catalogues, pamphlets and brochures

Meeting agendas and minutes

Inter- and intra-office memos

Safety records

Sales figures

Human resource records

Client records (this might be students, patients, constituents, etc. dependent on organization type)

Personal communication, documents and records – personal and often private communication that includes, but is not limited to:

 

Letters and emails

Journals, diaries and memoirs

Sketches and drawings

Poetry and stories

Photographs and videos

Medical records

Educational records

Household records, e.g. chequebook stubs, bills, insurance documents etc.

The media/contemporary entertainment – data here is often examined in relation to questions of content or portrayal, e.g. the content of personal ads, how often male characters are shown crying, or how often sexual assault has made the national news over the past two years. Data can come from:

 

Newspaper or magazine columns/articles/advertisements

News programmes and current affairs shows

TV dramas, sitcoms, and reality shows

Commercials

Music videos

Biographies and autobiographies

What type of data will you be exploring through your unobtrusive methods?

The arts – the arts have captured and recorded the human spirit and condition over the ages in every corner of the globe, making it perfect for comparing across both culture and time. Societal attitudes are well captured in:

 

Paintings, drawings and sketches

Photography

Music

Plays and films

Social artefacts – this can be quite fun and owes much to the seminal work of Webb et al. (1966), who identified a number of social ‘traces’, including those related to:

 

Erosion, which looks at wear and tear in relation to preference. For example, looking at how worn seats are to determine where people most prefer to sit on the train, or determining patient reading preferences by examining the condition of waiting room magazines

Accretion, which aims to understand society by examining what people leave behind. For example, looking through waste disposal bins in a hospital to see if staff are conforming with waste disposal policy, or examining attitudes to promiscuity by looking at toilet door graffiti

Knowing where data sources can be found

Once you know what you are looking for, you will need to know where you can find it and how you can get your hands on it. Now, as you saw in Table 6.4, unobtrusive methods are capable of collecting a huge array of data types and each type of data will demand quite distinct protocols for collection. For example, your data might be held by an organization, by an individual, or by a family. It might be on the television, at the movies, at a school, museum or park. It may be in the public domain, or it might be private. It may be held by other researchers, local government, national government or international agencies. And getting your hands on it may involve writing away for it, going to the library, making a personal appeal, or going into the field.

Given this diversity, the key to success is being prepared. You will need to know well in advance where your data sources are located; who the gatekeepers might be; how to best approach them; whether or not you will need to use a sampling strategy; and whether the collection of sensitive or private data will require ethics approval.

Assessing credibility

Because you’re working with existing data, assessing credibility is essential. Ask yourself if the pre-existing data you are working with is unbiased, complete and accurate. Does it give a full account? You also need to be able to recognize whether your data was produced with a particular agenda in mind (for example, political campaign paraphernalia, promotional materials, or even surveys that have been produced by those with a vested interest). It may be tempting to treat the printed word as truth, and to treat artefacts as conclusive evidence, but all data needs to be viewed with a critical pinch of salt.

Credibility will also rest on how well you are able to manage your own subjectivities. How you ‘read’ and make sense of your data will be coloured by your own researcher reality. You need to ensure that your biases do not colour your interpretations and understandings, and that your data is interpreted within its original context.

Strategies for ensuring credibility are similar to those used to authenticate observations and include: well-designed and reviewed methods; broad representation that explores multiple sources of data; crystallization and saturation that sees a full ‘story’ come together; triangulation of unobtrusive data with other data sources; and, if possible, checking in and comparing notes with an insider or other researcher.

Working with your data

When using unobtrusive methods, the data as you find it may not be ‘ready for use’. Your sources, whether they be quantitative data sets or a series of qualitative documents, will need to be explored in order for you to draw out relevant information. Now the first step here is to ask yourself questions about the ‘data’. Who produced it? Who did they produce it for? What were the circumstances of production? When, where and why was it produced? What type of data is it? Basically, you want to explore any background information that is available (sometimes called the latent content or unwitting evidence).

The next step involves exploration of the meaning of the ‘data’ itself (the manifest content or witting evidence). There are a few ways you can do this. The first is highly applicable to qualitative data and involves using an interview technique to ‘ask’ your data a series of questions. As with an interview, you will need to develop and ‘ask’ each question, and then find the answer within your documents.

The second method, also applicable for qualitative data, involves content analysis or the noting of occurrences. This process quantifies the use of particular words, phrases, images and/or concepts within a given situation, text or document. The researcher determines what is being ‘looked for’ and notes the frequency and perhaps the context of the occurrence (this is covered more fully in Chapter 11).

The third method is secondary analysis and is applicable to quantitative data. This method relies on statistical analysis (see Chapter 11) of existing data sets either on their own or in a combined form, for example, linking census data and crime statistics. Keep in mind that there is a plethora of existing quantitative data available at local, national and international scales, just sitting on the answers to many of our research questions.

MANIPULATING THE ENVIRONMENT: EXPERIMENTATION

 

‘It is inexcusable for scientists to torture animals; let them make their experiments on journalists and politicians!’

– Henrik Ibsen

 

Experimentation: Searching for cause and effect by varying an independent variable (something you believe is a key determinant in your study) in order to see if it has an impact on your dependent variable (the main object of your inquiry). In other words, you manipulate X to see if it has an effect on Y.

You might not have been able to define ‘experiment’ before reading the above, but it’s probably a term you were at least familiar with. After all, it’s the mainstay of medical researchers, crime scene investigators and mad scientists alike – and would be a method of choice if your goals included: evaluating the effects of pharmaceutical drugs; looking at the connection between suspect heights and bullet trajectories, or creating the perfect human–monster hybrid.

Now in terms of practical real-world experiments, you are unlikely to be working with cells, DNA, inanimate objects or laboratory animals. The likely object of your inquiry will be people in all their complexity. You will probably explore what people do, why they do it and how the things they do affect aspects of the real world. It’s also unlikely that your experiments will take place in the controlled confines of a laboratory. Your research is likely to take place in real-world settings, with all the advantages and shortcomings thereof.

Nevertheless, experimentation offers tremendous potential when researching real-world problems. For example, say the problem you were interested in was bored students who had difficulty engaging in learning. Real-world experimental design can allow you to manipulate classroom layout (the independent variable) to see how it affects student attentiveness (the dependent variable).

Or say the problem within a particular company is the high number of sick days taken. Your experimental design might involve introducing a workplace exercise programme (independent variable) to see if the number of sick days taken decreases (dependent variable). Finally, if the problem was high levels of domestic waste in a particular county or municipality, you might ‘experiment’ by introducing free household recycle bins (independent variable) to see if this leads to a reduction in household waste levels (dependent variable).

Experiment pros and cons

There is something very appealing about saying, ‘I wonder what would happen if I were to …’, and then be able to set up and assess the effects of that exact scenario. You would get to see it unfold for yourself. You would get to manipulate the environment and both witness and record the results. You would be in control. And these are just a few advantages of conducting experiments.

Experiments also allow you to:

assess cause and effect

compare groups

explore real actions and reactions; and if so designed, in real contexts

avoid reliance on respondent’s memory or reactions to hypothetical situations

generate both in-depth qualitative data and standardized, quantifiable data

generate nonverbal as well as verbal data

Sounds pretty good. But as you might already suspect, there’s no guarantee of smooth sailing. Unless you’re ‘Big Brother’ from the ‘Big Brother’ house, it’s hard to control it all. When experimenting on real-world problems, in real-world contexts, you will need to consider whether:

there is equity in your design (for example, will the manipulation of your independent variable advantage or disadvantage any individuals/or groups?)

your design will allow you to get informed consent from participants

participants will stay involved for the duration of the experiment

you can control for your own biases

your design can control for extraneous, confounding or intervening variables (the things that effect your study that are not a part of your methodological design)

Experiment options

There are three basic criteria for ‘pure’ or ‘true’ experimental design:

1. that independent variables are manipulated by the researcher

2. that control groups are used, and there is random assignment to both control and target groups

3. that experiments are conducted under controlled circumstances

FIGURE 6.2 FOCUSING IN ON HUMAN-CENTRED REAL-WORLD EXPERIMENTAL DESIGN

As you can imagine, these criteria, are easiest met in traditional laboratory-based research. But it’s a different story when you are dealing with real-world problems. Okay, at this stage you might be thinking, ‘Well actually, I’m pretty sure I am heading down the path of a more traditional lab-based experiment.’ And yes, there are plenty of real-world problems where ‘lab-based’ experimentation on ‘nonhuman’ objects of inquiry can be central to research design. For example, if your interest is in skin cancer, you may want to manipulate levels of UV light to see how they affect malignancy in human cells. Or if you are interested in the benefits and dangers of fad diets, you might start your investigation by seeing how changes in levels of dietary protein affect kidney functioning in laboratory rats.

Now while the basic elements of these types of experiments mirror those of human-centred real-world experiments, they tend to require more technical expertise and advice than is available within the scope of this text. So while I do offer a few readings at the end of this chapter that can help get you started on this path, the main focus of this chapter (as highlighted in Figure 6.2) is human-centred, real-world experimental design.

The search for cause and effect

On the radio the other day I heard that ‘eating fish increases your IQ’. The story reported on the latest research that found that children who eat fish at least once a week have higher IQs than their non-fish-eating peers. Hence the ‘eat fish and get smart’ headlines that led the story. But as I listened, I found out that the study looked at children’s IQs and compared them to a number of factors including diet, education, socio-economic status, age of the parents, parental marital status, parental education level, etc. etc. And one of the correlations they found was between eating fish and IQ. As fish consumption rose, so too did intelligence – but then again you could also say that as IQ rose so did fish consumption. Does eating fish make you smart or do smart people eat more fish? Correlation is simply not cause and effect.

An intervening or confounding variable might also come into play. Maybe it’s not the fish that makes you smart – maybe what’s going on is that smart parents feed their children fish, and their child’s IQ is determined by genetics.

Okay, let’s say you really want to get to the bottom of the great fish debate and you decide you want to determine cause and effect by conducting an experiment (after all, you’ve read that experiments really are the best way to work through this type of research problem). So how do you go about it?

Well, initial planning will involve lots of decision making. As you work through your methodological design you will need to decide on:

 

Your dependent and independent variables – you will need to identify the main focus of your study or what you are trying to assess (the dependent variable), as well as the variable you will manipulate in order to cause an effect (the independent variable). In this case, you are hypothesizing that IQ depends on fish consumption, thereby making IQ the dependent variable and fish consumption the independent variable that you will manipulate. This identification of variables by type is central to moving from correlation to cause and effect.

Assessment of change – in order to determine whether the manipulation of your independent variable has affected your dependent variable, you will need to be able to assess change. The most effective way to do this is by both pre- and post-testing, which in real-world contexts may mean collecting or having access to good baseline data and being able to collect comparable data after the experimental intervention. In our case, assessing change is relatively straightforward and would involve administering standardized IQ tests.

Research setting – consider whether you will be conducting your study in a controlled environment such as a laboratory or if you will use a natural setting. In this case a lab may give you total control, but as is the case for many real-world problems, it may not be practicable. Other options in our scenario are to ask parents to vary diets at home, or to make arrangements with maybe a day care centre to change their weekly menu.

Number of participants – the number of participants you will use is also crucial. Think about how many participants will be necessary for you to make any conclusive or statistically significant judgements. For example, if you find a pattern in five children, is it enough? (Chapter 5 covers the basics of determining sample size.)

Number of groups – you will also have to decide if you will use a control group. In our fish example, using a single group would involve testing the IQ of all the children, feeding all of them fish a set number of times a week, and testing them at some period thereafter to see what happens. With a control group you would test all of the children at the start, put half the children in a control group and the other half in a target group, and only give fish to the children in the target group. You would then test both groups again at a later date and compare findings.

Assignment strategy – if you are using a control group you will need to determine how you will assign your groups. Will children be randomly selected for fish consumption or will you use different criteria for selection? While randomization will provide you with stronger cause and effect arguments, you might find it more practical to select children based, for example, on the days of the week they’re in a day care centre.

Number of variables – will you test just one independent variable or will you test for others as well? For example, will you simply look at fish consumption or are there other aspects of the children’s diet you will explore, such as vegetable intake.

Ethics – consider whether you will need informed consent. In our fish consumption case, you will need parental consent. You will also need to consider if there are any advantages or potential threats to group members based on their inclusion in either a control or a target group. Now while there may not be high risks associated in eating or not eating fish, issues of equity represent a huge ethical dilemma in drug trials, treatment programmes and educational initiatives.

Controlling the environment – finally, you will need to consider how you will negotiate the balance between the practicalities of working in real-world situations and the need to control the environment. In other words, you need to consider how you can ensure your findings can be attributed to a true cause and effect relationship between your independent and dependent variables. Now the more controls you embed into your experimental design the more convincing your arguments will be. But without such controls, arguments can be spurious. For example:

without a controlled environment it can be hard to ensure that the only variable that has been changed, shifted, manipulated, or introduced is your particular independent variable, for example, other dietary changes, changes in sleep patterns, personal stress, etc. may be happening outside your experimental design

without adequate numbers it will be hard to show statistical significance or that results are more than coincidence

without a control group it’s hard to ensure that there is not some other factor that might account for changes in your target or dependent variable, for example, that improvements in IQ scores cannot be attributed to things like additional attention that the children might be receiving, practice in taking IQ tests, or the coincidental commencement of a new educational programme

without a random assignment strategy (which is often impractical in real-world research) you will need to argue that differences between the two groups are non-existent or at least minimal. In our case, if there is an innate difference in the learning abilities of the two groups, it would be impossible to attribute increased IQ to dietary habits.

In short, the bottom line in human-centred, real-world experiments is tradeoffs. There are always tradeoffs. The benefits of real-world contexts need to be weighed up against the benefits of a controlled environment. I think the best advice I can give to those who want to go on and conduct experiments is to read. Not only do you need to be familiar with the basic issues presented here, you also need to be familiar with the plethora of design options open to you. The readings offered at the end of this chapter should help get you started down that path.

COMPARING METHODS OF DATA COLLECTION

I thought I’d end this fairly long chapter by offering a few quotes related to the challenge of data collection (Box 6.4) and a summary table that compares the data collection methods we’ve discussed (Table 6.5). The table looks at how each method deals with or treats: populations, data, relationship with respondents, perspectives and ethics. While you need to build an understanding of each of these methods before you finalize, a quick design – a side-by-side comparison can be really useful.

Box 6.4 Collecting Data – Dealing with Facts

 

The importance of gathering facts and data …

 

‘Facts are the air of scientists. Without them you can never fly.’

Linus Pauling

But on the other hand …

‘Facts are stupid things.’

Ronald Reagan

‘Data is not information, information is not knowledge, knowledge is not understanding, understanding is not wisdom.’

Cliff Stoll and Gary Schubert

Perhaps even more important …

 ‘You can use all the data you can get, but you still have to distrust it and use your own intelligence and judgement.’

Alvin Toffler

‘Don’t become a mere recorder of facts, but try to penetrate the mystery of their origin.’

Ivan Pavlov

‘Science is facts; just as houses are made of stones, so is science made of facts; but a pile of stones is not a house and a collection of facts is not necessarily science.’

Henri Poincare