Research Methods Intro: Accuracy and Ethnography

In personality psychology researchers empirically investigate sources of accuracy by using information about both the perceiver – the person whose accuracy we are evaluating – and the target – the person the perceiver is accurate about. Using a round robin design – where every participant rates and is rated by every other participant, as well as peer and self-report ratings of each participant, researchers can examine and quantify the relative predictive power of different factors.

Participants rate themselves and others on different traits. Ratings are more accurate if the perceiver’s ratings of the target match an average of the target’s peer and self-report ratings. Researchers can, for example, simultaneously compare:Accuracy Illustration

  • normativity – the actual prevalence of the trait in a group


  • perceived similarity to the perceiver – the influence of distinctive traits about the perceiver (calculated by adjusting the average of the perceiver’s peer and self-reports for the average self-report for the entire group.)


  • distinctiveness  – the extent to which the target is higher or lower than average on the trait (calculated by adjusting the average of the target’s peer and self-reports for the average self-report for the entire group) (Human & Biesanz, 2011).


Theoretically, the influence of normative accuracy – the extent to which an individual references others against a norm – should be higher when perceivers and targets share a cultural background. On the one hand, normative accuracy is the product of experience. The more muembers of a group you meet, the better you estimate average behaviors. On the other hand, cultural norms also shape who we seek to become and how we express ourselves.

Perceived similarity, on the other hand, can bias the perceiver towards seeing her own distinctive traits in others, at least when she likes or in some way identifies with those others. For example, an ethnographer may tend to see informants that he likes as being more similar to him than they actual are and informants that he dislikes as either being contrasted against his perceptions of himself or more similar to his perception of the “average” informant.

When perceptions of normativity are less established, however, the target’s distinctiveness should be less biasing, given that the ethnographer may not know what traits are distinctive and what traits are common. In other words, as the ethnographer’s perception of the actual averages for the group of informants changes, the roles of similarity and distinctiveness may change as well.

One takeaway for the ethnographer, then, is to exercise greater caution and give attention to the influence of presumed normative behaviors, perceived similarity (or lack thereof), and target distinctiveness. However, where a round robin design is practical, the ethnographer could also apply this observational research to the field. Given a culturally-validated scale, the ethnographer could compare the respective roles of these different influences on person perception across cultures. Other analysis could compare normative accuracy as determined by the actual average ratings for the groups to stereotypic accuracy – as determined by participant ratings of an imaginary “average person.”

This quantified data could be used to contextualize participant observation and in depth interviews.

Other Considerations – What is Accuracy?:

Accuracy is multi-dimensional. For example, if asked to judge the prevalence of a certain trait in different social groups, a person could have poor absolute accuracy. In that case, they might consistently underestimate or overestimate the prevalence in each group. However, they might still have good relative accuracy – judging the differences between groups well.  As in the discussion above, accuracy is a continuous variable and it can increase or decrease over time. Our stereotyping intervention, for example, targeted absolute accuracy for a target social group. It could be expanded to target absolute accuracy for both the target’s social group and for the perceiver’s. Relative accuracy would then take care of itself.

Further, statistical measures of accuracy are blind to process. Other research examines how an observer learns about the group’s average rating on any trait. More research can disentangle the roles of shared social-desirability concerns, self-stereotyping, and other culturally-accessible influences on the self concept. These shared concerns could, for example, lead participants to report being more similar without actually being more similar.

Considering this relative complexity, stereotyping and prejudice interventions have to choose their target:

  • Improving the validity and reliability of the process by which we judge individual targets and target groups?
  • Improving the absolute accuracy of these judgments?
  • Improving the relative accuracy of these judgments?
  • Improving accuracy for certain traits, but not for others? (Accuracy may differ by trait).


Research Methods Intro: Participant Observation – A Brief Primer

In interacting with Maori individuals, the indigenous people of New Zealand, I found a few techniques particularly helpful. First, being an anthropologist requires a degree of “suspension of disbelief.” You are there to learn other peoples’ stories, stories that will sometimes clash with what you yourself believe. People will be sensitive to your disbelief, so focus on trying to see through the eyes of multiple community members, attending to them and your memories of them more than to your personal reactions.

Of course, sometimes, silence is awkward. When you must take a personal stance, try to make it ecumenical. For example, in a meeting of a smaller Maori health trust, we had just had morning prayer (Pai Marire) and were discussing religious orientations. I am agnostic. I mentioned that when I prayed, I prayed as a calling out, without specifying to what or whom I was calling out, or how often I did so. My response was tailored to show spiritual focus without identifying myself as having distinct, potentially troubling, beliefs. If I had been a firm atheist, as opposed to an agnostic, I could have emphasized that I believed in certain values – and listed a set of generically acceptable values.

Most of the time, however, you should be listening intently, not talking. Pay active attention to their facial expressions and gestures. Let yourself mimic these expressions, subtly. You should also be comparing what they’re saying to what they’ve said and what other people have said. You can draw out a more in depth response by looking really excited by an idea or asking a clarifying question. Affirm their emotions by your facial expressions or, more rarely, by offering a label (which they then might accept or reject.)

Make sure every conversation is about them. The primary logic of the ethnographic process is that subtle, iterative, queries and challenges combined with careful observation over a long period of time gleans insight we cannot find elsewhere. When doing fieldwork, you’re constantly seeking a group’s intersubjectivity; their overlapping impressions of a topic. You want to describe that intersubjectivity and understand how it arises.

You, of course, may contribute to this intersubjectivity. However, if you are approaching a community in order to advocate for change, be honest from the beginning. An anthropologist tells a full story from the perspective of multiple community members. She does not spy. Doing so hurts not only your reputation but every anthropologist’s – I was actually called a “spy” by one gentleman. I nodded in acknowledgment of his concern and then continued listening and asking questions. By the end of my time in New Zealand, I had won his trust, but it would have been more efficient if he had not been biased by the actions of one of my predecessors.

Research Methods Intro: Ethnography – How Do People Actually Talk, Think, and Behave With Regards To Race?

If we have the resources, it is helpful to start with a qualitative, particularly an ethnographic, perspective. This perspective helps us to generate a “thick description” of the phenomenon in question. This process provides inspiration for quantitative work and helps us to interpret quantitative results.

Relevant to our intervention-oriented research question – we can identify areas of contradiction and areas of lack awareness in the way that our participants think about race, which can then inform interventions. We can build upon their existing wisdom as well. We can then frame our interventions in a way that is accessible to participants – that shows an understanding of their perspectives.

Showing understanding can allow us to be supportive and affirming even as we challenge them in ways that could produce a general sense of threat. Our intervention depends on challenging, not threatening, our participants.

We could, for example, select two communities, one racially diverse and the other majority white. Then we could conduct a participant observation study – meeting with community members (white and non-white) and spending time with them formally and informally.

Formal contact might be in the form open-ended interviews in which we ask community members about racial attitudes, attitudes towards prejudice, interracial interaction, and discrimination. For my own experience as anthropologist, click here. We could also sit in on meetings in which community members are discussing related issues, including diversity but also including economic or political issues that they may see as relevant to race. Using both individual interviews and a record of public utterances, formal study can juxtapose public and private expression.

Informal contact may be more rare and will depend upon the rapport that you have built in formal interviews. In “hanging out” with community members, you may encounter a very different, more spontaneous, public and private expression. However, spontaneity does not mean that the expression of the attitude is “pure.” By talking with people and asking questions you inevitably influence what they later express and how they express it, at least to you. You’ve made ideas and the expression of ideas salient that may have only been inchoate before you began your research. Last, no matter how much rapport your develop, some ideas will not be expressed.

To get at those ideas, we can employ structured, interview, surveys, implicit attitudes tests, and behavioral experiments.

Research Methods Intro: Categories of Research Question – Example: Teaching Cutting Edge Thinking About Race

Every research project starts with a research question:

  • Observational: I believe that the world works in a certain way. I want to give evidence in favor of (or against) my observation.
  • Theoretical: This theory predicts that the world will work in this way but it hasn’t yet been tested in this particular context. Let’s do it!
  • Inferential: If the world works in one way, it probably works in a logically related way as well. Let’s see!
  • Incremental: The world has been shown to work in this way by numerous studies – let’s confirm the results of these studies and see if we can flesh out the story a bit.
  • Exploratory: Let’s see how the world works.
  • Practical: Can we get the world to work this way?

Let’s take an example. One of my specialties is the study of stereotyping and prejudice. When I hear people talking about stereotyping-related topics in the media and amongst themselves, I often find myself thinking, “Wow! You’re so busy being half right that you’re having this discussion all wrong!” That thought may be pretentious – but let’s go with it. Further, let’s pretend that I have a practical goal – to get a variety of audiences to embrace what I consider to be cutting-edge thinking and practice with regards to racial stereotypes.

So, I have a practical question which I am answering from an interdisciplinary perspective – is it true that I can get an audience to embrace this cutting-edge thinking?

Before we leap into speculating about potential measures of success or failure and potential tools for reaching our goals, let’s take a step back and consider relevant variables.

Research Methods: Purposes

I tackle socially-relevant questions and compare different methodological approaches to answering them.

My specialities are social psychology (M.A. University of Chicago, M.A. University of British Columbia), communication (Ph.D. The Ohio Stat University), and cultural anthropology (BA with honors Darmouth College). The first three required specialization in quantitative research, emphasizing rigorous statistical training and the use of subtle experimentation to identify key variables for understanding human behavior. You may have heard of a “crisis” in the social sciences. I prefer to think of it as growing: As social scientists, we have to be honest about the limitations of our tools and strive to overcome these limitations.

Dealing with the complexity of human behavior is inherently difficult. We are unable to have the same level of certainty that a physicist or chemist may have (and even drug companies have had great trouble replicating key, published and oft-cited, findings in their field). In order to detect a reliable pattern that is consistent across situations, the best human-behavior studies would have thousands of people doing hundreds of tasks, a practical difficulty that is only occasionally surmountable.

All is not lost, however. Even a smaller study can highlight an important relationship between different factors. At that point, it’s the field’s duty to conduct more research, replicate or fail to replicate the result, and to try to understand whether the initial result was due to chance or due to a third variable that moderates (that influences the strength of) the effect.

Looking from an interdisciplinary perspective, however, we can be inspired by laboratory-studied relationships and look for independent evidence of their relevance to “the real world.” The laboratory allows us to take a micro-view, to get at what people can’t or won’t tell us about themselves. We can then take these findings to the field and look for evidence for or against the laboratory results.

As a cultural anthropologist working with the Maori, New Zealand’s indigenous people, I was keenly aware of the fact that my cultural narratives, as well as the narratives the Maori used to describe themselves, drove the questions that I asked. I was able to ask questions that pitted these potential interpretations against one another and to record the response of individual Maori informants. Were they skeptical of the narrative? Did it make sense to them? What was their emotional response? Were some members of the community more open to my account than others? Were they open to the account, but did it strike them as novel? These are all questions an anthropologist asks.

As a communication scientists, I walk a middle ground between the controlled laboratory experiments of social psychology and the qualitative messiness of real world behavior. My collaborators and I manipulate websites carefully and observe average differences in behavior (e.g. selection of messages and time spent reading them). This step has both internal and external validity. However, we then use those differences in behavior to model observed changes in attitudes. We cannot say for certain that the behavior preceded the change in attitudes. Instead, attitudes may have shifted behavior. In our attempt to consider the wider implications of ecologically valid behavior, we lose some control.

Truly useful social science research is a balancing act.

For better or for worse, shaping the world in innovative ways requires an attention to both process and product, the navigation of multiple approaches, and a willingness to investigate and to challenge our most basic assumptions in a systematic way.

In the end, we all do the pragmatic thing. We all choose a side, take a stance, and act. However, we can do so humbly, aware of complexities, without blinders. We can do so boldly, honestly, and in a way that convinces others that we have selected the best response given what we know at the time.