Explained by Allison Tracy, Ph.D., WCW Methodologist and Research Scientist
Research & Action Report Spring/Summer 2006
The Wellesley Centers for Women (WCW) has been a leader in the study of issues of importance to women for more than 30 years. What makes our work somewhat different from many other “think tanks” around the country is that the Centers’ staff pursue research about which we are unabashedly passionate, with the goal of making a positive difference in the world. This vigor which we bring to our work may appear to some, particularly if the findings challenge their views, to be at the expense of scholarly rigor. This couldn’t be farther from the truth.
To start, each research project must be successfully funded. To be funded, a project must have a compelling topic, relevant to social issues identified as priority areas by the granting agencies. In other words, a funded project must demonstrate vigor. Still, however compelling the topic, the methods (e.g., sample selection, data collection protocol, and analysis of the data) to be used to address the research questions mustbe unquestionably rigorous. Indeed, the scientific rigor of proposed methods must pass the critical review of scholars and/or practitioners noted for their expertise in the field of interest.
Once a project has been funded, the methods faithfully executed, and the research questions analyzed, the moral mandate is still not satisfied until the findings have been delivered to policy makers and practitioners in order to benefit those for whom the data and analyses speak. The delivery of these messages often begins with publication in scholarly journals and withpresentations at academic conferences. These venues act as gatekeepers for the research community. Again, summaries of projects—from the justification of the importance of the topic, to a detailed description of the research methods and analysis, to a full discussion of the implications, limitations, and merits of the study results—are subjected to critical review by a panel of respected scholars.
WCW’s scholarly research bears the badge of commendation from the scientific community. But what exactly constitutes this scientific rigor? What are the methods used at WCW to conduct rigorous research? The answers to these questions are as varied as the many research projects undertaken. Every research question contains elements that suggest certain methodological tools and techniques. Most research methods and analysis can be classified as either quantitative in nature (summarized by numbers and subjected to statistical assumptions from which meaning is inferred), qualitative (consisting of non-numerical data such as language, interactions, or observations, from which discernable patterns can be extracted), or mixed (containing elements of qualitative and quantitative methods and/or analysis). Often, a research project will contain both qualitative and quantitative elements but will emphasize one more strongly than another.
Quantitatively based research often carries the most impact in influencing federal policy decisions (e.g., the requirements of statistical evidence underlying the No Child Left Behind Act) and is regarded as the “coin of the realm” in the biological and physical sciences, leading to the common assumption that statistical evidence is required for rigorous research. Indeed, there are many obvious strengths of mathematical modeling of social processes, including the opportunity to analyze the experiences of many more individuals, providing clear standards for replicability and validation, and offering standardized techniques and evaluative criteria.
For all its strengths, quantitative approaches do not represent the “silver bullet.” For instance, statistical analysis has historically meant summarizing research results with a single value such as a correlation, an odds ratio, or a regression weight. But, human experience is dizzyingly diverse and subject to many influences, both within a person and extending to her/his social and physical context. A single answer or value simply cannot satisfactorily sum up many of the nuanced social issues studied at WCW.
In recent years, advances in computer technology and statistical modeling approaches have allowed researchers to more accurately capture essential distinctions in experience and explore the forces at play and the implications of the differences detected. One study recently completed at WCW drew from a large, nationally representative dataset called the National Longitudinal Study of Adolescent Health. This WCW study looked at the various ways that participation in sports in high school might influence youths’ sexual decisions and behaviors into young adulthood. For this project, the team used a sophisticated statistical modeling technique called structural equation modeling, or SEM, that allowed the researchers to examine multiple pathways of influence (e.g., self efficacy, gender role ideology, social support, etc.) and compare the relevance and prevalence of these pathways across boys and girls and across youth from various racial/ethnic backgrounds. This modeling technique revealed dramatic differences across gender and varied, and sometimes countervailing, differences across race and ethnicity (Project: Sports as Protective of Girls’ High Risk Sexual Behavior; Researchers: Sumru Erkut and Allison Tracy).
Other existing datasets used at WCW include the National Institute of Child Health and Human Development Study of Early Child Care and Youth Development, for which WCW hosts one of a number of study sites nationally (Researchers: Nancy Marshall and Wendy Wagner Robeson), and the Health and Retirement Survey (Project: Older Workers and Health; Researcher: Nancy Marshall). Large national surveys such as these lend themselves to the testing of statistical models that more nearly capture the complexities of the conceptual models theorists at WCW use.
At times, WCW research is directed toward understanding local issues such as in the Massachusetts Afterschool Research Study (Research Team: The National Institute on Outof- School Time). Occasionally the researchers at WCW identify a need for a strong quantitative measure for a construct identified by theorists in the Centers such as in the Wellesley Relational Model Instrument Development Study (Researchers: Belle Liang and Allison Tracy). Quite often, WCW researchers are called on to empirically evaluate intervention/prevention programs such as the Jacksonville Afterschool Experiences Evaluation Project and Rosie’s Girls Evaluation Project (Researcher: Fern Marx). These projects require the collection of new data dedicated to the specific purpose of the project and are well-suited to quantitative analysis models.
Some of the Centers’ research topics are clearly pioneering and, for such topics, there is little or no existing data on which to base analyses. In these cases, we have the opportunity to craft our own sampling strategy, to design protocol, and to include ways to measure exactly the right constructs for the research questions. While this adds time and effort to the research process, it also enhances the validity of the results since the methods are tailor-made for the research questions at hand.
One such study is the Exploratory Study of Same-Sex Marriage (Research Team: Same-sex Marriage Study Group), for which a qualitative methods and analysis approach are best suited. While the data collection protocol for this study included a short written questionnaire, emphasis was placed on in-depth interviews conducted with both partners of committed same sex couples and any dependent children able and willing to participate.
The validity, rigor, and relevance of qualitative research relies on researchers’ ability to retain the integrity of participants’ unique experiences, using their own ways of relating their stories— their language and phraseology, the sequencing and emphasis (or lack of emphasis), etc.—rather than on methods which require standardization of meaning. Scientific rigor in the case of qualitative research is demonstrated by the clarity of the research process and decisions made in response to preliminary findings and in the logic of the analytic process, the clarity and consistency of the patterns the analysis discovers, and the insight of the researcher(s) in interpreting the meaning of the patterns.
Other current WCW research projects applying various rigorous qualitative approaches include Perceptions of Work Environments and Relationships (Researcher: Anne Noonan), the African American Intimate Partner Violence Study (Researcher: Katherine Morrison), and the Massachusetts Cost and Quality Study (Researchers: Nancy Marshall, Wendy Wagner Robeson, and Joanne Roberts). An ambitious project recently funded, the Mixed Ancestry Project (Researchers: Sumru Erkut, Michelle Porche, Allison Tracy, et al), represents the pilot for a nation-wide study of racial and ethnic identity development. This study will use quantitatively based components to examine typical patterns of development over late-adolescence and qualitatively based components to discover the meaning-making processes youths employ in constructing racial/ethnic identity in a culturally complex social context.
Although quantitatively based research findings are more easily generalizable to a larger population, and are more easily replicated, quantitative designs do not provide the in-depth understandings of individual experiences achieved using qualitative methods. Well designed qualitative research provides researchers with the opportunity to consider individual experience as well as social and physical context. Increasingly, the richness of mixed or integrated method approaches is recognized and rewarded in the research world. WCW has both helped to build this understanding as well as benefited from it, as our work increasingly draws from the full range of strengths available in both quantitative and qualitative approaches.
