Home > Scientific Areas > Methodology > Mixed Methods Research
Developing an R series plan that incorporated mixed methods research
Best Practices for Mixed Methods Research in the Health Sciences
TABLE OF CONTENTS
A primary mechanism for obtaining NIH grants to conduct mixed methods research is through research grants (R series) offered by the NIH Institutes and Centers. These grants include R01 research projects, R03 small grants, and R21 exploratory/developmental research grants. As researchers plan their mixed methods research projects and develop their R series applications, they need to follow the NIH instructions as well as incorporate key elements of mixed methods research.This section is organized in the following topics and offers suggestions for how to bring these two sets of criteria together:
General comments about preparing mixed methods research applications
- Follow the basic instructions for NIH R series applications.
Guidelines for the process of developing and submitting research applications to NIH can be found on the NIH website. R series applications need to follow the PHS SF424 standard forms as described in the "PHS SF424 (R&R) Application and Electronic Submission Information". The PHS SF424 instructions discuss requirements for the major sections of a research plan, including the study aims, the significance, innovation, and approach sections of the research strategy, and several additional categories of supporting information. In addition, there are several online resources to assist investigators with preparing research applications to NIH.
- Match the narrative page length to the appropriate funding mechanism. The PHS SF424 forms require that the project investigators include a 1-page statement of specific aims. In addition, the length of the research strategy narrative is limited to 12 pages for R01 applications and to 6 pages for R03 and R21 applications. These limits require investigators to carefully think through and articulate their mixed methods approach so that the application provides both the essential details and is concise.
- Embed mixed methods throughout the application elements. The use of a mixed methods approach has implications for each part of the development of the research plan. Investigators need to have basic knowledge about mixed methods research and embed important mixed methods components into the aims and research strategy of the application. These components target the application as a mixed methods investigation and relate to the quality of the overall mixed methods approach.
- Start the development process early to prepare a high quality mixed methods application. Preparing a successful mixed methods research plan within the narrative page limits is a challenge. It is important to think through the mixed methods components of a research plan in advance and to view the mixed methods approach as an integral aspect of the project as opposed to a minor add-on feature. Investigators need to allow sufficient time to develop and refine the overall mixed methods plan into a coherent and logical application.
- Reflect on the Research Project Evaluation Criteria while developing the application. The PHS SF424 instructions include "Research Project Evaluation Criteria" for evaluating the overall impact of a project as well as the quality of a project's significance, investigator(s), innovation, approach, and environment. These criteria are considered in the context of a mixed methods application in the Reviewing Mixed Methods Applications section of this document. Investigators should keep these criteria in mind to evaluate their own research plans and applications during the development and writing process.
back to top
The research plan
The PHS SF424 (R&R) instructions
call for several key sections within the overall research plan. Some sections (e.g., Introduction, Enrollment Report, and Progress Report) apply only to applications that are renewals or revisions, but all applications need to include a section on Specific Aims and sections on Significance, Innovation, and Approach within the Research Strategy narrative. A key to a successful mixed methods application is demonstrating that there is congruence among all the elements of the plan, including the study aims, the overall design, the specific procedures, the investigation team members, the available resources, and the budget parameters.
Here we consider how mixed methods research can be embedded within the following application elements:
The PHS SF424 instructions for Specific Aims state:
- State concisely the goals of the proposed research and summarize the expected outcome(s), including the impact that the results of the proposed research will exert on the research field(s) involved.
- List succinctly the specific objectives of the research proposed, e.g., to test a stated hypothesis, create a novel design, solve a specific problem, challenge an existing paradigm or clinical practice, address a critical barrier to progress in the field, or develop new technology.
Writing specific aims for a mixed methods application
- State an overall goal that addresses the overarching research problem or question.
Provide a statement of the overall goal (or purpose or intent) of the application. The types of goals that lend themselves to mixed methods inquiry are those that address multiple levels of influence, such as theory development and testing, process and outcomes, and context and meanings. Ensure that the goal informs the specific aims and research questions, leading naturally to a mixed methods approach.
- State study aims that identify discrete components of the goal to be achieved in the investigation.
Aims should be driven by the substantive research question, not by the methodology. The methodology should flow naturally from the aims and question.
- For each aim, identify the methodological approach to be used to accomplish it (which may be quantitative, qualitative, or mixed methods) to tie together the goals of the study with the methods.
- Aims that call for a qualitative approach may be inductive in nature; emphasize exploration; contextualize individual behavior, group behavior, organizational dynamics, and cultural influences; and convey the researchers' openness for learning from participants and data sources.
- Aims that call for a quantitative approach may be deductive in nature; emphasize measurable constructs; test theorized associations; and demonstrate probable causality or generalizability.
- Aims that call for a mixed methods approach require the integration of qualitative and quantitative data and results to yield multi-dimensional, synergistic understandings of the phenomena of interest; the use of qualitative methods to explain and elaborate quantitative findings; or the use of quantitative methods to generalize, test, or confirm qualitative findings.
back to top
- Order the aims so that they are congruent with the overall approach as discussed in the Research Strategy section of the application.
- Aims that will be addressed concurrently: If two or more aims will be addressed concurrently within the investigation, then consider the relative importance of the aims for addressing the overall goal. List the study aims in the order of their priority. Include at least one aim that explicitly calls for the integration of qualitative and quantitative data and results. For example, investigators studying the efficacy and process of a treatment may choose to state three aims. First, they state a primary aim related to assessing treatment outcomes that will be addressed with an experimental approach. Second, they state a secondary aim to describe the process that will be addressed with a qualitative approach embedded within the experiment. Third, they state an aim that specifically calls for understanding the treatment outcomes within the context of the treatment process, which requires an integrated, mixed methods approach.
- Aims that will be addressed sequentially: If two or more aims will be addressed sequentially within the investigation, then list the aims in the chronological order in which they will be addressed. This is particularly important when subsequent aims depend on the results of initial aims. If stating aims that separately lead to the use of qualitative and quantitative approaches, then provide an aim that explicitly calls for the investigators to connect from the results of an initial phase to the design and implementation of a subsequent phase. For example, investigators wanting to develop and test a culturally sensitive instrument may choose to state three aims. First, they state an aim to understand the cultural context of the phenomenon of interest that will be addressed with a qualitative approach. Second, they state an aim to develop the instrument based on the initial qualitative results that will be addressed with a connected mixed methods approach. Third, they state an aim to assess the construct of interest in the population that will be addressed with a quantitative approach utilizing the new instrument.
Research Strategy: Significance
The PHS SF424 instructions for the Significance section of the Research Strategy state:
- Explain the importance of the problem or critical barrier to progress in the field that the proposed project addresses.
- Explain how the proposed project will improve scientific knowledge, technical capability, and/or clinical practice in one or more broad fields.
- Describe how the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field will be changed if the proposed aims are achieved.
Writing the significance section for a mixed methods application
- Review literature on the topic that includes prior quantitative, qualitative, and mixed methods research. Consider adding critiques related to the methods that have and have not been used to study the topic.
- Identify a gap in the extant knowledge that suggests the need for gathering both quantitative and qualitative data to address the research problem.
- Identify the rationale for using mixed methods research, that is, the reasons for collecting both quantitative and qualitative data, in order to establish the importance of the problem and how the investigation will enhance scientific knowledge.
Research Strategy: Innovation
The PHS SF424 instructions for the Innovation section of the Research Strategy state:
- Explain how the application challenges and seeks to shift current research or clinical practice paradigms.
- Describe any novel theoretical concepts, approaches or methodologies, instrumentation or interventions to be developed or used, and any advantage over existing methodologies, instrumentation, or interventions.
- Explain any refinements, improvements, or new applications of theoretical concepts, approaches or methodologies, instrumentation, or interventions.
back to top
Writing the innovation section for a mixed methods application
- Provide arguments suggesting in what ways knowledge gained from the combination of quantitative and qualitative approaches will bring needed insights and improved practices that go beyond what could be learned from using one method alone.
- The use of mixed methods research may be an innovation in and of itself. If this is the case, investigators need to explain why the planned use of mixed methods can be considered an innovation. Examples of the use of mixed methods adding an innovative component to the investigation might be when the investigation includes novel approaches for integrating the quantitative and qualitative components of the project or when no prior research in the area of interest has used a mixed methods approach. Identify and highlight any innovative mixed methods approaches that are planned.
- Highlight any new tools or products that will be part of the mixed methods research approach, such as a new instrument or intervention that is developed and quantitatively tested based on qualitative findings.
Research Strategy: Approach
The PHS SF424 instructions for the Approach section of the Research Strategy state:
- Describe the overall strategy, methodology, and analyses to be used to accomplish the specific aims of the project. Unless addressed separately in Item 15 (Resource Sharing Plan), include how the data will be collected, analyzed, and interpreted as well as any resource sharing plans as appropriate.
- Discuss potential problems, alternative strategies, and benchmarks for success anticipated to achieve the aims.
- If the project is in the early stages of development, describe any strategy to establish feasibility, and address the management of any high-risk aspects of the proposed work.
- Preliminary Studies for New Applications: For new applications, include information on Preliminary Studies. Discuss the PD/PI's preliminary studies, data, and or experience pertinent to this application.
Writing the approach section for a mixed methods application
- Introduce mixed methods research and the specific mixed methods design. Since many reviewers may not be experienced with mixed methods research, investigators need to provide important background information about their choice to use mixed methods. To do this, investigators should:
- Identify the use of mixed methods research and provide a definition of this approach. Cite studies that used a mixed methods approach from the health/disease area of interest (if available). Studies that used a mixed methods approach can often be located in databases using search terms such as "mixed methods" or "quantitative AND qualitative" in combination with terms representing the content area of interest.
- Name the specific mixed methods design being used and cite studies illustrating its use. Locating examples can be a challenge, but searching databases using the relevant mixed methods design name [Consider examples of specific designs] may identify several examples. A good strategy is to search NIH's RePORTER database to locate successful applications that used a specific approach.
- State the reason(s) or rationale for the use of mixed methods and the selected design. Consider the variety of possible reasons for using mixed methods [Typical reasons for using mixed methods], such as seeking a more comprehensive account of a phenomenon, examining structure and process, or generating and testing hypotheses. The stated reason(s) should match the overall problem and research question and be congruent with the selected mixed methods design.
- Provide a diagram of the overall quantitative and qualitative procedures to assist reviewers in following the sequence of activities within the design. Since mixed methods approaches are necessarily complex, such a diagram will serve as an overview of the implementation of the quantitative and qualitative components of the study plan. Ivankova et al. (2006) provide guidelines for developing mixed methods diagrams.
- In addition to a diagram, many researchers find it useful to provide a table outlining the specific project sampling, procedures, and analytic strategies utilized to address each of the study aims. This helps to overview the research design in a concise format. See the provided examples: Box 2 and Box 3.
- Provide detailed information describing and justifying the rigor of the qualitative data collection and analysis plans. Qualitative data collection topics should include the qualitative design/approach, purposeful sampling strategy, sample size considerations, participant recruitment, data sources, data collection protocols, how data will be recorded, procedures, and relevant ethical issues. Qualitative data analysis topics should include data preparation procedures such as transcription, coding and theme development strategies, and qualitative data analysis software tools. In addition, detail specific strategies that relate to the rigor of the qualitative approach to ensure a high level of trustworthiness, credibility, transferability, and accuracy of the qualitative findings and conclusions. Examples of strategies include: triangulating multiple data sources or investigators when applicable, member checking results when participants are available to provide feedback, inter-coder agreement checks when reliability of codes is important, and prolonged engagement when data are collected in the field.
back to top
- Provide detailed information describing and justifying the rigor of the quantitative data collection and analysis plans. Quantitative data collection topics should include the quantitative design/approach, the population, sampling strategy, target sample size, participant recruitment, instruments (including reliability and validity evidence), intervention conditions (if relevant), procedures, data handling, and relevant ethical issues. Quantitative data analysis topics should include the tools and procedures for appropriate descriptive and inferential statistical analyses. In addition, detail specific strategies that relate to the rigor of the quantitative approach and address threats to internal and external validity.
- Balance the information provided about the qualitative and quantitative data collection and analysis plans. Although investigators should provide sufficient details about both types of data they plan to collect, many tend to write more about one method than the other because they know more about it. This can lead to a strong qualitative or quantitative section and a weak quantitative or qualitative section. The presentation of the information, however, should be in proportion to the goals of the investigation. Therefore, an imbalance may be particularly problematic in convergent approaches where the results of each method are to be compared. Investigators need to be cognizant of any potential imbalance and ensure that they are providing an adequate level of detail about each approach. If an imbalance in the presentation occurs that does not match the balance suggested by the aims, investigators may consider adjusting their team to augment the expertise in the weaker area.
- Explicitly state how the quantitative and qualitative components of the study will be combined. Mixed methods research requires the integration of the quantitative and qualitative approaches within an overall design. Investigators need to describe this essential step in their applications.
- In mixed methods approaches that involve the concurrent implementation of quantitative and qualitative methods, discuss the planned merging analytic and interpretation procedures for comparing, relating, or synthesizing the quantitative and qualitative data and results after discussing their separate analysis procedures. Include a discussion of how potential divergent or inconsistent findings will be managed and interpreted.
- In mixed methods approaches that involve the sequential implementation of the quantitative and qualitative methods, discuss the planned procedures for connecting from the results of one phase to the collection of data in a subsequent phase (e.g., how the qualitative findings will be used to develop a quantitative instrument or treatment or how the quantitative results will be used to inform or design a qualitative follow-up). Investigators also should state how they will interpret the two sets of connected results.
- Order the data collection and analysis topics to match the overall mixed methods designs and study aims. The order of these topics needs to facilitate a concise description by the application authors. The order also needs to be logical and easy to follow for the reviewers.
- In mixed methods approaches that involve the concurrent implementation of quantitative and qualitative methods, investigators often discuss the collection of both types of data (quantitative and qualitative) before discussing the analysis of both types of data (quantitative and qualitative). That is, a typical approach section for a concurrent implementation would be ordered:
- Data collection
- Data analysis and interpretation
- Integration/merging procedures
- In mixed methods approaches that involve the sequential implementation of the quantitative and qualitative methods, investigators often discuss the collection and analysis of the first type of data (quantitative or qualitative) and then discuss the collection and analysis of the subsequent type of data (qualitative or quantitative). That is, a typical approach section for a sequential implementation would be ordered:
- First phase (quantitative or qualitative)
- Data collection
- Data analysis and interpretation
- Connecting procedures (e.g., development of sampling procedures or materials based on the results from the first phase)
- Second phase (qualitative or quantitative)
- Data collection
- Data analysis and interpretation
- Describe potential methodological issues and challenges that may arise in using a particular mixed methods design. This description should identify specific challenges and include suggestions of how the investigators plan to address or overcome the challenges, citing examples from methodological references and past research using the design when possible.
- Include a timetable of the quantitative, qualitative, and mixed methods procedures involved in the design. Researchers unfamiliar with qualitative research and/or mixed methods research often underestimate the time required for completing qualitative data collection and analysis and for integrating the two research approaches. Be sure to plan on sufficient time to allow for transcribing qualitative data, meaningful qualitative data analysis, and the integration of the quantitative and qualitative data and results.
- In discussing Preliminary Studies for New Applications, highlight prior quantitative, qualitative, and mixed methods research experiences. Each of these approaches requires researchers to understand different sets of assumptions and procedures and to have skills for their implementation. Foundations for conducting quantitative, qualitative, and mixed methods research should be adequately documented through prior experiences and/or supplemented through the inclusion of individuals with expertise in these three areas.
Additional application elements
The use of mixed methods research can have implications for the research plan development that go beyond the Specific Aims and Research Strategy. Here are additional considerations relevant to other elements of an R series application.
- The project summary/abstract
- The protection of human subjects
- Facilities & other resources
- Biographical sketches
The project summary/abstract
Include key information about the mixed methods approach of the project in the Project Summary. A good mixed methods project summary highlights the use of mixed methods by including:
- terms that identify the use of mixed methods research;
- an indication of the mixed methods design;
- the general approach (quantitative, qualitative, or mixed methods) planned for accomplishing each study aim; and
- the primary means by which the quantitative and qualitative approaches will be merged, connected, or integrated.
The protection of human subjects
- Identify and describe issues related to the protection of human subjects as required with all forms of research involving human participants.
- Understand the ethical issues associated with quantitative and qualitative research procedures. Researchers using mixed methods need to be cognizant of the ethical issues typically associated with quantitative research approaches (e.g., implications of manipulating conditions experienced by participants) and qualitative research procedures (e.g., implications of gathering personal information through audio-recordings that could identify a participant).
- Anticipate ethical issues specifically related to the use of mixed methods research. The use of mixed methods research may introduce ethical issues that are in addition to those typically faced in research that uses a single quantitative or qualitative research approach. Here are three examples of ethical issues that may relate to the use of a mixed methods approach.
- Mixed methods research may require the collection of identifying information from participants. Quantitative survey research often is conducted in such a way as to maintain the anonymity of the participants. If this same procedure is part of a sequential mixed methods approach, however, the investigator may need to gather information that can identify participants so that they may be contacted for a qualitative follow-up phase. This requires that the researcher justify the need for gathering identifying information and put safeguards in place for protecting that information.
back to top
- Mixed methods research may require contacting participants at a later time for more information. A strength of many mixed methods approaches is that investigators return to participants at a later time to follow-up on initial results. Although this may help provide greater insights on a topic, it also requests more time from participants. If such follow-up activities are planned, investigators should inform participants about the possibility of a follow-up contact so they understand the full nature of their participation at the time of initial consent.
- Mixed methods research may place a higher burden on participants than single-method approaches, but may also provide greater benefit. Mixed methods research allows researchers to study a topic using different approaches. These different approaches, however, may place an additional burden on participants, particularly those who are experiencing health difficulties or who have a high level of constraint on their available time. Investigators should carefully consider and articulate the need for multiple forms of data and select means of data collection that will not overly burden participants. Likewise, investigators should carefully consider and identify any unique benefits that arise from the use of mixed methods, such as the potential for individual or community impact.
- Be prepared to educate IRB reviewers at your institution about mixed methods research. Just as many researchers are new to mixed methods research, many individuals serving on Institutional Review Boards may not have expertise in mixed methods research. Investigators need to be prepared to provide additional explanation and justification for the use of mixed methods procedures and the ethical considerations that go along with its use. As mixed methods studies are complex, some researchers have found a table or matrix can provide an effective overview of the project procedures and sampling. Explanation of sample size may be particularly challenging for the qualitative component for studies that use saturation (i.e., sampling to a point where no new information is obtained from participants) as a criterion for determining sample size. It may be prudent to estimate the upper limit of expected need for participants as IRBs may not be accustomed to exceeding projected levels of recruitment.
Facilities & other resources
Investigators need to describe the scientific environment that will contribute to the probability of success of the proposed mixed methods project. High-quality mixed methods applications should:
- Specify the equipment and expertise available to support sophisticated quantitative research. These resources might include computers, online survey software, statistical software packages (e.g., SPSS, SAS, etc.), resources to support intervention procedures, and expertise in statistics.
- Specify the equipment and expertise available to support sophisticated qualitative research. These resources might include computers, digital recording equipment, space for interviewing, transcription services, qualitative data analysis software packages (e.g., MAXQDA, NVivo, Atlas.ti, HyperResearch, etc.), resources to support entry into a site, and expertise in the procedures of the chosen qualitative approach.
- Specify the equipment and expertise available to support sophisticated mixed methods research. Examples of these resources might include software packages that facilitate the relating of quantitative and qualitative data, expertise in developing quantitative instruments from qualitative findings, and expertise in mixed methods research designs and approaches.
Investigators need to use the NIH format for providing biographical sketches related to each key personnel. This presents a good opportunity to highlight the backgrounds, contribution, and synergy of the mixed methods team members.
- Highlight prior quantitative, qualitative, and mixed methods experiences in the biographical sketches as appropriate to demonstrate investigators' expertise. This includes tying each person's experience and qualifications to their roles in the overall mixed methods study.
- Provide clear delineation of responsibility relative to the roles for integration as this typically requires collaboration between qualitative and quantitative researchers on the team.
- Synthesize across bio-sketches so that each person's unique role is clear, and the roles are seen as distinct yet complementary. When done well, this should demonstrate the synergy of the mixed methods team that has been assembled. In addition, clearly identify if a given person is going to serve as leader on all, or certain specific elements of the mixed methods activities.
- Include adequate personnel time and resources to complete the quantitative, qualitative, and mixed methods aspects of the project. Recognize that there are costs associated with the implementation of each component of a mixed methods project, including the integration, and that investigators need to budget accordingly. Examples of costs specifically associated with a mixed methods project include:
- providing training in quantitative, qualitative, and mixed methods procedures for personnel unfamiliar with one or more of these approaches;
- purchasing software that facilitates the linking of quantitative data to qualitative data;
- incorporating team aspects that support meaningful integration of study components; and
- covering the effort required to address the mixed methods aim(s), such as supporting additional analyses required to integrate two sets of results.
back to top
Key references and resources
References for writing about mixed methods research:
- Creswell, J. W., & Plano Clark, V. L. (2011). Designing and Conducting mixed methods research (2nd ed.). Thousand Oaks, CA: Sage.
- Sandelowski, M. (2003). Tables or tableaux? The challenges of writing and reading mixed methods studies. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 321-350). Thousand Oaks, CA: Sage.
- Stange, K. C., Crabtree, B. F., & Miller, W. L. (2006). Publishing multimethod research. Annals of Family Medicine, 4, 292-294.
References for developing diagrams for mixed methods research studies:
- Ivankova, N. V., Creswell, J. W., & Stick, S. (2006). Using mixed methods sequential explanatory design: From theory to practice. Field Methods, 18(1), 3-20.
- Plano Clark, V. L., & Creswell, J. W. (2008). The mixed methods reader. Thousand Oaks, CA: Sage.
References for reasons to mix methods:
- Bryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6(1), 97-113.
- Greene, J. C., Caracelli, V. J., & Graham, W. F. (1989). Toward a conceptual framework for mixed-method evaluation designs. Educational evaluation and policy analysis, 11(3), 255-274.
References for writing mixed methods study aims:
- Plano Clark, V. L., & Badiee, M. (2010). Research questions in mixed methods research. In A. Tashakkori & C. Teddlie, Handbook of Mixed Methods Research in Social & Behavioral Research (2nd ed., pp. 275-304). Thousand Oaks, CA: Sage
- Tashakkori, A., & Creswell, J. W. (2007). Exploring the nature of research questions in mixed methods research [Editorial]. Journal of mixed methods research, 1(3), 207-211.
References for developing high quality, rigorous, and ethical quantitative and qualitative research:
- National Institutes of Health (NIH). (1999). Qualitative methods in health research: Opportunities and considerations in application and review. Washington, DC: Author. Retrieved from: obssr.od.nih.gov/pdf/qualitative.pdf
- National Institutes of Health (NIH). (2010). Bioethics resources on the web. Retrieved from: http://bioethics.od.nih.gov/
- Cohen D., & Crabtree, B. (2006). Qualitative research guidelines project. http://www.qualres.org/
- Padgett, D. K. (2008). Qualitative methods in social work research (2nd ed.), Thousand Oaks, CA: Sage.
- Shadish, W. R., Cook, T. D., Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal Inference. Boston: Houghton Mifflin.
References for developing high quality, rigorous, and ethical quantitative and qualitative research:
- Everything You Wanted to Know About the NCI Grants Process: http://www3.cancer.gov/admin/gab/2005GPB/GPB05-LowRes.pdf
- NIAID's All About Grants tutorials page (multiple tutorials that apply regardless of research area): http://funding.niaid.nih.gov/researchfunding/grant/pages/aag.aspx
- NIH-OER on planning your application: http://grants.nih.gov/grants/planning_application.htm
- NIH-CSR review process videos: http://cms.csr.nih.gov/ResourcesforApplicants/InsidetheNIHGrantReviewProcessVideo.htm
- NIH-OER on writing your application: http://grants.nih.gov/grants/writing_application.htm
back to top