Clinical research is an essential feature of modern medical practice. Although most areas of medicine present challenges to conducting high-quality controlled studies, the prehospital and emergency care environments are particularly challenging due to numerous uncontrolled variables inherent in developing and implementing meaningful and ethical consent processes. Despite this, an ever-growing body of research in the area of EMS Medicine has helped solidify the subspecialty and moved prehospital care toward a more evidence-based practice. EMS physicians and medical directors must be able to interpret the literature and should aid in its advancement whenever possible.
Describe basic research concepts and definitions.
Describe patient groups and study designs.
Describe scientific levels of evidence and grades of recommendation.
Standardized reporting of clinical trials.
Describe the development of EMS research.
National Research EMS Agenda.
Describe the design of an EMS study, including development of the research question.
Describe the usual IRB process and approach to clinical research.
Describe the concept of emergency exception for informed consent.
Describe community consultation and public disclosure.
Define basic statistical terms.
Describe some basic research pitfalls.
List existing EMS research databases and discuss how to request access to the data.
List sources of potential funding for EMS-related research.
Scientific research is meant to draw investigators and readers of the scientific literature closer to “the truth.” Study design and limitations of conducting research in the clinical environment can drastically affect the results of clinical research, potentially altering the truth being sought. In cases where meticulous design considerations have limited the presence of bias and error, it is possible to utilize study results to advance the practice of medicine. This is the basis for the concept of evidence-based medicine. Acknowledging that not all research study results lead to applicable medical practices, it is important to consider how to best interpret current EMS research results and design future studies.
BASIC SCIENTIFIC CONCEPTS AND TERMS
In order for EMS physicians to provide state-of-the-art medical direction and oversight of EMS systems, a basic grasp of the methodology behind clinical research is required. Research concepts and the definitions of common research terms are prerequisites for this understanding.
Efficacy is a description of how well the treatment works in clinical trials (“explanatory”).
Effectiveness refers to how well the treatment works in the practice of medicine (“pragmatic”).
Validity in research is the degree to which a tool measures what it claims to measure. The validity of a study refers to determining the likelihood that the conclusions drawn from the study are correct or reasonable. A valid study asks the appropriate questions, uses the correct sample (in size and character), collects the correct outcome measures, and utilizes correct statistical methods. This is a very complex process that requires strict adherence and reassessments to produce a truly valid study.
Internal validity: Internal validity considers the direct effect of one variable (the independent variable) on another variable (the dependent variable). It is important in studies designed to show cause-and-effect relationships. At times, even properly designed studies may have confounding variables that interfere with internal validity.
Confounding variables: There are eight types of confounding variables that interfere with internal validity: (1) history—specific events occurring between measurements (in addition to any experimental variables); (2) maturation—participant changes over time (eg, becoming tired, growing older, etc); (3) testing effect—the effect that taking the first test has on taking any additional testing; (4) instrumentation— refers to changes in measurement tool calibration or the changes in observers that may change measurements; (5) statistical regression—when selected groups are selected based on their extreme score; (6) selection bias—results from the differential (nonrandom) selection of respondents for the comparison groups; (7) experimental mortality—the loss of participants from comparison groups; (8) selection-maturation interaction—occurs when participant variables (eg, hair or skin color) and time variables (eg, age, obesity) interact.1
External validity: Relates to the extent to which the results of a (internally valid) study remain true in other cases (ie, different populations, places, or times). Can the study findings be generalized? Are the research participants representative of the general population? Many studies are performed in a single geographic area, with smaller samples or patients, who also possess unique characteristics or are representative of a specific population only (ie, volunteers, military cadets, medical students). Studies that are not generalizable have low external validity. Other factors adversely affect external validity include (1) testing effect; (2) selection bias; (3) experimental arrangements— which are not generalizable to patients in a nonexperimental setting; and (4) multiple-treatments interferences—effects of previous treatments that interfere with present testing and are not erasable.
Internal versus external validity: It might appear as if internal and external validity contradicts each other. If a strict adherence to experimental designs control as many variables as possible, the study may have high internal validity. Yet, this highly artificial setting lowers the external validity. Alternatively, in performing observational research, it is difficult to control for interfering variables and lowers the low internal validity. However, the study of environmental or other measures in a natural setting results in higher external validity. Fortunately, these apparent contradictions are resolvable, as a great many studies primarily wish to deductively test a theory, in which the major consideration is the rigor (internal validity) of the study.
Blinding refers to procedures undertaken to ensure that neither the study participants nor any member of the study team know to which group the participant belongs (treatment or nontreatment). Some studies have been classified as single, double, or triple blinded, depending on whether it was the participants, care team, or outcome assessors that were blinded. It is currently accepted that investigators should refrain from this terminology and simply state the type of blinding within the test of the paper.2
Inclusion criteria are conditions that must be met for the appropriate recruitment of subjects into a clinical study.
Exclusion criteria are conditions that must be met for the appropriate rejection of subjects from a clinical study.
There are different paradigms for performing analysis of data from clinical studies. When attempting to limit bias and evaluate the effect of introducing a clinical intervention on a particular population it is best to utilize a study design that incorporates intention-to-treat analysis. Other poststudy analysis and interim analysis may be appropriate in some circumstances, but the conclusions drawn from these may be less accurate.
Intention–to-treat (ITT) analysis: The objective is to analyze each group exactly as they existed upon randomization. A true ITT analysis is possible only when complete outcome data are available for all randomized subjects. This means to include all subjects, including those that drop out. ITT analyses decrease outcome bias.
Subgroup analysis: Analyzing groups within the groups being studied. Subgroup analyses are discouraged because multiple comparisons may lead to false-positive findings that cannot be confirmed.
Interim analysis: A pretrial strategy for stopping a trial early if the results show large outcome differences between groups. It allows for periodic assessments for beneficial or harmful effect of treatment compared to concurrent placebo or control group while a study is ongoing. It is used as a cost-saving measure and importantly the ethical obligation that only the minimum number of patients should be entered into a trial to achieve the study's primary objective and reduce the participants' exposure to the inferior treatment. This analysis may occur following the inclusion of a certain number of treatments or after a set period of time. However, the way in which the interim analysis is to be conducted must be expressly stated in the study protocol. Additionally, the results of the interim analysis should be evaluated by an independent data monitoring committee.3 Other reasons to stop an interim analysis are that there are unacceptable side effects or toxicity, accumulation is so slow that the trial is no longer sufficient, outside information makes the trial unnecessary or unethical, poor execution compromises the studies' ability to meet its objectives, or disastrous fraud or misconduct.
Bias usually refers to any unintended influence that a particular facet of the study design may have that will alter or skew the results. Typically investigators seek to limit bias as much as possible; however, much of the medical literature is affected by bias in some form.
Selection bias: Usually results from an error in choosing the individuals or groups to participate in a study. It distorts the statistical analyses and may result in drawing incorrect conclusions regarding the study outcome(s). It weakens internal validity.
Sampling bias: A systematic error in a study that occurs because the participants do not represent a random sampling of the population. This occurs in some instances due to participant self-selection or prescreening of trial participants. The result is that some members of the population are less likely to be included than others. It weakens external validity.
Attrition bias: A kind of selection bias caused by the loss of participants. It includes patients that dropout or do not respond to a survey (nonresponders), or who withdraw or deviate from the study protocol. It results in biased results because a study intervention or nonintervention is unequal or underrepresented in the outcome.
Publication bias: A bias regarding what is most likely to be published, positive or negative findings. If negative findings are underreported, it leads to a misleading bias in the overall published literature. Studies suggest that positive studies are three times more likely to be published than negative studies. Trial registration is now required by many journals to ensure that unfavorable results are not withheld from publication.4
Participants are arbitrarily assigned to a treatment (intervention) or nontreatment (control) group. Randomization eliminates bias in group assignment, aids in blinding the investigator, participants, and other assessors from knowing the grouping of study participants, and allows the use of probability theory to express the likelihood that any outcome differences between groups merely indicate a chance finding.
Cluster randomization: A preexisting group of study participants (schools, poisoning victims, families) are randomly selected to receive (or not receive) an intervention. Cluster randomization is sometimes done due to factors related to study participants (ie, all family members are placed in the same treatment or nontreatment group)
Factorial randomization: Each participant is randomly assigned to a treatment (or nontreatment group) that receives a particular combination of interventions (or noninterventions).
Randomization procedure: Generation of an unpredictable sequence of allocations to be distributed to participants to treatment or control groups using an element of chance, following the patient's evaluation of eligibility and recruitment into the study.
Allocation concealment: Very strict protocols to ensure that patient group assignments are not revealed prior to their allocation to a group. Sequentially numbered, opaque, sealed envelopes (SNOSE) is a type of allocation concealment
As of July 1, 2005, the International Committee of Medical Journal Editors (ICMJE) announced that all RCTs must be registered to be considered for publication in member journals. This registration may occur late.5
During the design phase of a study, it is important to define groups. Based on the study type, there may be several different types of groups needed for a particular study.
Control group: A patient group that receives no treatment
Placebo control group: A patient group that receives no treatment, but will receive a “sham” or “placebo” treatment that will mimic what is being performed in the “test group” but without physiological or real effect.
Parallel group: Participants are randomly allocated to a group, and study participants either receive or do not receive an intervention.
Crossover group: Participants are randomly allocated to a group, and, over time, all study participants receive and do not receive an intervention in a random sequence.
EVALUATING THE SCIENTIFIC LITERATURE
Certain attributes of studies produce better evidence than others. It is important to consider the design of the study when determining how to interpret a study. Some studies may have seemingly ideal designs, whereas others do not. The constraints of design related to the clinical environment many times limit the investigators' ability to design a study with all of the ideal parameters. In general, the following attributes can be used to distinguish between stronger and weaker level of evidence.
Randomized studies are superior to nonrandomized ones.
Prospective studies are superior to retrospective studies.
Blinded studies (in which patients, and clinicians and data analysts where possible, do not know which intervention is being used) are superior to unblinded studies.
Controlled studies are superior to uncontrolled ones.
Experimental study designs are superior to observational study designs.
Contemporaneous (occurring at the same time) control groups are superior to historical control groups.
Internal control groups (ie, managed within the study) are superior to studies with external control groups.
Large studies (ie, involving enough patients to detect with acceptable confidence levels any true treatment effects) are superior to small studies (ie, properly powered studies are superior to underpowered studies).
Studies that clearly define patient populations, interventions, and outcome measures are superior to those that do not clearly define these parameters.
RANDOMIZED CONTROLLED TRIAL
In a randomized controlled trial (RCT), subjects are assessed for eligibility and recruitment into the trial. Then, before the intervention or treatment begins, the patients are randomly allocated to receive one treatment or another of the treatments being studied. After the patients are randomized to one treatment arm or the other(s), they are followed in exactly the same way, with the only difference between the two groups being the randomly allocated treatment(s). RCTs are the gold standard for a clinical trial. They are often used to test the efficacy or effectiveness of various types of medical interventions within a patient population. An RCT must contain a control group or a previously tested treatment (a positive-control study). The advantage of an RCT is that they are so strictly controlled that they reduce study bias and the confounder of causality. The disadvantages of RCTs are that they are very expensive, time consuming, may have lower external validity, and perhaps unethical for certain orphan diseases (rare) or rapidly morbid or mortal conditions or injuries which do not lend themselves to RCT evaluation. The types of RCTs are listed below.
Superiority trials: Most RCTs are superiority trials, wherein one intervention is hypothesized to be significantly superior to another.
Noninferiority trials: These RCTs seek to determine whether a new treatment is no worse than an established treatment.
Equivalence trials: These RCTs seek to determine whether two interventions are indistinguishable from each other.
RANDOMIZED (UNCONTROLLED) CLINICAL/COMPARATIVE TRIAL
A randomized (uncontrolled) clinical/comparative trial is a trial that compares multiple treatment groups with each other in the absence of a control group.
These types of studies also called quasi-experimental studies. They do not have random assignments to either a control or treatment group. In these studies, the researcher controls the assignment to a grouping (ie, sicker patients receive the treatment) or circumstances dictate grouping (ie, patients presenting on nights and weekends are allocated to the control arm). These studies typically have lower internal validity because the two groups may not have comparable baselines.
Observational studies typically are designed to follow patients of a particular type, or whom have specific common features or risk factors.
Cohort studies: These studies are longitudinal studies, in that they look at an effect of an intervention on a patient population over time. It may assess risk factors in a group of patients who share common characteristics (ie, diabetes or heart disease risk factors) and it may or may not compare the effect of the study intervention to a comparison group (ie, a similar group of patients without the intervention) or to historical groupings previously studied. Several types of cohort studies are discussed below.
Prospective cohort studies: A study that sets out to evaluate patients that present forward from some set time. They usually seek to determine risk factors that affect the studied group over time. These are important studies in that it is unethical to perform such as study as an RCT and purposely expose patients to risk factors. Prospective studies are of a higher level of evidence than retrospective studies.
Retrospective cohort studies: These studies are also called historic (post hoc) cohort studies. They review the records of patients that have already been treated or an event that has already taken place. These studies are easier to complete than prospective studies, less expensive and may allow for the study of rare diseases. The disadvantages are that some important statistical measures cannot be made and significant biases (selection and informational) may confound the results. Moreover, these studies rely on precise record taking that has already taken place, which may have key data points, epidemiological, or treatment information missing or unavailable.
Time series studies are studies run through a period of time, in order to test a hypothesis or make an observation.
Case-controlled study: This is an observational study in which two groups with different outcomes are identified and compared. These studies are often performed to identify contributing factors in patients with a particular condition with patients who do not have that particular condition. They are simpler to perform than other studies and less expensive. Additionally, they are usually used as preliminary studies for research questions related to topics that have limited known information. However, the conclusions drawn may be weaker than more rigorously performed studies.
Nested case-control study: This variation of a case-controlled study evaluates only a subset of controls from the cohort to the incident cases. This subset is selected to match risk sets with the incident cases. In a case-cohort study all incident cases are compared. These studies may be analyzed using methods that take missing covariates into consideration.
Cross-sectional studies: These studies are also called cross-sectional analyses, transversal studies, or prevalence studies. These are observational studies involving either all members of a population or a representative subset of that population, at a specific point in time.
Ecological study: This is an epidemiological study in which large populations, rather than individuals are evaluated. These studies are considered inferior to cohort and case-controlled studies because of the concept of ecological fallacy (a false interpretation of the data made because inferences are made regarding individuals based on inferences from the group to which those individuals belong).
SCIENTIFIC LEVELS OF EVIDENCE
This is a ranking system used in evidence-based medicine to assign a strength rating to a clinical trial or research study. There are five levels of evidence (I, II, III, IV, and V) with alphanumeric subsets (Table 10-1).
Level of Evidence
|Favorite Table|Download (.pdf) TABLE 10-1
Level of Evidence
|I a ||Systematic reviews (with homogeneity) of more than one RCT |
|I b ||Evidence from at least one well-designed RCT with a narrow confidence interval |
|II a ||Systematic review (with homogeneity) of at least one well-designed nonrandomized controlled trial or cohort study |
|II b ||Evidence from at least one well-designed cohort study or a lower quality RCT (> 80% follow-up) |
|III a ||Systematic review (with heterogeneity) of case-controlled studies |
|III b ||Evidence from individual case-controlled studies |
|IV ||Evidence from case series and lower quality cohort studies |
|V ||Expert opinion based on physiology, bench research, or “first principles” |
These are ranking of the strength of medical evidence. These “grades” are based on the level of scientific evidence on the subject matter and used in clinical practice guidelines to summarize work quality of the literature (Table 10-2).
Grades of Recommendation
|Favorite Table|Download (.pdf) TABLE 10-2
Grades of Recommendation
|A ||Consistent Level 1 Studies |
|B ||Consistent Level 2 or 3 studies or Extrapolations from Level 1 studies |
|C ||Level 4 studies or extrapolations from Level 2 or 3 studies |
|D ||Level 5 evidence or troublingly inconsistent or inconclusive studies of any level |
STANDARDIZED REPORTING OF CLINICAL TRIALS
The interpretation of clinical trials measuring effects of interventions on clinical outcomes is made more complex by inherent variability in the way trials have been designed and their results reported. This is made even more clear when attempting to compare and contrast results of multiple trials on the same intervention. Meta-analysis studies can be made more complex (a potentially less meaningful) when the body of literature on a particular intervention is limited by nonstandard reporting of outcomes/results.
Outcome research was introduced by health care research leaders such as Codman (1910) and Donabedian (1966). Outcome measurement is clearly a defining component of medical research. The various types of outcome measures include physical/clinical (mortality), performance/function (self-care), economic (cost/benefit), and humanistic (quality of life).6 When measuring outcomes it is common to utilize outcome measurement/indicator tools. These may be general health/generic measures, disease-specific measures, or functional status measurement tools. The tools may represent a direct measure of a desired outcome, or a tool designed to demonstrate an outcome that cannot be directly measured. These may be considered surrogate endpoints in some cases. Clinical endpoints may also be categorized as primary (one of more endpoints that have been powered for rejection of the null hypothesis), secondary (other prespecified endpoints that have been powered for hypothesis testing), and tertiary (exploratory). Endpoints can also be categorized by type of data as well (Table 10-3).7
Examples of Endpoints by Type of Data
|Favorite Table|Download (.pdf) TABLE 10-3
Examples of Endpoints by Type of Data
|Continuous measurements ||Blood pressures, weight, blood chemistry variables |
|Event times ||Time to recurrence of CHF exacerbation, survival time, length of stay |
|Counts ||Frequency of occurrence of migraine headaches, number of uses of rescue meds for asthma |
|Binary endpoints ||Recurrence (yes or no), major cardiac event in 30 days (yes or no) |
|Ordered categories (scales) ||Absent-mild-moderate-severe pain, NYHA class status |
|Unordered categories ||Categories of adverse experiences: GI, cardiac, etc |
In some cases, authors will combine endpoints after data analysis and may show statistic significance based on the combined outcome, despite the lack of significance when evaluating each independently. This may lead to some confusion and is not considered to be the preferred methodology.
Consolidated Standards of Reporting Trials
Consolidated standards of reporting trials (CONSORT) is a collection of initiatives developed by the CONSORT Group to help ameliorate problems that may arise from inadequate reporting of RCTs. The main objective of the CONSORT Group is the development of the CONSORT Statement.8 This is a minimum set of recommendations for the reporting of RCTs that is standardized and promotes “transparent reporting,” the reduction of bias, and aids readers in their appraisal and interpretation of the RCT. The most recent version contains a 25-item checklist for researchers as well as a flow diagram with descriptive text. There is an additional document called the CONSORT “Explanation and Elaboration” document that is advocated strongly for concomitant use with the CONSORT Statement.9
DEVELOPMENT OF EMS RESEARCH
The current areas of focus in EMS research are in the prehospital clinical sciences, EMS systems, and the EMS education realms. The current clinical areas of focus are out-of-hospital cardiac arrest (OHCS), major trauma, stroke, shock, and spinal immobilization. Systems-based areas of focus include transport of specialty patients, STEMI coordinated care, stroke coordinated care, trauma coordinated care, and ED crowding/ off-load times. Some educational areas of focus include evidence-based decision making in EMS, advanced-airway adjuncts, integration of advanced practice providers (ie, physician assistants, nurse practitioners), and community paramedicine.
COMPONENTS OF A SUCCESSFUL PROGRAM
Development of a successful research program can be a significant undertaking and may represent considerable investment in time and funds. In order to maximize return on the investment it is important to consider the components required for successful development.
Training: In an EMS system, researchers may come from a variety of different backgrounds. Basic and advanced training in study design, participant expectations, ethical principles, study development, and statistical analyses is advised. Many institutions provide modules on these topics, and the facility's Institutional Review Board (IRB) may be an invaluable resource.
Mentorship: In many studies, mentorship is essential. For those just beginning a research study, the mentorship of a seasoned investigator is very much advised. Many institutions, universities, and medical centers have such experienced mentors available.
Collaboration: Teaming up with like-minded researchers from related fields is advisable in many situations (Figure 10-1). For example, a study related to an advanced airway adjunct may benefit from collaborators from emergency medicine, surgery, and anesthesiology. Collaboration allows different viewpoints and experience to be shared among researchers, often leading to a more efficient and streamlined area of focus.
EMS research group meeting. Like-minded collaborators including a simulations expert, education program director, prehospital providers, and EMS physicians discuss their local research agenda.
NATIONAL EMS RESEARCH AGENDA
The National EMS Research Agenda, published by the National Highway Transportation and Safety Administration in 2001, provides an assessment of the state of EMS research and recommendations for the continued advancement of this critical area.10 This document outlines areas of weakness and specific challenges that limit the advancement of EMS research programs. It advocates for policy changes and enhanced funding for the development of researchers, research centers, and studies. Additionally, it proposes a greater organization of stakeholders to ensure research finders are utilized in the development of evidenced-based practice. The recommendations of the authors of the document are illustrated in Table 10-4 (paraphrased from the executive summary).
NHTSA National EMS Research Agenda
|Favorite Table|Download (.pdf) TABLE 10-4
NHTSA National EMS Research Agenda
|Recommendation 1 ||Large cadre of career EMS investigators should be developed and supported in the initial stages of their careers; training programs with content directed toward EMS research methodologies should be developed |
|Recommendation 2 ||Centers of excellence should be created; bring together experienced investigators, institutional expertise, and resources such as budgetary and information systems support |
|Recommendation 3 ||Federal research sponsoring agencies should acknowledge commitment to EMS research |
|Recommendation 4 ||States, corporations, and charitable foundations should be encouraged to support EMS research |
|Recommendation 5 ||Organize efforts of EMS professionals, delivery systems, academic centers, and public policy makers to support and apply results of research |
|Recommendation 6 ||EMS professionals of all levels should hold themselves to higher standards of requiring evidence before implementing new procedures, devices, or drugs |
|Recommendation 7 ||Standardized data collection methods at local, regional, state, and national levels; EMS provider agencies should adopt the uniform prehospital data elements for data collection |
|Recommendation 8 ||Food and Drug Administration (FDA) and Office for Human Research Protections (OHRP) should work with EMS research stakeholders to evaluate requirements for exception from informed consent in emergency situations and address serious impediments to conducting EMS research |
There is a new trend in scientific research termed team science. It involves the collaboration of researchers to enhance and further research goals. There are several types of research teams, including:
Independent research: Usually a single investigator working independently on a research question or with one or more collaborators.
Collaborative efforts: May range from independent research, to collaborative efforts among several investigators, to fully integrated research teams.11
FORMULATING THE RESEARCH QUESTION(S)
One of the most apparently simply but challenging steps in the research process is formulation of the “research question.” It is the first, and possibly the most important step in the research design. It sets the basis from which the study is designed and is supposed to ensure that the study actually answers the proposed question. Prior to formulating the hypothesis and null hypothesis for any proposed study, the researcher(s) must carefully develop the research question(s).
PICO process: This process is used in evidenced-based medicine research to properly frame and answer a clinical question (Box 10-1).
FINER criteria: This is a set of criteria developed to promote the development of a good research question. (Box 10-2)12 Box 10-1 The PICO Process
Patient problem or population to be studied
Intervention to be evaluated
Comparison between group(s)
Outcome of Study Box 10-2 The FINER Criteria
Feasible: Adequate numbers of subjects, appropriate technical expertise, affordable study in terms of time and funding, and with an a study that is manageable in its scope
Interesting: The project outcome is intriguing to researchers, peers, and the community
Novel: The study will confirm, refute, or extend previous work
Ethical: The study conforms to IRB specifications and moral principles
Relevant: The study outcome is pertinent to existing scientific knowledge, clinical and/or health care policy, and future research
REFINEMENT OF RESEARCH QUESTIONS, HYPOTHESES, AND RESEARCH OBJECTIVES
After the initial development of the research question is complete and stakeholders are satisfied with the basic principles surrounding the investigation, the question and hypotheses must be further elucidated and refined in order to define the specific research objectives. This can be accomplished by following a basic set of steps.
Systematic literature review: This is often the first step undertaken. It is performed to develop an understanding of previous work, the overall level of evidence of the subject matter, controversies in the literature, and knowledge gaps within the field study.
Intensive review of scientific trends and technological advancements within the field of study: This is important to assess the rigor with which the proposed study must be performed in order to align with the current highest levels of investigation.
Question refinement review(s): The question, once developed, must be shared with mentors, colleagues, and collaborators so that it can be refined and honed into the simplest yet most complete question that encompasses the proposed scope of the investigation.
FINER criteria review: Is then used to further test the appropriateness of the research question.
PICO criteria: Reapplied to ensure that appropriate aspects of the question are addressed.
Development of a research hypothesis: From the research question.
Development of primary and/or secondary objectives: This is done prior to commencement of the study and listed in the IRB and the formal study protocol.
This is the standard or “default” assumption/prediction made at the outset of scientific inquiry. It reasons that there will be no significant difference in outcomes between two groups treated in different ways. One can reject or disprove a null hypothesis. However, a null hypothesis cannot be accepted or proven.
This is the rival hypothesis to the null hypothesis. It states that there will be a significant difference in outcomes between two groups treated in different ways. If the null hypothesis is rejected, the alternative hypothesis is accepted.
BASIC STATISTICAL CONCEPTS
At times it may be appropriate to review study results and then apply a particular statistic methodology in order to satisfy a previously undefined research objective. However, in most cases the study design should include the statistical methodology by which results are to be sought prior to submission and implementation of the study design. It is important to understand the types of measures and the errors that can occur during this phase, and to choose appropriate statistical methodology.
MEASURES OF CENTRAL TENDENCY
Mean: This is probably the most often used descriptive statistic measured. It measures the “central tendency” of a variable and is usually reported with confidence intervals. It is usually calculated as the mathematical average of a set of values. For numbers 1, 3, 5, 7, 13, 25, 27, and 30, the mean is 15.85.
Median: The median is the value for which 50% of the observations will lie above that value and 50% will lie below that value (when all values are ranked). For numbers 1, 3, 7, 13, 25, 27, and 30, the median is 13.
Mode: The mode is the value that appears most often in a data set. It is the value most likely to be sampled. For numbers 1, 3, 7, 13, 13, 13, 25, and 27, the mode is 13.
Type I errors: or “false-positive” errors—an incorrect rejection of a true null hypothesis. It wrongly concludes that a treatment has a positive effect when it does not.
Type II errors: or “false-negative” errors—an incorrect acceptance of a false null hypothesis. It wrongly concludes that a treatment has a negative effect, when it actually is a positive study with regard to treatment significance.
P Value: A P value helps determine the significance of study results. It is a number between 0 and 1. The smaller the P value (≤0.05), the stronger the evidence against the null hypothesis (the null hypothesis is rejected). The larger the P value (>0.05), the weaker the evidence against the null hypothesis and the null hypothesis is not rejected.
Standard deviation (SD): The SD determines how much variation from the average exists with a data set. A low SD indicates that the data are very close to the mean. A high SD indicates that the data are spread out over a large range of values. The SD is the square root of its variance. It is used to measure confidence in statistical conclusions.
Standard error of the mean (SEM): The SEM is the standard deviation (SD) of the different sample means. Approximately 68.3% of the sample means would be within one SEM and 95.4% would be within two SEMs and 99.7% would be within three SEMs.
Confidence intervals: A range of values around the mean where it is expected that the true mean is located. For example, if the mean = 23 (p = 0.05) and the lower and upper confidence intervals are 19 and 27, respectively, then there is a 95% probability that the population mean is >19 and <27. Smaller P values result in wider confidence intervals. The confidence interval depends on the sample size and the variation of data values. Confidence intervals are based on the assumption that the variable is normally distributed in the population being studied. Larger sample sizes have more reliable means. The larger the variation, the less reliable the mean.
Sensitivity: Sensitivity is also called the true positive rate or the recall rate. It describes how well a test can detect a disease or condition in those that have the disease or condition. Sensitivity helps rule out disease when the result is negative (Sensitivity Rule Out = “Snout”). Sensitivity = true positives/(true positives + false negatives).
Specificity: Specificity refers to the percentage of people who test negative for a specific disease among a group of people without the disease. A highly specific positive test, it is highly certain that the patient actually has the disease (Specificity Rule In = “Spin”). A very specific test rules in a disease with a high degree of confidence. Specificity = true negatives/(true negatives + false positives).
Positive predictive value: This test asks the question: “If the test result is positive, what is the probability that the patient actually has the disease?” PPV = true positives/(true positives + false positives).
Negative predictive value: This test asks the question: “If the test result is negative, what is the probability that the patient does not have the disease?” NPV = true negatives/(true negatives + false negatives).
Likelihood ratios: A likelihood ratio is the ratio of two probabilities of the same event under different circumstances. It uses the sensitivity and specificity of a test to determine whether a test result usefully changes the probability that the condition exists. A positive likelihood ratio = sensitivity/1 – specificity. A negative likelihood ratio = 1 – sensitivity/specificity. A likelihood ratio >1 indicates that the test result is associated with the disease. A likelihood ratio <1 indicates that the result is associated with absence of the disease.
Odds ratios: The odds ratio measures the association between an exposure and an outcome. It represents the odds that an outcome will occur given a particular exposure, compared to the odds of the outcome occurring in the absence of that exposure. It is calculated by using a two-by-two frequency table.
Sample size calculation: This is a very important part of a study design that is performed to determine the appropriate number of subjects in each group that will be needed to make a valid inference about the population from which the subjects are taken.
Variable: An object or event that is being measured or evaluated.
Independent variable: A variable that is not changed by other variables being measured or assessed (ie, age, gender, education). When assessing the relationship between variables, one is usually attempting to discern if the independent variable caused a change in the other variables (the dependent variables). Independent variables may cause a change in a dependent variable. However, dependent variables cannot cause a change in an independent variable.
Dependent variable: A variable that may change, depending on the effects of other variables (ie, a test score is dependent upon prior night's restfulness and/or the duration or quality of pretest preparation).
Correlation: A correlation is a measure of the relation between two or more variables.
Pearson correlation: The most widely used type of correlation coefficient. It is also known as the linear or product correlation. It is used to determine the extent to which values of two variables are proportional to each other.
Proportionality: The extent to which two or more variables are linearly related (approximated by a straight line sloping either upward or downward). The line is called a regression line (or least squares line) and is determined such that the sum of the squared distances of all of the data points from the line is the lowest possible.
Correlation coefficients: Can range between −1.00 and +1.00.
Negative correlation: The value of −1.00 represents a perfect negative correlation. With a negative correlation, the relationship between two variables is such that as the value of one variable increases, the value of the other variable decreases.
Positive correlation: The value of +1.00 represents a perfect positive correlation. With a positive correlation, the relationship between two variables is such that as the value of one variable increases, the value of the other variable also increases.
No correlation: The value of 0.00 represents a complete lack of correlation between two or more variables.
TRADITIONAL STATISTICAL METHODS
The Student t test is used to determine if two sets of data are significantly different from each other. It is usually used to assess if any significant difference occurs between groups in response to an intervention measured with the same statistical unit or whether the slope of a regression line differs significantly from zero.
Analysis of variance (ANOVA) is a collection of statistical models used to analyze differences between group means and their associated interventions. It is used for comparing three or more groups' means for statistical significance. It assesses the “variation” both among and between groups. It tests whether or not the “means” of groups are equal. It generalizes the t test to more than two groups. When three or more groups are being compared, it is better to use ANOVA rather than performing multiple two-sample t tests (which might increase the chance of a type I error).
This is a process for evaluating the relationships among measured variables. It focuses on the relationship between a dependent variable and one or more independent variables and aids in the understanding of how the value of the dependent variable changes when any one of the independent variables is varied (while the other independent variables remain fixed.).
THE USE OF HUMANs IN RESEARCH STUDIES
There have been a large number of experiments that have been performed on humans that were unethical, illegal, and immoral. These experiments include deliberately infecting people with debilitating or deadly diseases, exposing subjects to biological weapons, chemical weapons, radiation, torture, interrogation techniques, mind-altering substances, and other similar types of testing. Often these “studies” were performed on at-risk patient populations (pregnant women, children, racial minorities, the mentally disabled, the sick, the elderly, the poor, and prisoners). These types of studies have been banned for 40 years and stringent regulations have been instituted to ensure human subjects are heavily protected from exploitation and dangerous clinical trials.
The Nuremberg Code is a set of research ethics and principles for human experimentation that resulted from the Nuremberg trials at the end of World War II (1947). It was developed in response to the “Doctors Trial” involving 23 doctors that conducted human experiments in concentration camps on German citizens.
THE DECLARATION OF HELSINKI
The Declaration of Helsinki is a set of ethical principles on human experimentation developed in 1964 by the World Medical Association and is considered the “cornerstone” document of human research ethics.
POLICIES FOR THE PROTECTION OF HUMAN SUBJECTS
This document was created by the National Institutes of Health in 1966. It recommended the establishment of IRBs to oversee clinical studies involving human subjects.
THE NATIONAL RESEARCH ACT
This legislation was established by the National Commission for the Protection of Human Subjects in 1974 and mandated that the Public Health Service develop regulations to protect the rights of humans involved in research studies.
The Belmont Report was issued in 1979 by the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. It outlines the ethical principles for research on humans. It was initiated at the Belmont Conference Center, which was once part of the Smithsonian Institution. The report was prompted by concerns related to the Tuskegee Syphilis Study. It outlines three fundamental ethical principles for any human subject research:
Respect for persons: Especially as it relates to the autonomy of all people and in treating them with courtesy and respect. It discusses informed consent and the duty that all researchers have to be truthful and without deception.
Beneficence: This is the philosophy of “do no harm” and in minimizing research subject risk while maximizing research benefits.
Justice: This doctrine ensures that reasonable, nonexploitive, and well thought-out procedures are administered fairly and equally.
The “Common Rule”: The Federal Policy for the Protection of Human Subjects came into effect in 1981, following the 1975 revision to the Declaration of Helsinki. It was fully encapsulated and published in 1991. It contains five subparts.
Subpart A: Contains the “Common Rule” which outlines the basic provisions for IRBs, informed consent, and assurances of compliance.
Subpart B: Provides additional protections for pregnant women, fetuses, and neonates.
Subpart C: Provides additional protections for prisoners.
Subpart D: Provides additional protections for children.
Subpart E: Provides additional protections for the registration of IRBs.
INSTITUTIONAL REVIEW BOARDS
Institutional Review Boards (IRBs) are also called ethical review boards and have been instituted worldwide to formally approve, monitor, and review all biomedical and behavioral research involving human subjects. These tasks are accomplished through uniform procedures and reviews that involve a risk-benefit analysis of the proposed research.
The primary focus of any IRB is to ensure that human subjects are protected from exposure to any potential harm.
EXEMPTIONS FROM IRB REVIEWS
IRB reviews are universally mandated in all but a few circumstances. Some of these exceptions include those involving the study of differing educational practices or testing, staffing instructional studies, curriculum studies, management methods, some surveys, observation studies (unless electronically recorded or those that contain identifiable information), and in some cases research involving the collection of existing data already in a system (deidentified data).
A cornerstone of an IRB process is the obtainment of permission prior to performing an intervention. The subject must understand the risks, benefits, implications, and future consequences of either treatment or nontreatment. The patient must also have appropriate reasoning faculties and the capacity to understand the entire informed consent document. It is very important that the patient understand that in research trials, the patient may not be allocated to the treatment group. Studies have consistently shown that many RCT subjects believe that they are certain to be in the treatment group, not understanding the difference between research treatment (therapeutic misconception).13
In the United States, between 1979 and 1993 emergency research involving critically ill patients who could not reasonably consent was often considered to be performed under a concept known as waived consent. However, in 1993 the branches of the Department of Health and Human Services (DHHS) charged with review and development of policy relative to clinical research found that there was no clear guidelines by which investigators could safely and ethically evoke waiver of consent for emergency research. This led to an ensuing debate and a 1993 federal moratorium on all research without prospective informed consent. Due to disagreement concerning the specific intent of components of the Common Rule, in 1995 the Final Rule was introduced and was intended as a mechanism investigator to establish enrollment parameters for emergency research and exception from informed consent in place of the previous waiver method.
EXCEPTION FROM INFORMED CONSENT
Final Rule, 61 Fed. Reg. 51498 (Oct. 2, 1996) 21 CFR 50.24 defined the new standard for this process. The new 1996 standard for emergency research involves the investigator(s), study sponsors, IRB, and also the general public.
In order to be considered for eligibility for granting of exception from informed consent (EFIC) by the IRB a specific set of eligibility criteria must be met.14 The human subjects are in a life-threatening situation, available treatments are unproven or unsatisfactory, and the collection of valid scientific evidence is necessary to determine the safety and effectiveness of particular interventions.
Obtaining informed consent is not feasible because:
The subjects will not be able to give their informed consent as a result of their medical condition
The intervention under investigation must be administered before consent from the subjects' legally authorized representatives is feasible
There is no reasonable way to identify prospectively the individuals likely to become eligible for participation in the clinical investigation
Participation in the research holds out the prospect of direct benefit to the subjects because:
Subjects are facing a life-threatening situation that necessitates intervention
Appropriate animal and other preclinical studies have been conducted, and the information derived from those studies and related evidence support the potential for the intervention to provide a direct benefit to the individual subjects
Risks associated with the investigation are reasonable in relation to what is known about the medical condition of the potential class of subjects, the risks and benefits of standard therapy, if any, and what is known about the risks and benefits of the proposed intervention or activity
The clinical investigation could not practicably be carried out without the waiver.
The proposed investigational plan defines the length of the potential therapeutic window based on scientific evidence, and the investigator has committed to attempting to contact a legally authorized representative for each subject within that window of time and, if feasible, to asking the legally authorized representative contacted for consent within that window rather than proceeding without consent.
The IRB has reviewed and approved informed consent procedures and an informed consent document consistent with 50.25.
The process requires five major components:
Community consultation: There must be consultation with the leaders of the community about the research design and plans.
Public disclosure: The investigators (and sponsor) must inform the community about the research.
Multiple consent process: Investigators must prepare and utilize informed consent: a standard informed consent for the participant, a form for the family or legally authorized representative (proxy informed consent), and a consent to continue in the research.
Independent data monitoring: The study data must be monitored independently to ensure ethical practices relative to early termination in cases of harm (or poor risk/benefit ratio) based on interval results.
Community reporting: Investigators are also required to inform the community of the study's results and the demographic characteristics of the participants.
TYPES OF SCIENTIFIC ABSTRACTS
There are two basic types of abstracts:
Informational abstracts: These are short abstracts that contain an introduction or purpose, methods, scope, results, conclusions, and recommendations.
Descriptive abstracts: These are also short abstracts, sometimes less than 100 words. They contain a purpose, methods, scope. They do not contain results, conclusions, or recommendations.
High-quality abstracts are effective and succinct. It is helpful to reread your abstract several times, focusing on the main components, individually and as part of the whole document. Work to remove extraneous information and wordiness. Analyze the work for errors in grammar and mechanics. Have a scientific mentor review your work, correcting weaknesses in organization and coherence.
Scientific abstracts can be presented in a number of ways. Typically preliminary results or the initial report of a study result will take the form of a poster presentation of an oral abstract presentation. Peer-reviewed manuscripts are the backbone of the scientific literature and require more stringent review and presentation of the study results and therefore take more preparation and work to submit.
These are usually 15- to 30-minute presentations containing the background, methods, results, discussion, conclusions, and areas for future research. Afterward, time is allotted for questions and discussions with the audience. Important points to remember for an oral presentation are to have something important to say, in a meaningful and easy-to-understand manner. First impressions are important. The message is important. However, body language and vocal tone are the qualities your audience evaluates first. The best oral presenters appear earnest, competent, trustworthy, and do not seek to belittle or humiliate others. Allow the audience to interact with you in a friendly and congenial manner. Slides should be simple (5-7 lines per slide), of a readable sized font with bullets (not sentences). Stick to one key point per slide. Use color only to gain attention (avoid overuse of red).
A poster is the visual representation of scientific work. It is usually 48 in wide × 36 in tall and attached to poster boards. The submitting presenter is expected to be alongside the poster to discuss its contents at specific times when the judges will be reviewing the submitted works. The top of the poster contains the title, authors and affiliations, and an institutional logo (ensure that it is approved for use). The title should be legible from across the room (4 cm high). The text should be in bullets and contain an introduction, material and methods, results, discussion, conclusions, references, and acknowledgments. The results are best represented in tables, charts, or graphs. These tables or graphs should have explanations and be able to standalone in the information they impart. Informative pictures are desired. Vivid contrasting colors may be beneficial in this setting (as opposed to their use in oral presentations). Poster handouts, typically an 8 in × 11 in copy of the presentation with contact information should be on hand. It is as important to practice poster presentations as it is to practice oral presentations. There is a tendency to under-review a poster presentation, as the information is readily available for reference. However, fluid flow and smooth delivery are important components of a well-presented poster, and cannot be extemporaneously produced.
Abstracts may also be submitted as part of scientific manuscript submission to a medical journal. Every journal has its own set of author instructions. However, some rules are fairly ubiquitous:
Journal's aim and scope: Each journal has its own area of interest and expertise. They desire manuscripts that are technically sound and of interests to the specialists within their field. Additionally, the work must not have been published or submitted elsewhere.
Journal format/author's instructions: This must be adhered to stringently. Most journals will have a typeset maximum for pages, usually double-spaced with guidelines for tables and graphs and a word count. The abstract will usually also have a word count. These limitations are to be strictly followed. The journal will also dictate the exact format for the presentation: abstract, introduction, background, material, methods, statistics, results, discussion, and references. Many journals limit the number of references and disallow footnotes.
Electronic submissions: Some journals have electronic submissions that do not allow for fancy fonts, bold or italic lettering, and many have only online submission systems. Journals may also specify that the text be submitted only as .doc or .docx documents. Similarly, figures may only be accepted only in jpeg form. Ensure compliance with all instructions, check for typos numerous times, and submit before the deadline.
EMS SYSTEM DATA COLLECTION
The Utstein-style template is a standardized data set and form of reporting that allows for the comparison of different studies and their results as they relate to cardiac arrest. The latest version was updated by the International Liaison Committee on Resuscitation (ILCOR) in 2004 and the reporting template (for use by study investigators) is shown in Figure 10-2.15 It may be reasonable to expect EMS and event medicine (mass gathering) study investigators to also report cardiac arrest related data using this format.
Utstein template for reporting cardiac arrest data. Utstein reporting template for core data elements. ED, emergency department; OR, operating room; CCU/ICU, critical care unit/intensive care unit; and PEA, pulseless electrical activity.
(NHTSA) UNIFORM PREHOSPITAL EMS DATA SET
In 1993, the National Highway Traffic Safety Administration released their Uniform Prehospital EMS Data Set. The data set contains data elements that have been determined to be essential components of the patient care report that lead to database population in many states and at the national level. The purpose of the uniform (standardized) data set is to allow local, regional, state, and national EMS organizations and agencies to perform analysis on key system indicators and allows benchmark comparisons to be made across systems and throughout published EMS research studies. The current version of the data set is version 3.3.1 although many systems are still currently using a version 2 build. The data are expected to be collected and incorporated into electronic databases in all participating states. The states (and territories) then upload data to NEMSIS. NEMSIS is the National EMS Information system and currently 90% of states and territories are participating (http://www.nemsis.org). The goals of NEMSIS are to allow “reporting capabilities, allowing Federal, State and Local EMS stakeholders access to performance and benchmarking metrics.” NEMSIS has obvious research value as well.
NATIONAL TRAUMA DATA BANK
The National Trauma Data Bank (NTDB) is associated with the American College of Surgeons and comprises a large network of participating hospitals that maintain a trauma database (usual designated trauma centers). The NTDB utilizes a standardized data dictionary and contains many data fields including some prehospital data and ultimately outcome data. This is another excellent source for EMS quality improvement and research investigation. Each participating hospital maintains the ability to perform studies with their own data set.
CRASH OUTCOME DATA EVALUATION SYSTEM
Each state has members of the nationwide database designed to bring further clarity to the causes and outcomes of motor vehicle crashes. Crash Outcome Data Evaluation System (CODES) is also an administrative and cost-benefit database including crash data with special attention to “type, severity and cost in relation to the characteristics of the crash, vehicles, and persons involved.” Researchers with special focus and interest in injury prevention may find utilizing these type of data in conjunction with other data may lead to greater in-depth investigation and more meaningful conclusions concerning motor vehicle-related injuries and death.
Other databases that may provide some data of interest to EMS researchers include the Hospital Available Beds for Emergencies and Disaster (HAvBED) system, the Emergency System for Advanced Registration of Volunteer Health Professionals (ESAR-VHP), and the Health Alert Network (HAN) maintained by the Centers for Disease Control.
Much advancement and evolution has occurred in the field of EMS medicine due to increasing development of research based in the prehospital arena. Failure to maintain vigilance in the design and implementation of prehospital EMS research investigations will lead to improper information on the effectiveness and appropriateness of current and future EMS practices. It is essential to utilize proper ethical procedures in compliance with established standards and to utilize standardized data systems when possible. Failure to implement these tools will lead to a failure to advance EMS medicine as an evidence-based field of medical practice. EMS physicians and medical directors should utilize available science to gauge and modify their practice when appropriate.
EMS research design and interpretation require understanding of key components and statistics.
Emergency research utilizing exception from informed consent (EFIC) requires adherence to specific qualifying attributes and adherence to specific procedural guidelines.
EMS data should be recorded and reported using standardized/uniform formats to allow for research result comparisons and system benchmarking.
Building a research team with incorporation of education, mentorship, and collaboration is a key component to successfully engaging in EMS research.
et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials”. Br Med J. 2010;340:c869.
M. Interim analyses, stopping rules and data monitoring in clinical trials in Europe. Stat Med
. March 1993;12(5-6):509–520.
et al. Publication bias and clinical trials. Control Clin Trials
J. The ethics of randomised controlled trials from the perspectives of patients, the public, and healthcare professionals. Br Med J. 1998;317(7167):1209–1212.
BP. An overview of outcomes research and measurement. J Healthc Qual
. November-December 1999;21(6):4–9.
D; CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials.
. 2010 Mar 23;340:c332.
et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ
SD; National EMS Agenda Writing Team. National EMS Research Agenda.
Prehosp Emerg Care
. July-September;6(3 suppl):S1–S43. Review.
S. Collaboration and Team Science: A Field Guide. Bethesda, MD: National Institutes of Health. August 2010, NIH Publication, no. 10-7660.
et al. Designing clinical research. 3rd ed. Philadelphia, PA): Lippincott Williams and Wilkins; 2007.
C. The therapeutic misconception: informed consent in psychiatric research. Int J Law Psychiatry
U.S. Department of Health and Human Services, Food and Drug Administration, Office of Good Clinical Practice, Center for Drug Evaluation and Research, Center for Biologics Evaluation and Research, Center for Devices and Radiological Health. Guidance for institutional review boards, clinical investigators, and sponsors exception from informed consent requirements for emergency research. March 2011. Updated April 2013. http://www.fda.gov/downloads/RegulatoryInformation/Guidances/UCM249673.pdf
. Accessed November 1, 2013.
ILCOR Task Force on Cardiac Arrest and Cardiopulmonary Resuscitation Outcomes. Cardiac arrest and cardiopulmonary resuscitation outcome reports: update and simplification of the Utstein templates for resuscitation registries: a statement for healthcare professionals from a task force of the International Liaison Committee on Resuscitation. Circulation. Nevember 23, 2004;110(21):3385-3397.