Extracted from the NAEMSP Research Course
The Research Idea
Research can be inspired by: observations made; frustration felt when confronted with a difficult problem; gaps in the medical literature; or discussions with colleagues.
The research idea is developed into a research question, which leads to the development & execution of a formal research study.
When developing an idea consider capitalizing on unique areas of expertise, your own special interests, departmental commitment to an area of investigation, or strong liaisons you may have with other investigators.
____________________________________________________ (your idea goes here)
A good research question will be: Feasible, Interesting, Novel, Ethical, and Relevant...
Test your idea:
Feasible?
Interesting?
Novel?
Ethical?
Relevant?
REMEMBER...The study design you will use depends on the question being asked...
Reviewing The Literature
Once the research question has been formulated, a review of the literature is in order. If done well, you will soon become an expert (or at least a quasi-expert) on the topic you’re researching.
Reviewing literature pertinent to your question serves several purposes.
· You will know what has been published, and by whom.
· You will be able to review the methods used and the conclusions formulated.
· You will have at your fingertips, access to information about the strengths and weaknesses of the studies, and occasionally the authors opinions of what requires further study and why.
A thorough review of the literature will also provide you with good background information to use in writing your protocol and later your manuscript. Be relentless in your documentation. Keep a detailed and accurate bibliography and organize the copies of your articles accordingly. This will save you hours of work later.
Sources:
http://www.ncbi.nlm.nih. PubMed (National Center for Biotechnology Information. National Library of Medicine, National Institutes Of Health.)
http://www.nhtsa.dot.gov National Highway Traffic Safety Administration
http://www/amhrt.org American Heart Association
Select articles germane to your topic and not more than 10 years old (unless they’re classics). Review the study structure independent of the content. Look at the study design, methodology, & sample selections. What methodology or parts thereof might be applicable to your developing study? Review the references for other major pertinent articles you may have missed in your initial search.
List two major references pertinent to your research question below. Include a fact or two that you can use to support your research question.
Selecting A Research Design
The double-blind, randomized controlled trial is considered the “gold standard” of research design in biomedical research. This design is superior to all others because it controls investigator bias, sampling bias and other (sometimes unknown) threats to internal validity. However, the double-blind, randomized, controlled trial is rarely used in EMS research, usually because it is difficult to implement studies in the field according to these criteria.
Alternative designs are often chosen for EMS research; a compromise-if you will-between what is feasible in the field, and what is the most scientifically sound. EMS research largely focuses on investigating patients, providers or methods of providing care. Creativity in designing a field study can make a seemingly impossible project successful. Be prepared to step back from your idea and look at it from different points of view. Talk to others who have an interest in the question at hand. Don’t be afraid to give up an idea that will not provide you with results you need.
See handout on Study Design Categorization for a list of designs and their strengths/weaknesses.
Questions to Consider When Selecting A Research Design
- Is the study prospective in nature or retrospective?
- Who are the subjects or what is the treatment to be studied
- Would a cross-over of treatments be feasible?
- Is randomization or blinding possible?
Patient Selection: Inclusion and Exclusion Criteria:
These criteria will define your “patient” population by specifying which patients are eligible to be included in your study. Likewise, specify explicitly who should be excluded from your study, and why.
List below any Inclusion and Exclusion Criteria applicable to your question.
BASIC STATISTICAL TERMS
Perhaps the easiest area of statistics to understand involves the use of numbers to describe the study populations-descriptive statistics. These statistics are easily understood as graphs, and their computations are often presented as the familiar mean, median, mode, and the standard deviation.
MEAN The mathematical average as determined by dividing the sum of the observations by the number of observations made is the mean. For example, the mean of 9, 9, 8, 4, 4, 3, 3, 3, and 11 would be 54 divided by 9 = 6.
MEDIAN The observation above which half the points lie & below which are the remaining observations is defined as the median. In the above series the median would be 4 since 11, 9, 9, and 8 are all above 4 and 4, 3, 3, and 3 are all equal or below 4.
MODE The most common observation in a series is termed the mode. For example, in the above series the mode would be 3 since it occurs more frequently than any other number.
STANDARD DEVIATION (SD) Standard deviation is a number reflective of the degree of variability of data points in relationship to the mean. One standard deviation (1 SD) encompasses about 68% of the sample range of values, while 2 SD encompasses about 95%; 3 SD encompasses over 99%. Thus, if a mean of a sample is 20 and the SD is 3, then about 68% of the sample should have values between 20 + 3 and 20 - 3 or between 23 and 17; 95% would be between 20 + 6 and 20 - 6 or between 14 and 26. The SD is written with both a plus (+) and a minus (-) sign in front of it (±) to indicate it is establishing a range about the mean.
STANDARD ERROR OF THE MEAN (SEM) The standard error of the mean is a calculation performed to estimate the mean of the entire population from which the sample was obtained.
- Standard deviation, in distinction, is the measure of the variability of data points about the mean of a sample. The terms, although similar sounding, refer to entirely different populations.
- The standard error of the mean (SEM) is derived by dividing the standard deviation (SD) by the square root of the number of sample observations. If a study determined that the SEM of theophylline levels was 10mg plus or minus 2mg, it would indicate that, if other samples were taken from patients with the same sample characteristics, their mean would fall between 8mg and 12mg 68% of the time. Because of the mathematical formula by which SEM is derived, it will always be smaller than the SD.
Statistics
Statistics are useful for describing study samples or populations (descriptive statistics), or to help us draw conclusions about populations based on the occurrence of events in a small group (statistical inference).
Descriptive Statistics
These help distill large amounts of information to essential features (e.g., statistics on the back of baseball cards). Descriptive statistics are reported as a pair of numbers: one summarizing central tendency (mean, median) and one summarizing dispersion or "spread" (standard deviation, confidence interval).
Inferential Statistics
The second step in some, but not all, studies is to infer the likelihood that the observed results can be generalized to other samples of individuals.
To show that barbecue sauce is an effective treatment for arthritis or that IQ is related to hair color, we are attempting to make a general statement that goes beyond the particular individuals studied.
The rub is that differences between groups can rarely be attributed simply to the experimental intervention: some people in the barbecue sauce group may get worse, and some people in the placebo group may get better. The goal of inferential statistics is to determine the likelihood that these differences could have occurred by chance.
Types of Experimental Data
The choice of statistical test is dictated by the type of data collected and the general study design. There are essentially three types of data:
Nominal: the weakest level of measurement in which objects are placed into named categories. Our clam juice versus placebo is one such variable, as is the gender of the patient, or the diagnosis of a group of patients. Clinical research is often concerned with determining the presence or absence of risk factors, assessing the presence or absence of particular diseases, and estimating survival (the presence or absence of death). These measures are all nominal categories, and the numbers are body counts (e.g., the number of people with angina, treated with a beta-blocker, who survived 1 year). These kinds of data require non-parametric statistics for analysis. The non-parametric test used for nominal data is the chi-square test.
Ordinal: A somewhat stronger level of measurement than nominal data. Items are still placed into mutually exclusive groups; however, there is a definite order or ranking to these groups. A common example in the medical literature is the subjective judgment of disease staging in cancer, using categories such as stage I, II, or III. Although we can safely say that stage II is worse than stage I, and better than stage III, we don't really know by how much. Another example--no device exists to measure the amount of pain experienced by a patient. An investigator evaluating a new analgesic might ask his patients whether their pain is absent, mild, moderate, or severe. The most common statistical tests for analyzing ordinal are the Mann-Whitney U test and the Wilcoxon signed rank test.
Interval: The highest level of measurement that can be achieved, and they possess truly quantitative characteristics. Like the ordinal scale, there is a definite sense of order to interval data. Additionally, the distance between consecutive numbers on the scale is constant, the numbers have a defined unit of measurement, and they represent real numbers. Examples include such items as blood pressure measured in MM Hg, temperature measured in °C, serum glucose level, or white blood count.
By far the most common interval statistical test is the Student's t-test. The t-test was developed by W.S. Gosset, who wrote many papers under the pseudonym "Student". He was not a professional mathematician (his work was, in fact, in the Guinness's brewery in Dublin) but his hobby was statistics (this was before TV). The t-test depends on a comparison of means (and SD) from two groups.
Preparing A Protocol
There are many ways a protocol can be put together. No one method is right or wrong. It’s important to follow any instructions or use any templates that may be required at your individual institution. If you are applying for Grant Funds, follow the instructions exactly.
TITLE
Important, but not necessary during protocol development. Keep options in mind as you work.
INTRODUCTION/SCIENTIFIC BACKGROUND
Information from the review of the literature can be incorporated (and referenced) into the introduction/background/review of literature section of your protocol. You should build a structure that describes the rationale for what you will be doing. Keep the writing focused on the problem to be studied. Using references as evidence for the importance of the question you are asking. End with a statement of purpose or your major objective.
OBJECTIVES
List objectives clearly. Make sure all objectives are measurable and that their measurement is feasible, and as objective (vs. subjective) as possible. Be very clear about identifying your major outcome objective. Limit secondary objectives to one or two.
STUDY DESIGN AND METHODOLOLGY
Describe the study design and any methods you will use to strengthen the study (blinding, randomization). Don’t get hung up on terminology.
TREATMENT/INTERVENTION
Give a detailed statement of the treatment or intervention. Describe what will happen from the beginning of the intervention until the end.
MEASUREMENTS
What are your main measurements and how will they be evaluated? Are there rescue medications involved? What are the criteria for stopping the study?
How Many Subjects?
Once its been decided who and what you will study, the next question is “How Many?” The best study designed may fail to answer the research question if the sample size is too small. The goal is to estimate the number of subjects your study will need to answer the question without doing more work than you have to.
Sample Size Estimates
- Are “estimates” - They’re only as good as the information upon which they are based.
- Sample size planning is a scientific way of making a ballpark estimate.
- Should be done as a part of the planning process when changes can still be incorporated into your plan. (Never after the study is completed)
- Formulas used to calculate sample size are based on mathematical assumptions that differ for each test. You must know how you plan to analyze the data (what tests you will use) before you can calculate sample size.
Understanding sample size calculations: terminology
EFFECT SIZE
Set by the investigator. This is the size of the association you would like to be able to detect in your sample. Selecting an appropriate effect size is the most difficult aspect of sample size planning. It is possible to use information from other studies or from pilot data to “guess” about effect size. Some investigators will estimate the smallest difference that would be clinically meaningful.
ALPHA
The probability of making a Type I error (also called the level of statistical significance)
BETA
The probability of making a Type II error
POWER
The probability of observing an effect in the sample if one of a specified effect size or greater exists.
How Many Subjects?
Supply The Following Information Needed To Calculate An Estimated Sample Size
Which statistical test do you plan to use to evaluate your main outcome objective? (The type of data you collect determines the test used)
What is the size of the effect you anticipate in this outcome objective?
Power is conventionally set at 0.80. Do you anticipate that you will need to demonstrate a power greater than 80%? Yes No
Alpha (the level of significance) is generally set at 0.05 and is “two-tailed”.
A two-tailed hypothesis states only that an association exists; it does not specify the direction. Tails refer to the tail ends of the normal distribution (The Bell Shaped Curve). One tail represents a positive effect or association, and the other a negative effect.
List the alpha value you require, ______or the default 0.05 and state the anticipated direction of your results _____________. (Unknown is OK, use a two-tailed test)
Additional Related Concepts
- P-VALUE: The probability of obtaining the study results by chance if the null hypothesis is true. (Significant results: P-values less than alpha. Non-significant results: P values greater than alpha). P- values are calculated after the study is complete.
- TYPE I ERROR: Rejecting the null hypothesis (there is NO difference...) when it is actually true. (false positive)
- TYPE II ERROR: Not rejecting the null hypothesis when it is actually FALSE (false negative)
Convention Dictates The Following “Default” Values
Alpha: 0.05 (5% chance of incorrectly rejecting the null hypothesis)
Beta: 0.20 (A 20% chance of missing an association of a given effect size if it truly exists) Corresponds to a Power of .80 (1- Beta)
Power: .80 (80% chance of detecting the specified difference, IF that difference exists.)