Quantitative
Research Design
As
indicated above, quantitative research focuses on analysis of numeric data. The
approach often follows particular scientific methods (e.g., design, sampling,
measurement). Quantitative research can be classified into three types shown in
Table 4 (Trochim, 2006).
Table
4. Quantitative Research DesignsResearch Design Non-Experiment Quasi-Experiment Experiment
Random
Assignment of Subjects to Group No No Yes
Control
Group or Multiple Waves of Measurement No Yes Yes
Non-experimental
designs
Non-experimental
designs do not involve random assignment of subjects to groups, nor is there a
control or comparison group. Non-experimental designs also do not involve
multiple waves of measurement. This type of design is very useful for
descriptive research questions such as:
What
percentage of students is involved in community service?
Do male
students have different attitudes than females about the need for social
service agencies?
How
many faculty members have taught a service-learning course in the past three
years?
The
simplest, very common form of non-experiment is a one-shot survey. For example,
a researcher might conduct a survey of opinions about community activism. In a
variation on this, a researcher might measure attitudes at the end of a
semester in a service-learning course. This design (called the post-test only,
single group design, Campbell & Stanley, 1963) lacks a comparison, and therefore
the ability to conclude that the outcome was the result of the servicelearning
experience.
Correlational
research designs evaluate the nature and degree of association between two
naturally occurring variables. The correlation coefficient is a statistical
summary of the nature of the inferred association between two constructs that
have been operationalized as variables. The correlation coefficient contains
two pieces of information (a) a number, which summarizes the degree to which
the two variables are linearly associated; and (b) a sign, which summarizes the
nature or direction of the relationship. The numeric value of a correlation
coefficient can range from +1.0 to -1.0. Larger absolute values indicate
greater linear association; numbers close to zero indicate no linear
relationships. A positive sign indicates that higher values on one variable are
associated with higher values on the other variable; a negative sign indicates
an inverse relationship between the variables such that higher values on one
variable are associated with lower values on the other variable.
Causal
inferences are very difficult to make from a single correlation because the
correlation does not assist in determining the direction of causality. For
example, a positive correlation between volunteering and self-esteem indicates
that more volunteering is associated with higher self-esteem. However, the
correlation does not differentiate among at least three possibilities, (a) that
volunteering affects self-esteem; (b) that self-esteem promotes volunteering;
or (c) that a third variable (e.g., self-efficacy) is responsible for the
correlation between selfesteem and volunteering.
Experimental
designs
In
contrast to correlational methods that assess the patterns between naturally
occurring variables, experiments manipulate a variable, the independent
variable, and see what consequence that manipulation has on another variable,
the dependent variable. Not all experimental designs are equally good at
allowing the researcher to make causal inferences. An outline of experimental
designs is presented below. Note that this section is only intended to be an
introduction to the topic. For more specific information on experimental
research design, the reader should consult a research methodology text (e.g.,
Campbell & Stanley, 1966; Cook & Campbell, 1979; Cozby, 2009;
Kerlinger, 1986). Consultation with experienced research colleagues is also
helpful. Some online resources on design are listed in the appendices of this
document.
The
strongest research design in terms of drawing cause-and-effect conclusions
(internal validity) is the randomized or true experiment. In this "gold
standard" of quantitative research designs, subjects are randomly assigned
to different groups or treatments in the study. Traditionally these groups of
subjects are referred to as the experimental or treatment group(s) (e.g.,
students in a service-learning course) and the comparison or control group(s)
(e.g., students in a traditional course). Note that random assignment of
subjects to a group in an experiment is different from the random selection of
subjects to be involved in the study. Random assignment makes it unlikely that
the treatment and control groups differ significantly at the beginning of a
study on any relevant variable, and increases the likelihood that differences
on the dependent variable result from differences on the independent variable
(treated group vs. control group). Random assignment controls for
self-selection and pre-existing differences between groups; random selection or
sampling is relevant to the generalizability or external validity of the
research.
There
are a variety of designs that utilize random assignment of subjects, but true
experimental studies are relatively rare in service-learning research, as in
most educational research. This is because it is usually difficult, especially
in higher education settings, to randomly assign students to service-learning
versus traditional courses or to different levels of a variable in the
instruction. Nevertheless, the U.S. Department of Education has proposed that
all research use random assignment so that education practice can be based on
research with internal validity
(www2.ed.gov/rschstat/eval/resources/randomqa.html). A close approximation of
random assignment occurs when students are not aware that some sections of a
course will be service-learning and some will not be service-learning when they
register for courses (Markus, Howard, & King, 1993; Osborne et al., 1998).
Also, there may be opportunities to randomly assign students to different
conditions in service-learning classes (e.g., students are randomly assigned to
(a) written reflection or (b) reflection through group discussion).
Quasi-experimental
designs
Like
experimental designs, quasi-experimental designs involve the manipulation of an
independent variable to examine the consequence of that variable on another
(dependent) variable. The key difference between experimental and
quasi-experimental designs is that the latter do not involve random assignment
of subjects to groups. A large portion of past quantitative research on
service-learning involves quasi-experimental design. We do not intend to
comprehensively cover all quasi-experimental designs in this primer; instead we
will discuss some designs commonly seen in service-learning research. For more
advanced information, or for information on other designs not discussed here,
we recommend that the reader consult a graduate-level research methodology text
(e.g., Campbell & Stanley, 1966; Cook & Campbell, 1979; Cozby, 2009;
Kerlinger, 1986). Consultation with experienced research colleagues is also
helpful. In addition, some online resources are listed in the appendices of
this document.
One
aspect of designing a study relates to temporal arrangements. Some researchers
are interested in the developmental aspects of service-learning, or in the
effects of service-learning over time. For example, they may be interested in
the question of whether involvement in volunteer service during high school
leads to increased involvement in service during and after college. There are
two approaches to designing research to answer these types of questions. In a
cross-sectional design the researcher gathers data from several different
groups of subjects at approximately the same point in time. For example, a
researcher might choose to conduct interviews with groups of college freshmen,
juniors, graduating seniors, and alumni. Longitudinal studies (sometimes also
called time series designs) involve gathering information about one group of
people at several different points in time. Astin, Sax, and Avalos (1999), for
example, collected survey data from entering freshmen in 1985, then surveyed
the same group of students four years later in 1989, and again to the
now-alumni in 1994-95. Longitudinal studies are extremely valuable sources of
information for studying long-term consequences of servicelearning, but they
are rare in service-learning research because of the practical, technical, and
financial difficulties in following a group of people
over time.
Other
researchers focus their interest on questions that do not relate to
developmental issues or impact over a long period of time. In fact, many if not
most service-learning studies are limited to one semester or sometimes one year
in length. A common strategy is to give an attitude measure to students in a
service-learning course at the beginning and end of a semester. This pre-test,
post-test single group design examines the difference between pre- and
post-test scores for one group of students. Unfortunately, there is no
assurance that the difference in pretest and post-test scores is due to what
took place in the service-learning class. The difference in attitudes could be
attributable to other events in the students' lives (history), natural growth
changes (maturation), dropout of the least motivated students during the course
(mortality), or carryover effects from the pre-test to the post-test (testing).
Another
experimental design is the post-test only, static groups design1, which
compares the outcomes of a pre-existing treated group to the outcomes of a
pre-existing untreated group. Using this design, an instructor could give an
attitude scale at the end of the semester to a service-learning section of the
course and also to a section that did not contain service-learning. This design
suffers from the limitation that it is not possible to conclude that the
difference on the dependent variable, attitudes, is due to the difference in
instruction because it is not known if the two groups were equivalent in their
attitudes at the beginning of the semester.
An
alternative arrangement, the nonequivalent (or untreated) control group design
with pre- and post-test, is to give a pre-test and a post-test to both a
service-learning section of a course and to a traditional section that does not
include a service component. In this design the researcher can evaluate whether
or not the two groups were equivalent at the beginning of the semester, but
only on the measured variables. A second step is to examine the pattern of
changes between the two groups across the semester.
The
biggest problem with the nonequivalent groups design is self-selection bias,
described above in the section "Common Problems in Service-Learning
Research." Frequently in higher education, and sometimes in high school
settings, service-learning courses are optional for graduation, and/or service
is an optional component of a particular course. That is, students must select
or opt to be in the class and to participate in service. The result is that
students are nonrandomly assigned to the treatment group (service-learning
course) and thus there is non-random assignment of students to groups. There
are likely to be many differences between students who choose to be involved in
service-learning classes and those who do not (Eyler& Giles, 1999). Even
with a pre-test to compare equivalence of groups at the beginning of the study,
a researcher could never completely eliminate the possibility that there are
differences on other, unpre-tested variables, or that post-test differences are
due to inherent differences in the groups, rather than differences in the
educational intervention. Sometimes researchers use multiple measures preand
post-treatment to help assess whether groups are equivalent on several relevant
variables; statistical procedures (i.e., analysis of covariance) also can help
control for differences between treatment and non-treatment groups, but only
for measures that are obtained prior to the educational intervention. Of
course, the best solution is random assignment of students to groups, which
makes this an experimental design, rather than a quasi-experimental one.
A
common variation of the nonequivalent groups design occurs when students in two
sections (one including a service component and one not) of a course are being
compared, but the two sections are taught by different instructors. This
creates a problem in interpretation because one cannot infer that post-test
differences in scores are due to the style of pedagogy (service-learning)
rather than other differences between instructors. Another variation is to
compare two sections of the same course, one involving service and one not, but
taught in different semesters. In this case it is possible that differences in
post-test scores are due to events extraneous to the study, which happened
during one semester but not the other. In sum, it is important for the
researcher to be aware of potential pitfalls of any research design and to take
these into account when drawing conclusions from the study.
