Nils F. McGee
The effects of using an audience response system in the classroom: A review of the literature
1. Introduction
1.1 Overview
Audience response systems (ARS) are a technology that allows students to responds to questions presented by the teacher in class, the teacher can display the distribution of class responses in a variety of different formats, usually a bar graph that anonymously displays the frequency of each possible response. These systems may alternately be referred to as personal response systems (PRS), classroom response systems (CRS), class performance systems (CPS), student response system (SRS), interactive classroom communications systems (ICCS), audience response technology (ARS), and response pad system (RPS). The technology consists of individual student response pads, a receiving unit, and a computer with the required software. The questions can be embedded within slide-show presentations, or as stand-alone questions. The response pads may be registered to a specific student in order to collect formative or summative data.
This review of the research on the use of an ARS in the classroom was performed in order in order to summarize the conclusions regarding the effect of ARSs on student learning, on student attitudes, on student behaviors, and to summarize the conclusions regarding best practice in the use of an ARS in the classroom.
1.2 History of use
Audience response systems have been used in academic settings since the 1960s (Judson & Sawada, 2002). At that time, they consisted of dials or keypads that would provide the students with feedback by buzzing or lighting up. The instructor received information from gauges that showed the percentage of students responding to each choice (Judson & Sawada, 2002). Better Education, Inc. began development of an ARS in 1985, and tested the system in 1988. This system, called Classtalk, became commercially available in 1994 (Kay & LeSage, 2009; Better Education, Inc.). Since the introduction of Classtalk, several other companies have developed ARSs. One website lists nineteen different vendors of ARSs (Audience Response Info, 2010). The most commonly used ARS in the literature reviewed were TurningPoint and Interwrite PRS.
There are many different reasons to use an ARS. The literature reviewed indicates that the most common use of an ARS is to promote student participation and interaction. They also indicated that feedback to students were provided using an ARS, that student participation, interactions, or active learning were promoted using an ARS, that formative assessment were given through an ARS. They also indicated that an ARS was for summative evaluation in terms of attendance, and quizzes that counted toward the students’ final grade.
2. Methods
A keyword search of ERIC (EBSCO), Educator’s Reference Complete (Gale Group), Education Full Text (Wilson), and Professional Development Collection (EBSCO) databases using the term “audience response system” returned 34 papers. Of these papers, 10 were not directly related to the use of audience response systems, three were literature reviews, two articles described the author’s perspective on best practice for the use of an ARS in the classroom The remaining 18 articles were included in this review. Two of the papers used a true experimental design, seven used a quasi-experimental design, six were descriptive, and three used a mixed (quasi-experimental and descriptive) design. Summaries of these articles are summarized in the appendix.
3. Results
3.1 Participation
Two studies reported the use of an ARS in the classroom increased the percentage of students responding to each question. One of these studies used a true-experimental design in which the participation rates under four different treatments were measured, either by the ARS, or by trained viewers who tallied participation rates (Stowell & Nelson, 2007). A second study used a quasi-experimental design. Surveys completed by instructors and tutors of students either used or did not use an ARS indicated that the ARS students had greater participation rates (Mula & Kavanagh, 2009).
3.2 Attendance
Large undergraduate lecture classes do not commonly track attendance. ARS can be used to keep a record of student attendance. Two studies report that the use of ARS for attendance purposes is positively correlated with student attendance. One of these studies was quasi-experimental. ARS was used for attendance purposes in one group, while ARS was not used for attendance in the other group (Edens, 2008). The other study was descriptive in nature, reporting student attitudes related to when an ARS system was used for attendance (MacGeorge, et al., 2008).
3.3 Attitudes
Student attitudes toward the use of clickers were found in relation three main topics. These included the use of an ARS for grading, the effectiveness of an ARS for learning, attitudes toward participation, and student enjoyment of class when an ARS was used.
3.3.1 Grading
Four studies investigated student attitudes toward the use of the ARS to give quizzes that counted toward their grades. Using a quasi-experimental design Edens (2008) measured student attitudes toward the use of an ARS under different pedagogies. The behaviorist group took ARS quizzes that made up 25% of their grade for the course. The metacognitive group took the same ARS quizzes, but their quizzes were used to provide formative assessment and feedback only. The students in the behaviorist group felt significantly more stressed than the metacognitive group. The second study reported that female students were significantly more negative about the use of ARS when it was used for grading compared to when ARS was used and results did not count toward their grade (Kay, 2009). The third study was descriptive. On a Likert-type rating scale for the question “I would have preferred if the voting (or some of the voting) was an assessable component of this subject”, 21 out of 145 students replied “strongly agree” or “agree” (Gauci, Dantas, Williams, & Kemm, 2009, p. 65). A fourth study reported that responses to a survey regarding the aspect of ARS they liked the least included many negative responses to the use of ARS quizzes for grading purposes (Graham, Tripp, Seawright, & Joeckel, 2007)
3.3.2 Effectiveness
Eleven studies measured student attitudes toward the use of an ARS in the classroom. One study used a true experimental design to test the perceptions of health professionals in continuing medical education presentations toward the use of ARS. This study found that the participants in the ARS group had a significantly higher rating of quality of the presentation and speaker and their attention to the presentation than the participants in the non-ARS group (Miller, Ashar, & Getz, 2003). Another study used a quasi-experimental design to test the attitudes of undergraduate psychology students toward the use of an ARS system in the classroom. This study found that participants in the ARS group rated the lecture as more effective, involving, and intellectually stimulating than the participants in the non-ARS group (Shaffer & Collura, 2009). The remaining nine studies used descriptive statistics to report student attitudes without comparison to non-ARS classes. All nine of these studies reported that students felt that the when an ARS was used in the classroom, it was an effective technique for promoting learning.
3.3.3 Participation
Five of the studies measured student attitudes toward participation or perceptions of classroom interactions in the context of an ARS (Barnes, 2008; Mun, Hew, & Cheung, 2009; Shaffer & Collura, 2009; Sprague & Dahl, 2010; Graham, et al., 2007). All of these studies report that the use of an ARS system resulted in positive attitudes toward participation. Two of the studies (Sprague & Dahl, 2010; Graham, et al., 2007) reported that the use of an ARS increased the participation of students who were identified as less likely to participate than their peers.
3.3.4 Enjoyment
Seven studies reported that students enjoyed classes when an ARS was used. All of these studies were descriptive in nature. Since there is no comparison, it is hard to tell if the students enjoyed classes that use an ARS more than a class that does not.
3.4 Performance outcomes
Eleven studies reported on the relationship between the use of an ARS and student performance. Four studies reported that the use of an ARS did not significantly increase student performance. Of these four studies, two were true experimental design, and two were quasi-experimental designs. In one of the studies that used a true-experimental design, volunteers were solicited to participate in a research study. They were randomly assigned to four different treatment groups. After the lecture, a post-lecture quiz was taken.
The remaining seven studies were all quasi-experimental designs and they all reported that the use of an ARS did significantly increase student performance. The most compelling of these studies used a non-equivalent groups model with a pre-test/post-test design to test the effect of an ARS on learning secondary school physics concepts in Singapore. The results showed a significant improvement at the 0.05 level of significance with an effect size (Cohen’s d) of 0.851 (Mun, et al., 2009), which is considered to be large. A second study compared the post-test results of students in an ARS class to students in a non-ARS class. Both treatment groups were presented with the same questions in class. The results of the post-test showed no difference in the scores between the ARS group and the non-ARS group however, the scores of the ARS group were significantly greater for different, yet related questions (Mayer, et al., 2009)
3.5 Pedagogical issues
Two studies examined how different groups of students responded to the use of an ARS in the classroom. One study found that students had significantly lower enjoyment and significantly higher stress when the ARS quizzes counted toward their grade, even though this treatment did not affect their performance on summative evaluations (Edens, 2008). This was also found to be true in a different study that compared attitudes of students toward the use of an ARS in classes that used the system regularly for grading to those in classes that did not use the system for grading (Graham, et al., 2007). Two studies also found that male attitudes toward the use of an ARS system were significantly more positive than female attitudes (Edens, 2008; Kay, 2009). Two studies also reported that the use of an ARS in the classroom has a larger positive correlation with the attitudes and behaviors toward participation of students that have been identified as less likely to participate in a regular classroom setting (Graham, et al., 2007; Sprague & Dahl, 2010).
3.6 Subjects of the studies
The research on ARS was primarily focused on large lecture classrooms where traditional lectures do not provide much opportunity for participation and interaction. Fifteen of the studies were performed at the undergraduate level, usually on large introductory lecture classes in the sciences. Five of the studies focused on the effects of an ARS at the secondary level. Four of these studies were conducted in the sciences.
4. Discussion
Audience response systems have a positive impact when used in the classroom. Students enjoyed the classes more, felt like the classes were more effective, reluctant participants participated more and students performed better.
More quasi-experimental studies have to be performed where there are methods used to eliminate multiple-group threats to internal validity that are inherent in a quasi-experimental design. Quasi-experimental designs are more susceptible to threats to internal validity than true-experimental designs because the effects of confounding variables are difficult to account for, but they are less susceptible to threats to external validity than true-experimental designs because the conditions under which the data are collected are usually the real condition being studied. Very few of the quasi-experimental designs attempted to determine if there were other differences in the groups they were studying that may have contributed to the effect they were reporting. Results from the true-experimental designs are also suspect because they were not collected under actual class conditions.
Since most of the research designed around the potential of the ARS to improve participation in large lecture classes, more research needs to be designed around the potential of ARS to improve the quality of participation, feedback, and interactions in classrooms regardless of size. The findings would be more generalizable to secondary school classes as well as smaller undergraduate classes. Research should also be expanded to other subjects.
Bibliography
Audience Response Info. (2010). Audience Response Vendors. Retrieved July 24, 2010, from Audience Response Info: http://www.audienceresponseinfo.com/audience-response-vendors/
Barnes, L. J. (2008). Lecture-free high school biology using an audience response system. The American Biology Teacher, 70(9), 531-536.
Bertrand, P. (2009). Using a classroom response system to improve multiple-choice performanc in AP Physics. The Physics Teacher, 47, 216-219.
Better Education, Inc. (n.d.). History of Classtalk System at Better Education. Retrieved July 24, 2010, from Better Education-Classtalk, The First Interactive Classroom Communication System: http://www.bedu.com/history.html#aboutbe
Conoley, J. W., Croom, D. B., Moore, G. E., & Flowers, J. L. (2007). Using electronic audience response systems in high school agriscience courses. Journal of Agricultural Education, 48(3), 67-77.
Deleo, P. A., Eichenholz, S., & Sosin, A. A. (2009). Bridging the information literacy gap with clickers. The Journal of Academic Librarianship, 35(5), 438-444.
Edens, K. M. (2008). The interaction of pedagogical approach, gender, self-regulation, and goal orientation using student response system technology. Journal of Research on Technology in Education, 41(2), 161-177.
Gauci, S. A., Dantas, A. M., Williams, D. A., & Kemm, R. E. (2009). Promoting student-centered active learning in lectures with a personal response system. Advances in Physiology Education, 33, 60-71.
Graham, C. R., Tripp, T. R., Seawright, L., & Joeckel, G. (2007). Empowering or compelling reluctant participators using audience response systems. Active Learning in Higher Education, 8(3), 233-258.
Griff, E. R., & Matter, S. F. (2008). Early identification of at-risk students using a personal response system. British Journal of Educational Technology, 39(6), 1124-1130.
Judson, E., & Sawada, D. (2002). Learning from Past and Present: Electronic response systems in college lecture halls. Journal of Computers in Mathematics and Science Teaching, 21(2), 167-181.
Kay, R. H. (2009). Examining gender differences in attitudes toward interactive classroom communications systems (ICCS). Computers & Education, 52, 730-740.
Kay, R. H., & LeSage, A. (2009). Examining the benefits and challenges of using audience response systems: A review of the literature. Computers & Education(53), 819-827.
MacGeorge, E. L., Homan, S. R., Dunning, J. B., Elmore, D., Bodie, G. D., Evans, E., . . . Geddes, B. (2008, April). Student evaluation of audience response technology in large lecture classes. Educational Technology Research & Development, 56(2), 125-145.
Mayer, R. E., Stull, A., DeLeeuw, K., Almeroth, K., Bimber, B., Chun, D., . . . Zhang, H. (2009). Clickers in college classrooms: Fostering learnning with questioning methods in large lecture classes. Contemporary Educational Psychology, 34, 51-57.
Miller, R. G., Ashar, B. H., & Getz, K. J. (2003). Evaluation of an audience response system for the continuing education of health professionals. The Journal of Continuing Education in the Health Professions, 23(2), 109-115.
Mula, J. M., & Kavanagh, M. (2009). Click go the students, click-click-click: the efficacy of a student response system for engaging students to improve feedback and performance. e-Journal of Business Education & Scholarship of Teaching, 3(1), 1-17.
Mun, W. K., Hew, K. F., & Cheung, W. S. (2009). The impact of the use of response pad system on the learning of secondary school physics concepts: A Singapore quasi-experiment study. British Journal of Educational Technology, 40(5), 848-860.
Shaffer, D. M., & Collura, M. J. (2009). Evaluating the effectiveness of a personal response system in the classroom. Teaching of Psychology, 36(4), 273-277.
Skinner, S. (2009). On clickers, quesitons, and learning. Journal of College Science Teaching, 38(4), 20-23.
Sprague, E. W., & Dahl, D. W. (2010). Learning to click: An evaluation of the personal response system clicker technology in introductory marketing courses. Journal of Marketing Education, 32(1), 93-103.
Stowell, J. R., & Nelson, J. M. (2007). Benefits of electronic audience response systems on student participation, learning, and emotion. Teaching of Psychology, 34(4), 253-258.
Appendix
Summary of empirical studies reviewed
Author(s) and year |
Research purpose |
Research method |
Data Collection Technique |
Context |
Barnes (2008) |
Examine student performance and attitudes in lecture-free versus traditional lecture classes. |
Mixed (quasi-experimental and descriptive) |
Pre-test and two post-tests, anonymous survey |
N=122 public high school biology students in Idaho. The study encompassed three units of study. Two sections used a lecture free method including the use of an ARS, Two sections used traditional lecture without an ARS. |
Bertrand (2009) |
Examine effect of an ARS on student performance on AP exams. |
Quasi-experimental |
Post-test only |
Secondary students in AP Physics over multiples years. Each year, the use of an ARS increased. The ARS was used for the purpose of practicing AP physics test multiple choice questions, providing feedback, and discussing strategies for answering the questions. |
Conoley, Croom, Moore, & Flowers (2007) |
Determine whether using an ARS improves student achievement. |
Quasi-experimental |
Post-test only |
Secondary students in an Agriscience Applications course were collected. Two sections used an ARS to provide feedback, while one section received traditional instruction, then the treatments were switched. |
Deleo, Eichenholz, & Sosin (2009) |
Determine if and how an ARS be used to improve the quality of information literacy instruction. |
Quasi-experimental |
Survey |
Graduate Educational Leadership in Technology students in three different cohort years were treated with different pedagogies in a single session course on information literacy. |
Edens (2008) |
Determine if the instructional approach used with an ARS influence student achievement, attitudes, and behaviors. |
Quasi-experimental |
Pre-test/post-test, anonymous survey |
N=120 undergraduate introductory educational psychology student. One group was treated to an ARS method where responses and attendance counted toward their grade, the other groups was treated to an ARS method where responses were used for formative feedback. |
Gauci, et al. (2009) |
Determine if active lectures using an ARS improve student engagement and learning outcomes in large group physiology lectures. |
Descriptive |
Post-test, surveys, interviews |
N=175 undergraduate science students in Physiology. The ARS was used for taking attendance, responding to questions, and providing feedback. |
Author(s) and year |
Research purpose |
Research method |
Data Collection Technique |
Context |
Graham, et al. (2007) |
Investigate the impact of an ARS on student engagement in undergraduate university courses. |
Descriptive |
Survey |
N=688 undergraduate students campus wide at Brigham Young University. The ARS was used to increase participation and provide feedback. |
Griff & Matter (2008) |
Determine if registration order of ARS student response pads with the university Blackboard system can be used to identify at risk students. |
Descriptive |
Registration data from the university Blackboard system, student final grades |
N=746 undergraduate students in four sections of Anatomy and Physiology and one section of Introduction to Biology at the University of Cincinnati. The university’s Blackboard system was used to register students’ response pads. |
Kay (2009) |
Determine if there are gender differences in attitudes toward the use of an ARS in the classroom. |
Quasi-Experimental |
Surveys |
N=659 students in grades 9-12 in various courses. The techniques used with the ARS varied between disciplines and instructors. |
MacGeorge, et al. (2008) |
Measure student perceptions of the uses of an ARS in the classroom. |
Descriptive |
Surveys |
N=854 students who used an ARS in three large lecture university courses. The techniques used with the ARS varied between disciplines and instructors. |
Mayer, et al. (2009) |
To determine the effect of using an ARS in large lecture classroom to encourage participation on student performance. |
Quasi-experimental |
Post-test |
N=385 college student who completed Educational Psychology in three sections over three years. An ARS system was used to generate discussion in one group, typed questions were used to generate discussion in a second group, and participants in the third group were asked if they had any questions. |
Miller, et al. (2003) |
Determine the effect of an ARS in continuing medical education on participant learning and attitudes. |
True-experimental |
Voluntary post-test and survey |
N=283 health care professionals. Participants in one treatment group used an ARS to generate discussion during a continuing medical education presentation, Participants in the other treatment group did not used questioning and hand-raising to generate discussion. |
Author(s) and year |
Research purpose |
Research method |
Data Collection Technique |
Context |
Mula & Kavanagh (2009) |
Determine the effect of an ARS on student perceptions, participation, and performance. |
Mixed (quasi-experimental and descriptive) |
Time-series tests, post-tests, surveys |
N=120 undergraduate accounting students at an Australian university enrolled in three different sections. An ARS was used to provide feedback during one semester, but not during the two other semesters. |
Mun, et al. (2009) |
Determine the impact of an ARS on learning secondary school physics concepts. |
Mixed (quasi-experimental, descriptive) |
Pre-test/post-test, instructor survey, student survey |
N=70 secondary physics students in Singapore. One group used an ARS to answer in class questions and provide feedback. The other group used verbal in class questions. |
Shaffer & Collura (2009) |
Evaluate the effectiveness of an ARS in the classroom. |
Quasi-experimental |
Post-test, survey |
N=177 introduction to psychology students in four sections at The Ohio State University. One group used the ARS to answer four questions during a one hour lecture class, and the ARS was used to provide feedback and generate discussion. The other group answered the questions through hand-raising, followed by feedback and discussion. |
Skinner (2009) |
Describe student perceptions and performance when an ARS was used during lectures. |
Descriptive |
Surveys, post-test |
Community college pre-nursing/allied health students in an anatomy and physiology class. The ARS was used to answer questions, provide feedback, and generate discussion during lectures. |
Sprague & Dahl (2010) |
Describe the relationship between student perceptions, student self-assessment and final grades when an ARS system is used. |
Descriptive |
Surveys, post-test |
N=93 undergraduate students in Introductory Marketing. The ARS was used for posing questions, responding to questions, providing feedback, and generating discussion. |
Stowell & Nelson (2007) |
Determine the effects of an ARS on student participation, learning, and emotion. |
True-experimental |
Post-test, visual review of videotaped lectures, surveys |
N=140 undergraduate introduction to psychology students who volunteered to participate in a 30 minute lecture on the central nervous system. One group was presented with a standard lecture with spontaneous questions and individual student responses were solicited via hand-raising. A second group was presented with the same lecture, formal questions were asked, and hand-raising was used to tally student responses. A third group used response cards to answer questions, and the fourth group used an ARS to answer questions. In the second, third, and fourth groups, feedback was given following each response. |