Type of Publication?

Book chapter.

Reviewed What?

Reviewed the history of the use of audience response systems.

Strengths?

Interesting and enlightening review. Who knew, for example, that audience response systems were first used in the 1960’s?

Evaluation?

Provides a wide-ranging introduction to the history and thinking around audience response systems.

Type of Publication?

Descriptive article in refereed journal on learning.

Tested What?

Did not compare. Instead, the authors describe their experiences using laptops to teach computer science. They did measure student reactions. 

Strengths?

One of the first articles to examine the use of wireless laptops in the classroom.

Weaknesses?

No comparison data.

Evaluation?

Not much evidence for effectiveness.

Type of Publication?

Practice review article in refereed journal on physics.

Reviewed What?

The University of Massachusetts Physic’s department’s approach to using audience response systems.

Strengths?

Very thoughtful review of their approach to using audience response systems in teaching physics.

Weaknesses?

It’s just one approach.

Evaluation?

Great article by very thoughtful practitioners. Recommended for non-physics folks as well. They make a nice case that instructors ought to have more than just content goals for their learners. Instructors ought to also have a process (cognitive) goal, and a metacognitive goal. For their physics courses, the process goals include 12 “habits of mind” that all their learners are expected to come away with as well.

Type of Publication?

Research article (available online) on learning.

Tested What?

Compared class sessions that used an audience response system with class sessions that did not, where both types of sessions utilized active-engagement techniques. Looked at learner satisfaction, objective measures of learner engagement, and at learning results.

Strengths?

Very strong research design. Compared same learners in a counterbalanced design using audience response systems in half the class sessions and not using them in other half of sessions. Measured learning results, not just learner satisfaction ratings.

Weaknesses?

Mostly minor issues, except the fact that the authors didn’t cite raw learning results, making analysis difficult. Can’t rule out instructor-enthusiasm effects. Measures of learner engagement are not convincing because they don’t account for handset responses and they are analyzed by only one observer who was not blind to condition.

Evaluation?

Seems to demonstrate improvements in learning on exam questions that were most similar to audience-response questions used in class, but no improvement for exam questions unrelated to class questions. Also shows the typical result of high learner satisfaction with audience response systems. However, while the learners rated the audience-response systems highly, on questions about the course in general, the ratings were similar whether learners had just finished the series of sessions using (or not using) audience response systems.

Type of Publication?

Research and practice review article in refereed journal on physics.

Reviewed What?

Eric Mazur’s (and Catherine Crouch’s) Peer Instruction practice and results. Research results compare classes taught in traditional ways to those taught with the Peer-Instruction methods.

Strengths?

Peer Instruction is one of the most widespread methodologies in use and the authors do a nice job of covering the practice and results. At least some of the research comparisons use standardized tests of physics competence, providing good face validity and lessening opportunities for bias. Many instructors utilized.

Weaknesses?

Peer Instruction is just one approach. Research comparisons are versus previously-taught traditional classes, leaving open the possibility for learning gains due to differences in instructor enthusiasm and instructor preparation. Also, because no pretests were provided on the traditional courses, we can’t rule out bias due to differences in learners.

Evaluation?

Great article by thoughtful practitioners. Research comparisons, while not perfect, are suggestive of the benefits of Peer Instruction methodologies. Note that authors do NOT focus on the use of audience response systems. In fact, in the research cited they didn’t always use them. In an endnote they say, “We did not see any significant changes in student learning on introducing the classroom network system, and find the main advantages of the [system] are anonymity of student responses and data collection…” (page 976, endnote 9). On the other hand, the improvements in results they cite (in Figure 2 on page 971 and Table 1 on page 972) all come after the audience response systems were introduced. In any case, the study did not test audience response systems vs. no audience response systems. It examined the Peer Instruction method.

Type of Publication?

Research article in refereed journal on learning.

Tested What?

Active learning methods in a specially-designed classroom that enabled simulation and visualization versus traditional lecture method in a standard classroom.

Strengths?

Interesting discussion.

Weaknesses?

Major methodological weaknesses. Comparison groups likely sampled from different populations. Conflated active learning with the many technology interventions. The treatment group received conceptual questions. The control group did not. Couldn’t rule out effects of exciting new classroom or instructor-enthusiasm effects.

Evaluation?

Major methodological weaknesses. Moreover, can’t differentiate between effects of audience response systems and the many other variables in play.

Draper, S. W., & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20, 81-94.

Type of Publication?

Research article in refereed journal on learning.

Tested What? 

Student opinions of audience response systems.

Strengths?

Looked at lots of classrooms and instructors. Very thoughtful analysis of potential benefits.

Weaknesses?

Only measured student opinions. No comparison with non-ARS situation.

Evaluation?

Methodology weak, but can tell us learner opinion, and did show improvements of opinion over time.

Type of Publication?

Descriptive article in refereed journal on learning.

Tested What?

Did not compare. Instead, the authors describe their institution’s experiences using audience response systems to improve classroom sessions. They did measure learner and instructor reactions.

Strengths?

Looked at many different topics taught with many different learners and instructors. Thoughtful analysis.

Weaknesses?

No comparison data. No data on learning improvements, only learner and instructor opinions.

Evaluation?

No evidence for learning effectiveness, but some suggestive evidence for learner satisfaction (though there is no comparison to traditional methods). Very thoughtful analysis with many helpful suggestions for how to use audience response technology.

Type of Publication?

Descriptive article in refereed journal on learning.

Tested What?

Did not compare. Instead, the authors describe their experiences using the audience response system and show some graphs of student reactions to using the response systems and active engagement.

Strengths?

An early look at how audience response systems were used.

Weaknesses?

No comparison data.

Evaluation?

Not much evidence for effectiveness.

Type of Publication?

Short book (pamphlet).

Reviewed What?

Provides an introduction to using handsets in the classroom

Strengths?

Covers many important areas. Good introduction.

Weaknesses?

Recommendations are limited to only a few questioning strategies. Book now a bit dated. Not generally available unless you contact publisher.

Evaluation?

Good primer. Could be better.

Type of Publication?

Research article in refereed journal on physics.

Tested What?

Active learning methods vs. traditional methods in classrooms.

Strengths?

Lots of classrooms, instructors, and institutions. Used nicely standardized relevant test in a pretest-posttest design.

Weaknesses?

Data seriously biased by collection method. Can’t rule out instructor bias and instructor preparation effects.

Evaluation?

Focuses not on ARS usage, but on active learning. Poor methodology makes results dubious.

Type of Publication?

Research article in refereed journal on learning.

Tested What?

Compared learner ratings on both peer instruction and classwide discussions, both of which were used in conjunction with audience response systems.  

Strengths?

Used a within-subject design, examining the feelings of the same students to both peer-instruction techniques and classwide discussion techniques. Used several methods to gather learner opinions.

Weaknesses?

Measured only learner opinions, not actual learning results.

Evaluation?

Shows that learners seem to prefer peer-instruction techniques to classwide-discussion techniques (when both are used with audience response systems).

Type of Publication?

Short research-like article in refereed journal on physics.

Tested What?

Compared examination pass rates over the years when physics classes used, or did not use, simple one-button handsets. 

Strengths?

Used an ABA design, measuring actual learning results, comparing no-handset classes in Year’s 1 and 2 versus handset classes in Year’s 3 and 4, and no-handset classes in Year’s 5 and 6.

Weaknesses?

Methodology section is too brief to really assess the experimental methodology. The one-button handsets in use are much more basic than today’s handsets. Use of handsets was conflated with active-engagement techniques, so we can’t be sure what produced the effect, handsets or active-engagement. No statistics were used, making it difficult to assess comparisons, though the handset classes appear to generally—but not always—outperform the no-handset classes.

Evaluation?

Shows suggestive improvements in learning due to active engagement and handset use, but methodology is quite suspect.

Coming...

What other articles would you like to see annotated? Contact me with your requests. Also, if you disagree with my evaluation, let me know. Send me an email by clicking here.