Abstract
Bullying is a problem for schools around the world, and is an important topic for research because it has been associated with negative outcomes on numerous social, psychological, and academic measures. Antibullying prevention and intervention programs have varied greatly in their outcomes, with some studies reporting positive results while others have reported little or no positive impacts. Prompted by accountability demands, many agencies have developed standards with which to assess whether social programs are effective. Antibullying program evaluations have not been systematically reviewed to determine whether these types of standards are being applied. The purpose of this study was to assess the rigor of recent peer-reviewed antibullying program evaluations. Thirty-one peer-reviewed evaluations of antibullying programs, published within the last 10 years, were identified and coded for study characteristics. Shortcomings were identified in many of these program evaluations. In order to improve evaluation practices, researchers should consider using more rigorous designs to identify cause-effect relationships, including control conditions and random assignment, using more appropriate pre-post intervals, using more advanced methods of analyses such as hierarchical linear modeling, and systematically verifying program integrity to obtain dosage data that can be used in the outcome analyses.
Similar content being viewed by others
References
Alsaker, F. D. (2004). Bernese programme against victimization in kindergarten and elementary school. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 289–306). New York: Cambridge University Press.
Alsaker, F. D., & Valkanover, S. (2001). Early diagnosis and prevention of victimization in kindergarten. In J. Juvonen & S. Graham (Eds.), Peer harassment in school: The plight of the vulnerable and victimized (pp. 175–195). New York: Guilford Press.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
Baldry, A. C., & Farrington, D. P. (2004). Evaluation of an intervention program for the reduction of bullying and victimization in schools. Aggressive Behavior, 30, 1–15. doi:10.1002/ab.20000.
Baldry, A. C., & Farrington, D. P. (2007). Effectiveness of programs to prevent school bullying. Victims and Offenders, 2, 183–204. doi:10.1080/15564880701263155.
Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.
Chatterji, M. (2007). Grades of evidence: Variability in quality of findings in effectiveness studies of complex field interventions. The American Journal of Evaluation, 28, 239–255. doi:10.1177/1098214007304884.
Coie, J. D. (2004). The impact of negative social experiences on the development of antisocial behavior. In J. B. Kupersmidt & K. A. Dodge (Eds.), Children’s peer relations: From development to intervention (pp. 243–267). Washington, DC: American Psychological Association.
Connolly, J., Pepler, D., Craig, W., & Taradash, A. (2000). Dating experiences of bullies in early adolescence. Child Maltreatment, 5, 299–310. doi:10.1177/1077559500005004002.
Cowie, H., & Olafsson, R. (2000). The role of peer support in helping the victims of bullying in a school with high levels of aggression. School Psychology International, 21, 79–95. doi:10.1177/0143034300211006.
Crick, N. R., & Bigbee, M. A. (1998). Relational and overt forms of peer victimization: A multiinformant approach. Journal of Consulting and Clinical Psychology, 66, 337–347. doi:10.1037/0022-006X.66.2.337.
Cross, D., Hall, M., Hamilton, G., Pintabona, Y., & Erceg, E. (2004). Australia: The Friendly Schools Project. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 187–210). New York: Cambridge University Press.
Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45. doi:10.1016/S0272-7358(97)00043-3.
DeRosier, M. E. (2004). Building relationships and combating bullying: Effectiveness of a school-based social skills group intervention. Journal of Clinical Child and Adolescent Psychology, 33, 196–201. doi:10.1207/S15374424JCCP3301_18.
Edwards, D., Hunt, M. H., Meyers, J., Grogg, K. R., & Jarrett, O. (2005). Acceptability and student outcomes of a violence prevention curriculum. The Journal of Primary Prevention, 26, 401–418. doi:10.1007/s10935-005-0002-z.
Fekkes, M., Pijpers, F. I. M., & Verloove-Vanhorick, P. (2006). Effects of antibullying school program on bullying and health complaints. Archives of Pediatrics & Adolescent Medicine, 160, 638–644. doi:10.1001/archpedi.160.6.638.
Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., et al. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6, 151–175. doi:10.1007/s11121-005-5553-y.
Frey, K. S., Hirschstein, M. K., Snell, J. L., Edstrom, L. V. S., MacKenzie, E. P., & Broderick, C. J. (2005). Reducing playground bullying and supporting beliefs: An experimental trial of the Steps to Respect program. Developmental Psychology, 41, 479–491. doi:10.1037/0012-1649.41.3.479.
Gollwitzer, M., Eisenbach, K., Atria, M., Strohmeier, D., & Banse, R. (2006). Evaluation of aggression-reducing effects of the “Viennese Social Competence Training.” Swiss Journal of Psychology, 65, 125–135. doi:10.1024/1421-0185.65.2.125.
Greenberg, M. T., Domitrovich, C. E., Graczyk, P. A., & Zins, J. E. (2005). The study of implementation in school-based preventive interventions: Theory, research, and practice (Vol. 3). Rockville, MD: Center for Mental Health Services, Substance Abuse and Mental Health Services Administration.
Hall, G. E., & Hord, S. M. (2006). Implementing change: Patterns, principles, and potholes (2nd ed.). New York: Pearson.
Heydenberk, R. A., Heydenberk, W. R., & Tzenova, V. (2006). Conflict resolution and bully prevention: Skills for school success. Conflict Resolution Quarterly, 24, 55–69. doi:10.1002/crq.157.
Hunt, C. (2007). The effect of an education program on attitudes and beliefs about bullying and bullying behaviour in junior secondary school students. Child and Adolescent Mental Health, 12, 21–26. doi:10.1111/j.1475-3588.2006.00417.x.
Jennifer, D., & Shaughnessy, J. (2005). Promoting non-violence in schools: The role of cultural, organisational and managerial factors. Educational and Child Psychology, 22, 58–66.
Kim, J. (2006). The effect of a bullying prevention program on responsibility and victimization of bullied children in Korea. International Journal of Reality Therapy, 26, 4–8.
Kyler, S. J., Bumbarger, B. K., & Greenberg, M. T. (2005). Technical assistance fact sheet on evidence-based programs. University Park, PA: Prevention Research Center for the Promotion of Human Development. Retrieved Oct. 1, 2007 from http://www.prevention.psu.edu/pubs/documents/EBP_factsheet.pdf.
Leadbeater, B., & Hoglund, W. (2006). Changing the social contexts of peer victimization. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 15, 21–26.
Leadbeater, B., Hoglund, W., & Wood, T. (2003). Changing contexts? The effects of a primary prevention program on classroom levels of peer relational and physical victimization. Journal of Community Psychology, 31, 397–418. doi:10.1002/jcop.10057.
Leff, S. S., Power, T. J., & Goldstein, A. B. (2004). Outcome measures to assess the effectiveness of bullying prevention programs in schools. In D. L. Espelage & S. M. Swearer (Eds.), Bullying in American schools: A social-ecological perspective on prevention and intervention (pp. 269–293). Mahwah, NJ: Lawrence Erlbaum.
Limber, S. P., & Small, M. A. (2003). State laws and policies to address bullying in schools. School Psychology Review, 32, 445–455.
Limber, S. P., Nation, M., Tracy, A. J., Melton, G. G., & Flerx, V. (2004). Implementation of the Olweus Bullying Prevention programme in the Southeastern United States. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 55–79). New York: Cambridge University Press.
McLaughlin, L., Laux, J. M., & Pescara-Kovach, L. (2006). Using multimedia to reduce bullying and victimization in third-grade urban schools. Professional School Counseling, 10, 153–160.
Menesini, E., Codecasa, E., Benelli, B., & Cowie, H. (2003). Enhancing children’s responsibility to take action against bullying: Evaluation of a befriending intervention in Italian middle schools. Aggressive Behavior, 29, 10–14.
Merrell, K. W., Gueldner, B. A., Ross, S. W., & Isava, D. M. (2008). How effective are school bullying intervention programs? A meta-analysis of intervention research. School Psychology Quarterly, 23, 26–42. doi:10.1037/1045-3830.23.1.26.
Meyer, N., & Lesch, E. (2000). An analysis of the limitations of a behavioural programme for bullying boys from a subeconomic environment. South African Journal of Child and Adolescent Mental Health, 12, 59–69.
Mihalic, S. (2007). Matrix of programs. Boulder, CO: Center for the Study of Prevention of Violence, Blueprints for Violence Prevention. Retrieved on September 24, 2007, from www.colorado.edu/cspv/blueprints.
Mooij, T. (2005). National campaign effects on secondary pupils’ bullying and violence. The British Journal of Educational Psychology, 75, 489–511. doi:10.1348/000709904X23727.
Nansel, T. R., Overpeck, M., Pilla, R. S., Ruan, W. J., Simons-Marton, B., & Scheidt, P. (2001). Bullying behaviors among US youth: Prevalence and association with psychosocial adjustment. Journal of the American Medical Association, 285, 2094–2100. doi:10.1001/jama.285.16.2094.
Naylor, P., & Cowie, H. (1999). The effectiveness of peer support systems in challenging school bullying: The perspectives and experiences of teachers and pupils. Journal of Adolescence, 22, 467–479. doi:10.1006/jado.1999.0241.
Newman-Carlson, D., & Horne, A. M. (2004). Bully Busters: A psychoeducational intervention for reducing bullying behavior in middle school students. Journal of Counseling and Development, 82, 259–267.
No Child Left Behind Act of 2001, P.L. 107-110, 115 Stat.1425 (2002).
Okumura, T. (2007). Sample size determination for hierarchical linear models considering uncertainty in parameter estimates. Behaviormetrica, 34, 79–93. doi:10.2333/bhmk.34.79.
Olweus, D. (1991). Bully/victim problems among school children: Basic facts and effects of a school-based intervention program. In D. Pepler & K. H. Rubin (Eds.), The development and treatment of childhood aggression (pp. 411–448). Hillsdale, NJ: Erlbaum.
Olweus, D. (1993). Bullying at school: What we know and what we can do. Malden, MA: Blackwell.
Olweus, D. (2005). A useful evaluation design, and effects of the Olweus Bullying Prevention Program. Psychology, Crime & Law, 11, 389–402. doi:10.1080/10683160500255471.
O’Moore, A. M., & Minton, S. J. (2004). Ireland: The Donegal Primary Schools’ anti-bullying project. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 275–287). New York: Cambridge University Press.
O’Moore, A. M., & Minton, S. J. (2005). Evaluation of the effectiveness of an anti-bullying programme in primary schools. Aggressive Behavior, 31, 609–622. doi:10.1002/ab.20098.
Orpinas, P., & Horne, A. M. (2006). Bullies: The problem and its impact. In P. Orpinas & A. M. Horne (Eds.), Bullying prevention: Creating a positive school climate and developing social competence (pp. 11–31). Washington, DC: American Psychological Association.
Orpinas, P., Home, A. M., & Staniszewski, D. (2003). School bullying: Changing the problem by changing the school. School Psychology Review, 32, 431–444.
Ortega, R., Del Rey, R., & Mora-Merchan, J. A. (2004). SAVE model: An anti-bullying intervention in Spain. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 167–185). New York: Cambridge University Press.
Patchin, J. W., & Hinduja, S. (2006). Bullies move beyond the schoolyard: A preliminary look at cyberbullying. Youth Violence and Juvenile Justice, 4, 148–169. doi:10.1177/1541204006286288.
Pawson, R., & Tilley, N. (1998). Cook-book methods and disastrous recipes: A rejoinder to Farrington. Evaluation, 4, 211–213. doi:10.1177/13563899822208545.
Pellegrini, A., & Bartini, M. (2000). An empirical comparison of methods of sampling aggression and victimization in school settings. Journal of Educational Psychology, 92, 360–366. doi:10.1037/0022-0663.92.2.360.
Peterson, L., & Rigby, K. (1999). Countering bullying at an Australian secondary school with students as helpers. Journal of Adolescence, 22, 481–492. doi:10.1006/jado.1999.0242.
Rahey, L., & Craig, W. M. (2002). Evaluation of an ecological program to reduce bullying in schools. Canadian Journal of Counselling, 36, 281–296.
Roland, E. (2002). Bullying, depressive symptoms and suicidal thoughts. Educational Research, 4, 55–67. doi:10.1080/00131880110107351.
Rosenbluth, B., Whitaker, D. J., Sanchez, E., & Valle, L. A. (2004). The Expect Respect project: Preventing bullying and sexual harassment in US elementary schools. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 211–233). New York: Cambridge University Press.
Salmivalli, C. (2001). Peer-led intervention campaign against school bullying: Who considered it useful, who benefited? Educational Research, 43, 263–278. doi:10.1080/00131880110081035.
Salmivalli, C., Kaukiainen, A., Voeten, M., & Sinisammal, M. (2004). Targeting the group as a whole: The Finnish anti-bullying intervention. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 251–273). New York: Cambridge University Press.
Salmivalli, C., Kaukiainen, A., & Voeten, M. (2005). Anti-bullying intervention: Implementation and outcome. The British Journal of Educational Psychology, 75, 465–487. doi:10.1348/000709905X26011.
Smith, P. K., Pepler, D., & Rigby, K. (Eds.) (2004). Bullying in schools: How successful can interventions be? New York: Cambridge University Press.
Smith, J. D., Schneider, B. H., Smith, P. K., & Ananiadou, K. (2004). The effectiveness of whole-school anti-bullying programs: A synthesis of evaluation research. School Psychology Review, 33, 548–561.
Smith, J. D., Ryan, W., & Cousins, J. B. (2007). Antibullying programs: A survey of evaluation activities in public schools. Studies in Educational Evaluation, 33, 120–134. doi:10.1016/j.stueduc.2007.04.002.
Standards of Evidence Committee. (2004). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Retrieved September 5, 2008 from http://www.jstor.org.proxy.bib.uottawa.ca/stable/pdfplus/1049953.pdf.
Stevens, V., Van Oost, P., & De Bourdeaudhuij, I. (2000). The effects of an anti-bullying intervention programme on peers’ attitudes and behaviour. Journal of Adolescence, 23, 21–34. doi:10.1006/jado.1999.0296.
Stevens, V., Van Oost, P., & De Bourdeaudhuij, I. (2001). Implementation process of the Flemish antibullying intervention and relation with program effectiveness. Journal of School Psychology, 39, 303–317. doi:10.1016/S0022-4405(01)00073-5.
Stevens, V., Van Oost, P., & De Bourdeaudhuij, I. (2004). Interventions against bullying in Flemish schools: Programme development and evaluation. In P. K. Smith, D. Pepler, & K. Rigby (Eds.), Bullying in schools: How successful can interventions be? (pp. 141–165). New York: Cambridge University Press.
Thompson, B. (2002). What future quantitative social science research could look like: Confidence intervals for effect sizes. Educational Researcher, 31, 25–32. doi:10.3102/0013189X031003025.
Torgerson, C. J., & Torgerson, D. J. (2001). The need for randomized controlled trials in educational research. British Journal of Educational Studies, 49, 316–328. doi:10.1111/1467-8527.t01-1-00178.
Vreeman, R. C., & Carroll, A. E. (2007). A systematic review of school-based interventions to prevent bullying. Archives of Pediatrics & Adolescent Medicine, 161, 78–88. doi:10.1001/archpedi.161.1.78.
Whitaker, D. J., Rosenbluth, B., Valle, L. A., & Sanchez, E. (2004). Expect Respect: A school-based intervention to promote awareness and effective responses to bullying and sexual harassment. In D. L. Espelage & S. M. Swearer (Eds.), Bullying in American schools: A social-ecological perspective on prevention and intervention (pp. 327–350). Mahwah, NJ: Erlbaum.
Wolff, N. (2000). Using randomized controlled trials to evaluate socially complex services: Problems, challenges and recommendations. The Journal of Mental Health Policy and Economics, 3, 97–109. doi:10.1002/1099-176X(200006)3:2<97::AID-MHP77>3.0.CO;2-S.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Ryan, W., Smith, J.D. Antibullying Programs in Schools: How Effective are Evaluation Practices?. Prev Sci 10, 248–259 (2009). https://doi.org/10.1007/s11121-009-0128-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11121-009-0128-y