Vol. 31 No. 2 (2016)
Research Articles

Redesigning Design: Field Testing a Revised Design Rubric Based of iNACOL Quality Course Standards

David Adelstein
Wayne State University
Michael K Barbour

Published 2016-11-15


  • K-12 online learning,
  • K-12 distance education,
  • virtual school,
  • cyber school,
  • online course design

How to Cite

Adelstein, D., & Barbour, M. K. (2016). Redesigning Design: Field Testing a Revised Design Rubric Based of iNACOL Quality Course Standards. International Journal of E-Learning & Distance Education Revue Internationale Du E-Learning Et La Formation à Distance, 31(2). Retrieved from https://www.ijede.ca/index.php/jde/article/view/976


Designers have a limited selection of K-12 online course creation standards to choose from that are not blocked behind proprietary or pay walls. For numerous institutions and states, the use of the iNACOL National Standards for Quality Online Courses is becoming a widely used resource. This article presents the final phase in a three-part study to test the validity and reliability of the iNACOL standards specifically to online course design. Phase three was a field test of the revised rubric based on the iNACOL standards against current K-12 online courses. While the results show a strong exact match percentage, there is more work to be done with the revised rubric.


Les concepteurs ont une sélection limitée des normes K-12 de création de cours en ligne à choisir qui ne sont pas bloqués derrière des propriétés exclusives ou des péages informatiques. Pour de nombreuses institutions et états, l'utilisation des Normes nationales pour les cours en ligne de qualité iNACOL devient une ressource largement utilisée. Cet article présente la phase finale d’une étude en trois parties pour tester la validité et la fiabilité des normes iNACOL spécifiquement liées à la conception de cours en ligne. La phase trois était une mise à l’essai sur le terrain de la rubrique révisée établie en fonction des normes iNACOL par rapport aux cours en ligne K-12 actuels. Bien que les résultats montrent un fort pourcentage de correspondance exacte, il y a plus de travail à faire avec la rubrique révisée.


  1. Adelstein, D., & Barbour, M. K. (2016). Building better courses: Examining the construct validity of the iNACOL national standards for quality online courses. Journal of Online Learning Research, 2(1), 41-73. Retrieved from https://www.learntechlib.org/d/171515
  2. Adelstein, D., & Barbour, M. K. (in press ). Improving the K-12 online course design review process: Experts weigh in on iNACOL National Standards for Quality Online Courses. International Review of Research in Open and Distance Learning, 18(1)
  3. Barbour, M. K. (2005). The design of web-based courses for secondary students. Journal of Distance Learning, 9(1), 27-36. Retrieved from http://journals.akoaotearoa.ac.nz/index.php/JOFDL/article/view/122
  4. Barbour, M. K. (2007a). Principles of effective web-based content for secondary school students: Teacher and developer perceptions. Journal of Distance Education, 21(3), 93-114. Retrieved from http://www.jofde.ca/index.php/jde/article/view/30
  5. Barbour, M. K. (2007b). Portrait of rural virtual schooling. Canadian Journal of Educational Administration and Policy, 59. Retrieved from http://www.umanitoba.ca/publications/cjeap/articles/barbour.html
  6. Barbour, M. K. (2013). The landscape of K-12 online learning: Examining what is known. In M. G. Moore (Eds.), Handbook of distance education (3rd ed.) (pp. 574-593). New York: Routledge.
  7. Barbour, M. K., & Adelstein, D. (2013a). High-school students’ perceptions of effective online course design. The Morning Watch, 41(1-2), 56-65. Retrieved from http://www.mun.ca/educ/faculty/mwatch/vol41/fall2013/michaelBarbour.pdf
  8. Barbour, M. K., & Adelstein, D. (2013b). Voracious appetite of online teaching: Examining labour issues related to K-12 online learning. Vancouver, BC: British Columbia Teachers Federation. Retrieved from http://www.bctf.ca/uploadedFiles/Public/Issues/Technology/VoraciousAppetite.pdf
  9. Barbour, M. K., & Cooze, M. (2004). All for one and one for all: Designing web-based courses for students based upon individual learning styles. Staff and Educational Development International, 8(2/3), 95-108.
  10. Barbour, M. K., Morrison, J., & Adelstein, D. (2014). The forgotten teachers in K-12 online learning: Examining the perceptions of teachers who develop K-12 online courses. International Journal of Online Pedagogy and Course Design, 4(3), 18-33.
  11. Brennan, P. F., & Hays, B. J. (1992). Focus on psychometrics: The kappa statistic for establishing interrater reliability in the secondary analysis of qualitative clinical data. Research in Nursing & Health, 15(2), 153-158.
  12. Bresciani, M., Oakleaf, M., Kolkhorst, F., Nebeker, C., Barlow, J., Duncan, K., & Hickmott, J.
  13. (2009). Examining design and inter-rater reliability of a rubric measuring research quality across multiple disciplines. Practical Assessment, Research & Evaluation, 14(12). Retrieved from http://www.pareonline.net/getvn.asp?v=14&n=12
  14. Davis, N., Roblyer, M. P., Charania, A., Ferdig, R., Harms, C., Compton, L. K. L., & Cho, M. O. (2007). Illustrating the “virtual” in virtual schooling: Challenges and strategies for creating real tools to prepare virtual teachers. The Internet and Higher Education, 10(1), 27-39.
  15. Dawley, L., Rice, K., & Hinck, G. (2010). Going virtual! 2010: The status of professional development and unique needs of K-12 online teachers. Boise ID: Boise State University. Retrieved from https://edtech.boisestate.edu/goingvirtual/goingvirtual3.pdf
  16. Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods. New York: McGraw Hill.
  17. Elbaum, B., McIntyre, C., & Smith, A. (2002). Essential elements: Prepare, design, and teach your online course. Madison, WI: Atwood Publishing.
  18. Friend, B., & Johnston, S. (2005). Florida virtual school: A choice for all students. In Z. L. Berge & T. Clark (Eds.), Virtual schools: Planning for success (pp. 97–117). New York: Teachers College Press.
  19. Gemin, B., Pape, L., Vashaw, L. & Watson, J. (2015). Keeping pace with K-12 digital learning: An annual review of policy and practice, 2015. Durango, CO: Evergreen Education Group. Retrieved from http://www.kpk12.com/wp-content/uploads/Evergreen_KeepingPace_2015.pdf
  20. Huett, K. C., Huett, J. B., & Ringlaben, R. (2011). From bricks to clicks: Building quality K-12 online classes through an innovative course review project. Online Journal of Distance Learning Administration, 14(4). Retrieved from http://www.westga.edu/~distance/ojdla/winter144/huett_huett_ringlaben.html
  21. Keeler, C. G., & Horney, M. A. (2007). Online course designs: Are special needs being met? American Journal of Distance Education, 21(2), 61–75.
  22. Keeler, C., Richter, J., Anderson-Inman, L., Horney, M., & Ditson, M. (2007). Exceptional learners: Differentiated instruction online. In C. Cavanaugh & R. L. Blomeyer, (Eds.), What works in K-12 online learning (pp. 125-141). Eugene, OR: International Society for Technology in Education.
  23. Legon, R., & Runyon, J. (2007). Research on the impact of the quality matters course review process. In 23rd Annual Conference on Distance Teaching & Learning (pp. 8-10). Madison, WI: Division of Continuing Studies, University of Wisconsin-Madison.
  24. Michigan Virtual Learning Research Institute. (2016). 2015-16 directives support and accelerate innovation in online and blended learning. East Lansing, MI: Author. Retrieved from http://mvlri.org/About-Us/2015-16-Directives
  25. Molnar, A. (Ed.), Rice, J. K., Huerta, L., Shafer, S. R., Barbour, M. K., Miron, G., Gulosino, C, Horvitz, B. (2014). Virtual schools in the U.S. 2014: Politics, performance, policy, and research evidence. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/virtual-schools-annual-2014
  26. Moore, P. (2015). An employer toolkit for employee training and policies related to social media and crisis communications in health care organizations. Unpublished doctoral dissertation, Ball State University, Muncie, IN.
  27. Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage.
  28. Rice, K., & Dawley, L. (2007). Going virtual! The status of professional development for K-12 online teachers. Boise ID: Boise State University. Retrieved from https://edtech.boisestate.edu/goingvirtual/goingvirtual1.pdf
  29. Rice, K., Dawley, L., Gasell, C., & Florez, C. (2008). Going virtual! Unique needs and challenges of K-12 online teachers. Boise ID: Boise State University. Retrieved from https://edtech.boisestate.edu/goingvirtual/goingvirtual2.pdf
  30. Taggart, G., S., Phifer, S. J., Nixon, J. A., & Wood, M. (1998). Rubrics: A handbook for construction and use. Lancaster, PA: Technomic Publishing Co.
  31. Yamashiro, K., & Zucker, A. (1999). An expert panel review of the quality of Virtual High School courses: Final report. Arlington, VA: SRI International. Retrieved from http://www.thevhscollaborative.org/sites/default/files/public/vhsexprt.pdf
  32. Zucker, A., & Kozma, R. (2003). The Virtual High School: Teaching generation V. New York: Teachers College Press.