MOOC Research and Evaluation

Overview

The University of Toronto is committed to exploring new ways of teaching and sharing knowledge in the 21st Century, and its participation in the two platforms Coursera and EdX are an essential part of this. Some of the topics that we aim to investigate as part of our research program include:

  • Student demographics and learning goals/approaches.
  • Range of pedagogical approaches that are available to instructors.
  • Patterns of engagement and learning among participants.

Institutional Research:
For a descriptive report see: Open UToronto MOOC Initiative: Report on Second Year of Activity (September 2014). In addition to the research on learners, and student interactions with courses, we are also interested in instructors, their experiences and attitudes, and the process of constructing courses. OLS has launched a separate research project, supervised by Dr. Carol Rolheiser from the Centre for Teaching Support and Innovation, which aims to interview all UofT MOOC instructors about their experience of building and running a MOOC, as well as analyzing the course designs and pedagogical approaches of the various courses.

Demographic Report:
The Coursera platform provides rich data on learner interaction with the courses, and we will complement this learning analytics data with data on student demographics and learning intents from intro- and exit-surveys. The Office of Online Learning Strategy is coordinating individual research projects by the different PIs, and facilitating access to data from Coursera, as well as compiling general demographic data now available. See: Demographic Report on University of Toronto Coursera MOOCs.

Gates Foundation Development and Research Grants:
Two of these courses (“Statistics: Making Sense of Data”, and “Introduction to Psychology”) have received special support from the Gates Foundation to research their MOOCs. The Statistics MOOC is also being used in another Gates funded project, a multi-campus study of MOOCs as a deeply integrated instructional resource for blended learning at universities across the Maryland system.

MOOC Research Initiative Grants:
University of Toronto’s leadership in the area of evaluation of the pedagogical design potential of MOOCs will be extended as a result of MOOC Research Initiative funding for three additional research studies.

The reports of these studies are now published on the MOOC Research website.

Recent Faculty Publications:

Upcoming

Najafi, H., Rolheiser, C., Harrison, L., & Heikoop, W. (In review). Connecting Learner Motivation to Learner Progress and Completion in On-demand MOOCs.

2016

Aarabi, P., Norouzi, N., Wu, J., & Spears, M. (2016). 7 surprising lessons learned from teaching iOS programming to 30,000+ MOOC students. In 2016 IEEE Frontiers in Education Conference (FIE) (pp. 1–4). https://doi.org/10.1109/FIE.2016.7757448

Najafi, H., Rolheiser, C., Harrison, L., & Håklev, S. (In review). Variations in the pedagogical design of Massive Open Online Courses across disciplines.

Najafi, H., Rolheiser, C., Harrison, L., & Håklev, S. (In preparation). Learner motivation, intention, and achievement in on-demand courses.

Najafi, H., Slotta, J., Haklev, S. (In preparation). Fostering reflective practice in a teacher professional development MOOC.

2015

Najafi, H., Rolheiser, C., Harrison, L., & Håklev, S. (2015). University of Toronto instructors’ experiences with developing MOOCs. The International Review Of Research In Open And Distributed Learning, 16(3). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/2073

Harris, J., Heikoop, W., Van Beek, A., & Wallace, J. (2015) Teaching and Advanced Engineering MOOC: Lessons Learned. Proceedings of the Canadian Engineering Education Association (CEEA 2015). Retrieved from http://library.queensu.ca/ojs/index.php/PCEEA/article/view/5840

2014

Campbell, J., Gibbs, A., Najafi, H., & Severinski, C. (2014). A comparison of learner intent and behaviour in live and archived MOOCs. The International Review of Research In Open And Distributed Learning, 15(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1854

Campbell, J., Horton, D., Craig, M., & Gries, P. (2014). Evaluating an Inverted CS1. In Proceedings of the 45th ACM technical symposium on Computer science education (SIGCSE 2014). ACM, New York, NY. (pp.  307–312).

Gibbs, A. (2014). Experiences Teaching an Introductory Statistics MOOC. In Proceedings of the Ninth International Conference on Teaching Statistics (ICOTS9, July, 2014). Voorburg, The Netherlands: International Statistical Institute.

Horton, D., Craig, M., Campbell, J., Gries, P. & Zingaro, D. (2014). Comparing Outcomes in Inverted and Traditional CS1. In Proceedings of the 2014 conference on Innovation & technology in computer science education (ITiCSE 2014). ACM, New York, NY. (pp. 261-266).

Najafi, H., Evans, R., & Federico, C. (2014). MOOC integration into secondary school courses. The International Review Of Research In Open And Distributed Learning, 15(5). Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/1861/3098

2013

Restoule, J.P. (2013). Massive Open Online Courses and the Future of Adult Education. In C. Kawalilak & J. Groen (Eds.) Proceedings from Canadian Association for the Study of Adult Education (CASAE) 2013 (pp. 514-520).

Conference Presentations:

2016

Najafi, H., Rolheiser, C., Harrison, L., & Haklev, S. (2016, April). Pedagogical Design of Massive Open Online Courses Across Disciplines: Variations and Implications for Student Behaviour. Roundtable presentation at the American Educational Research Association annual conference, Washington, DC.

Harrison, L., Håklev, S. Najafi, H. (2016, March) Barriers to Collaboration: Challenges and opportunities In accessing, using, and sharing MOOC data. Research and Strategy Track panel discussion at Coursera Partners Conference. The Hague, Netherlands.

Najafi, H., Håklev, S., Harrison, L., Rolheiser, C., & Heikoop, W. (2016, March). How can we characterize learner completion and achievement in on-demand Coursera MOOCs? Research track presentation and main conference poster presentation at the Coursera 2016 Conference, The Hague, Netherlands.

2015

Håklev, S., Slotta, J., & Najafi, H. (2015, October). MOOC Design that Supports User-Contributed Content, Interaction and Teamwork. Paper presented at the Learning with MOOCs, Columbia University, New York.

Najafi, H., Håklev, S., Slotta, J. & Evans, R. (2015, October). Design Considerations for a Teacher Professional Development MOOC. Paper presented at the Learning with MOOCs Conference, Columbia University, New York.

Chen, B., Haklev, S., Harrison, L., Najafi, H., & Rolheiser, C. (2015, April). How do MOOC learners’ intentions relate to their behaviors and overall outcomes? Poster presented at the American Educational Research Association annual conference, Chicago, IL.

Najafi, H., Rolheiser, C., Harrison, L., & Håklev, S. (2015, March) MOOC Development and Delivery at the University of Toronto: Implications for Research and Pedagogy. Poster presented at Coursera Partners Conference. Newport, CA.

Dineen, C., Harris, G. & Newman, W. (2015). MOOCs Unshushed: Lessons from creating a professional development Massively Open Online Course (MOOC): Library Advocacy Unshushed on edX [poster session]. Toronto, ON: Ontario Library Association Super Conference.

Other

Interviews and Articles:

Short videos and articles of faculty research on MOOCs.

Jennifer Campbell interview on her MOOC research with Alison Gibbs for e-Literate TV.

Several MOOC instructors and their experiences teaching online are featured in UofT Magazine, Winter 2014

MOOC Learning and Secondary School Students Webinar on Open EdX

Poster presentation on How Do MOOC Learners’ Intentions Relate to Their Behaviours and Overall Outcomes

MOOC Research Symposium:

Shared slides are now available.

Beyond and Between “Traditional” MOOCs: Agile and Just-in-Time Learning
Jennifer Campbell (Dept. of Computer Science) and Alison Gibbs (Dept. of Statistics)

This project compares the use of MOOCs as active, instructor-led, open-facilitated courses with their use as archived, self-directed learning resources.  In order to understand potential differentiated usage patterns and learning outcomes, learner demographics, motivations, activities and completion, and levels of satisfaction are examined and compared across the two models of content delivery.

Secondary School Students and MOOC’s: A Comparison between Independent MOOC Participation and Blended Learning
Rosemary Evans, Christopher Federico, Hedieh Najafi (UTS/Rotman School of Management)

This project compares secondary school student achievement of learning outcomes and levels of student engagement and persistence under two models of instruction:

  1. Through independent engagement with a MOOC
  2. Through a blended model involving teacher support and engagement with a MOOC

Use of MOOCs in the Inverted Classroom Model
Paul Gries (Dept. of Computer Science)

Several MOOC instructors used the Coursera platform to provide video content and in-video quizzes in UofT degree courses. This was combined with active learning in the inverted classroom as well as TA support. This presentation describes students’ perceptions of the experience as well as learning outcomes and additional metrics as reflected in course assessments and other data sources.

Hatch, match, and dispatch: Examining the relationship between student intent, expectations, behaviours and outcomes in six Coursera MOOCs at the University of Toronto
Chris Teplovs (Learning Analytics Expert) and Stian Haklev (Institutional Researcher)

Early reflections on the experience of instructors and students have led us to consider the interplay of intent and expectations (corresponding to the “hatch” aspect of the title), behaviour, and outcomes (“dispatch”) more closely. Our primary goal in this research project was to use survey, clickstream, and assessment outcome data from the University of Toronto MOOCs that have been offered to understand how those dimensions interact (the “match” aspect). This presentation will focus on some of the methods used to analyze data from these courses and some of the emergent findings.

Should I Stay or Should I Go Now? Efforts, Analyses and Revelations Related to “Retention” in MOOCs
Steve Joordens (Dept. of Psychology)

Survey data on motivation and personality were collected using existing valid and reliable instruments, to examine the influence of these traits on MOOC learning and course performance. In particular, dimensions of intrinsic and extrinsic motivation are explored, in addition to “Big 5” personality traits.

For more information contact Stian Haklev, Institutional Researcher for the Open Utoronto initiatives, or Laurie Harrison, Director of Online Learning Strategies.