Open UToronto

Open UToronto MOOC Initiative Year One

Open UToronto MOOC Initiative: Report on First Year of Activity

August 2013
Laurie Harrison

[ Download the PDF Version of this Report

The University of Toronto took the lead among Canadian universities, beginning our exploration of the Massive Open Online Course (MOOC) arena in the summer of 2012. With little research or data available on this emergent phenomenon, a strategy of learning through the experience of design and implementation was adopted in order to inform our understanding of the potential impact both within our institution and beyond. Under the umbrella of the broader Open UToronto initiative, the following overarching goals were identified: 

◦      Contribute to the education community and the broader public through sharing of institutional expertise and open curriculum content

◦       Explore and evaluate a range of pedagogical approaches and open course platforms

◦       Leverage use of open educational resources through integration into University of Toronto degree program courses

◦       Showcase the University of Toronto’s capacity as a leading institution for teaching, learning and research

◦       Advance global innovation in online learning through development of new course structures, methods and instructional strategies

The University of Toronto currently has partnerships with two MOOC platform providers. In July 2012 an agreement was put in place with Coursera, an education company that provides a specialized MOOC platform at no cost to the university. A second partnership arrangement was undertaken with the EdX initiative, a non-profit created by founding partners Harvard and MIT. In both cases the learning environment is hosted by the partner organization, and course content design and instruction is the responsibility of the University of Toronto.

To date, seven Coursera MOOCs have been offered across a range of discipline areas. These short, fully online “coursettes” ranged from 4 to 8 weeks in length, and have attracted more than 500,000 registrants before, during and after the scheduled sessions.  Seven more MOOCs, including new EdX-hosted offerings are currently in the design phase.

The following is a summary of MOOCs that have been offered, scheduled or are in development.

MOOCs Offered

Description

Platform

Sessions

Learn to Program: The Fundamentals

Jennifer Campbell and Paul Gries

Department of Computer Science

 Coursera

Oct. 2012

 Aug. 2013

Neural Networks for Machine Learning

Geoffrey Hinton

Department of Computer Science

 Coursera

Oct. 2012

 

The Social Context of Mental Health and Illness  

Charmaine Williams

Factor-Inwentash Faculty of Social Work

 Coursera

Jan. 2013

 June 2013

Aboriginal Worldviews and Education

Jean-Paul Restoule

Ontario Institute for Studies in Education

 Coursera

Feb. 2013

 

Learn to Program: Crafting Quality Code

Jennifer Campbell and Paul Gries

Department of Computer Science

 Coursera

March  2013

 

Statistics: Making Sense of Data

Alison Gibbs and Jeffrey Rosenthal

Department of Statistics

 Coursera

April 2013

 

Introduction to Psychology

Steve Joordens

Psychology – UT Scarborough

 Coursera

May 2013

 

MOOCs Scheduled in Next Six Months

Description

Platform

Sessions

Behavioral Economics in Action

Dilip Soman

Rotman School of Management

 EdX

Oct. 2013

Our Energetic Earth

Bryan Karney

Faculty of Engineering


 EdX

Oct. 2013

Death 101: Measuring Global Causes of Death

Prabhat Jha

Dalla Lana School of Public Health


 EdX

May 2014

MOOCs Currently In Proposal Development

Description

 Platform

Sessions

BioInformatic Methods I

BioInformatic Methods II

Nicholas Provart

Department of Cell and Systems Biology

 Coursera

TBC

The Logic of Business: Building Blocks for Organizational Design

Mihnea Moldoveanu

Rotman School of Management

 EdX

TBC

Library Advocacy

Wendy Newman

iSchool

 EdX

TBC

Patterns of Activity:

Of the total 366,424 registrants during Coursera MOOC sessions completed to date, 207,566 accessed video content, thus 57% may be considered to be active learners. Many of these learners were curious about the subject area, browsed for information on particular topics or explored the MOOC format in general, taking advantage of the ease of registration to participate to the level appropriate to their needs and interests. The aggregate completion rate for Coursera MOOCs offered to date is 8% for all registrants, and 17% for active learners, which is well within the normal range for this mode of delivery. A total of 6 Coursera MOOCs are currently complete. 

MOOC Activity Summary Data for Completed Scheduled Sessions – July 25, 2013

Activity and Completion Statistics

total registrants

active learners*

posts in discussion 

successfully completed

percentage registrants who completed

percentage active learners who completed

Complete

 

 

 

 

 

 

Learn to Program: The Fundamentals

 

80000

75450

23055

8240

10%

11%

Neural Networks for Machine Learning

 

49550

15903

5192

1398

3%

9%

The Social Context of Mental Health and Illness

 

23491

8193

13289

1423

6%

17%

Aboriginal Worldviews and Education

 

20966

8860

34112

3381

16%

38%

Learn to Program: Crafting Quality Code

 

53974

17224

4709

3352

6%

19%

Statistics: Making Sense of Data

 

62488

36356

5224

2825

5%

8%

Total

366424

115042

207566

17794

8%

17%

* an active learner is person who watched at least one video

An unexpected ongoing pattern of activity is learner registration and use of MOOC materials after the scheduled session is completed. An additional 136,008 users have registered and are seen to be active in Coursera MOOCs after the close of scheduled MOOCs. Additional research is needed to understand the implications of this activity and is being taken up by faculty researchers.

MOOC Summary Data During Archive Period – July 25, 2013

MOOC archive

registration at end of session

total registration to date

# archive users

Learn to Program: The Fundamentals

80000

139852

59852

Neural Networks for Machine Learning

49550

77464

27914

The Social Context of Mental Health and Illness

23491

34060

10569

Aboriginal Worldviews and Education

20966

30056

9090

Learn to Program: Crafting Quality Code

53974

76945

22971

Statistics: Making Sense of Data

62488

68100

5612

Total Archive Users

 

 

136008

Research Program

To further our understanding of these new online course formats, Online Learning Strategies (OLS), together with the MOOC instructors, have developed an extensive research program around MOOCs and inverted (flipped) classrooms. Some of the topics being investigated include:

  • Student demographics and learning goals/approaches
  • Range of pedagogical approaches that are available to instructors
  • Patterns of engagement and learning among participants. 

The faculty members contributing to this research program to date are:

Two of these courses (Statistics: Making Sense of Data, and Introduction to Psychology) have received special support from the Gates Foundation to research their MOOCs. The Statistics MOOC is also being used in another Gates funded project, a multi-campus study of MOOCs as a deeply integrated instructional resource for blended learning at universities across the Maryland system.

Current Data Analysis Activities and Studies

Quantitative Data Analysis: The Coursera platform provides rich data on learner interaction with the courses, and we will complement this learning analytics data with data on student demographics and learning intents from intro- and exit-surveys. The OLS is coordinating individual research projects by the different PIs, and facilitating access to data from Coursera, as well as compiling general demographic data now available. See Appendix A: Demographic Report on University of Toronto Coursera MOOCs.

Qualitative Analysis of Instructor Experience and Design Outcomes: In addition to the research on learners, and student interactions with courses, we are also interested in instructors, their experiences and attitudes, and the process of constructing courses. OLS has launched a separate research project, supervised by Dr. Carol Rolheiser from the Center for Teaching Support and Innovation (CTSI), which aims to interview all UofT MOOC instructors about their experience of building and running a MOOC, as well as analyzing the course designs and pedagogical approaches of the various courses.

A full institutional research report is anticipated in August 2013. As well, research studies being undertaken by individual faculty members are forthcoming in the near future. Online Learning Strategies is facilitating a MOOC research group that is working collaboratively on joint studies, development of protocols for ethics review and discussion of data management and analysis strategies. 

Design Process:

Given that both our Coursera and EdX partnerships are institutional initiatives, each of the MOOCs proposed for development at the University of Toronto is first reviewed by the divisional Dean, and also endorsed by the Office of the Provost. Advice on governance and process is provided by the Open UToronto Advisory Committee. A workflow for developing an open online course was developed and published in February of 2013. This documentation along with additional resources to support planning and design is available on the Open UToronto web site:

◦       Overview of Approaches to Open Course Content

◦       MOOC Resourcing and Planning Guidelines

◦       MOOC Design and Development Guidelines

◦       MOOC vs. Degree Courses – FAQs

Our design process relies heavily on a team-based process. Typical teams include the instructor(s), educational technology leads, librarians, AV technicians and TAs. While development teams meet regularly to discuss design and production plans, we have also held regular “round table” meetings where all teams could meet together to discuss ideas and progress.

Of particular interest was the EdX Design Workshop day, when design teams gathered at CTSI)for a day of learning, collaborative planning and discussion of instructional challenges in the MOOC environment. Two guests from EdX visited from Boston for the day to provide coaching on use of the EdX tool kit and provide advice on MOOC video production strategies. The event was loosely based on the Course Design Institute model that is offered annually by CTSI. The EdX MOOC cohort is using and adapting well-established design theories and models to the MOOC environment. These include:

◦       Backward Design

Wiggins, G. P., & McTighe, J. A. (2005). Understanding by design. ASCD.

◦       Community of Inquiry

Akyol, Z., & Garrison, D. R. (2013). Educational Communities of Inquiry: Theoretical Framework, Research and Practice. IGI Global.

◦       Best Practice Models for e-learning

Staffordshire University. (n.d.). Best Practice Models for e-learning. Retrieved July 31, 2013, from http://learning.staffs.ac.uk/bestpracticemodels/

Each MOOC team is responsible for the design and structure of content and learning activities, specific to their discipline and learner needs. In each of our MOOCs, the learning design undertakings extend beyond provision of lecture material.  Individualized activities, discussion frameworks, assessment processes and community building have been the focus of exploration, pushing the boundaries of constraint within the large scale of the MOOC environment.

Inverted Classroom Design

Several of our MOOCs have been, or will be used as curriculum content for use in redesign of University of Toronto degree courses for the inverted classroom or “flipped” model. These include:

◦       Learn to Program: The Fundamentals

◦       Learn to Program: Crafting Quality Code

◦       Neural Networks for Machine Learning

◦       Statistics: Making Sense of Data

◦       Behavioral Economics in Action

The inverted classroom design strategy leverages the MOOC platform to deliver course content in advance of scheduled classes. As a result, the amount of time spent on active learning strategies during scheduled meeting time can be increased. A typical format involves viewing instructor-created videos and completing online tasks or quizzes assigned as homework. Students arrive prepared to apply their new knowledge by solving problems or doing practical work together in the classroom, guided through the process by the instructor. Other possible benefits are use of the materials prior to a course commencing, to ensure all students are familiar with prerequisite content and skills, or during a course for remediation on any challenging concepts or lessons.

Three of our faculty members in the Faculty of Arts and Science have research studies underway that are specifically focused on learning more about the potential of the inverted classroom model. Our involvement in the development of MOOCs has sparked broader interest in both the potential of online instructional design strategies as well as innovative approaches to campus-based degree course curriculum.

Next Steps:

We are moving from the early pioneering phase into our second year of exploration of the potential of the MOOC model. Our faculty and teams are well informed by our experience to date, but all agree that we have much to learn within the fast-shifting MOOC landscape. We are now entering a period where MOOCs are being offered for a second time, with adjustments and adaptation based on student and instructor feedback. Six new MOOCs are in the design phase, with registrations rolling in for the most recently scheduled sessions. Research teams are engaged in data gathering and analysis. As we move into this next period, our strategy going forward has three underlying principles:

◦       Learn about the potential of emergent online models from the leading edge of design and instruction innovation.

◦       Provide design support and team-based development strategies to build capacity and ensure the quality of our online learning initiatives.

◦       Engage in research and evaluation activities to gather evidence and inform our future planning related to MOOCs and online learning.

Acknowledgements:

This report reflects the collective achievement of all those involved in supporting design and implementation of our MOOCs at the University of Toronto. Colleagues whose continuing commitment to ensuring the highest quality experience for our learners include our award-winning faculty, instructional design teams, library colleagues, educational technology professionals, teaching assistants, advisory committee members and many others who have contributed to our success.

Appendix A: Demographic Report on University of Toronto Coursera MOOCs

Prepared June 16, 2013 by Stian Haklev

Background

In order to support MOOC-related research and evaluation processes, the Office of Online Learning Strategies is involved in administrating access to detailed user data sets from Coursera and cleaning it for analysis. The data is being stored centrally and made available for a range of purposes, including Institutional planning processes. It will also be used in a research study among five MOOCs that launched between January and May 2013. The course instructors are all co-PIs on the ethics proposal, and will share the analytics data. For a full description of activities visit: Open Utoronto MOOC Research and Evaluation.

The data

In addition to the intro- and exit-surveys, for students who choose to respond, we also have access to learning analytics data for all students in the chosen courses, including when students log on, their activity, their success at quizzes, etc. The data is anonymized, but we are able to link intro- and exit-surveys with student activity for each individual student. The co-PIs are planning to explore the relationships between different variables, but for this initial institutional report, we are simply reporting some of the key variables from the intro-survey.

The survey

A generic intro- and exit- survey was developed, and offered to students in all courses (with some additional questions and categories for specific courses). This survey, with adaptation as appropriate, will be used in all future UofT MOOCs. Due to the timing of the ethics proposal, the survey was offered to participants in The Social Context of Mental Health and Illness, and Aboriginal Worldviews and Education quite late in the course, which can explain the low response rate.

MOOC Courses

Course name

Instructor(s)

Enrolment

Response rate

The Social Context of Mental Health and Illness

Charmaine Williams

 

22.500

6%

Aboriginal Worldviews and Education

Jean-Paul Restoule

20.600

8%

Learn to Program: Crafting Quality Code

Paul Gries

Jennifer Campbell

54.700

17%

Statistics: Making Sense of Data

Allison Gibbs

Jeffrey Rosenthal

51.500

34%

Introduction to Psychology

Steve Joordens

60.700

32%

Tables reflecting summary data collected intro surveys for these five MOOCs are provided below. For more information contact Stian Haklev, Institutional Researcher for the Open Utoronto initiatives. 

 

User Categories

Graph of User Categories

 

Education Levels

Graph of Education Levels

 

Age Groups

Graph Displaying Age Groups

 

Why Students Enrolled

Graph depicting the various reasons why students enrolled in a MOOC

 

Gender

Graph showing Gender distribution in the MOOCs

 

Level of English

Graph showing the level of English for MOOC users