Open UToronto

Open UToronto MOOC Initiative Year Two

Open UToronto MOOC Initiative: Report on Second Year of Activity

September 2014
Laurie Harrison

[ Download the pdf version of this report ]

The University of Toronto is recognized as a Canadian leader with regard to our exploration of the potential of Massive Open Online Courses (MOOCs) over the past two years. Under the umbrella of the broader Open UToronto initiative, partnerships with both Coursera and EdX have been leveraged to stimulate activity and discussion related to the shifting landscape of online learning. During the first year of engagement with this emergent pedagogical model a clearer understanding of the relationship of the MOOC to our existing practice and process was established. Practical experience, enhanced by an active community of practice, informed our development of institutional support strategies, as described in our Open UToronto MOOC Initiative: Report on First Year of Activity which was published in July 2013. At that time we confirmed our institutional strategy and commenced further exploration, seeking to address the following goals:

·      Contribute to the education community and the broader public through sharing of institutional expertise and open curriculum content

·      Explore and evaluate a range of pedagogical approaches and open course platforms

·      Leverage use of open educational resources through integration into University of Toronto degree program courses

·      Showcase the University of Toronto’s capacity as a leading institution for teaching, learning and research

Our aim has been to advance innovation in online learning through development of new models, methods and instructional strategies that will have a positive impact both within our institution and on the broader global stage. 

As we commenced our second year of exploration, based on input from our advisory committee and feedback from the MOOC instructor community, several target activities provided a focus for evolution of our work in this arena. Our identified priorities were as follows:

1. Continue to explore the potential of emergent pedagogical models both inside the institution and beyond.

2. Provide design support and team-based development strategies to build capacity and ensure the quality of our online learning initiatives.

3. Engage in research and evaluation activities to gather evidence and inform our future planning related to MOOCs and online learning.


The activities and progress in supporting each of these three targets are highly complementary. Our Open UToronto initiative has acted as a catalyst for exploration of new online learning terrain within the MOOC model and beyond as follows:

Priority 1: Continue to explore the potential of emergent pedagogical models

Discussion of the potential uses of the MOOC model has catalyzed many fruitful discussions regarding online teaching and learning strategies and their impact on learners both inside our classrooms and within our online communities. Following what the New York Times called “The Year of the MOOC” (Pappano, 2012), by the fall of 2013 the conversations moved beyond the initial excitement into a stage of exploration and integration. While the media discussion of the “disruption” and pending “tsunami” receded, instructors and researchers continued the exploration of “connection,” “pedagogy,” and “hybrid options” (Stewart, 2013). Online learning is not new, yet MOOCs introduced unique elements to the instructional repertoire at the University of Toronto. Characteristics that have been of particular interest include flexible access to open course resources and platforms; scalability of online learning instructional strategies; and outreach beyond our institutional borders to the broader community.

One of the unique opportunities provided through our MOOC partnerships was the possibility of experimenting with these new elements in an innovation “sandbox.” All of our MOOC courses have begun from a point of exploration, outside of our degree-credit programs. New models of content presentation, learner engagement and practice, formative assessment and peer-basedlearning have been first employed using the design utilities available in our MOOC partner hosted environments before any pilot activities were introduced within the institution’s academic program offerings. Further, having partnerships with both Coursera and EdX has allowed us the possibility to compare the two platforms, as they take quite different fundamental approaches to pedagogical process design. The Coursera course design is built around a core set of video resources, while EdX format takes the form of a learning activity sequence approach.

As our second year of the MOOC initiative progressed, it became evident that our instructors each had different target audiences, different goals, and different subject or discipline specific needs. An informal review of the design features of UofT MOOCs identified a range of approaches, correlating with the learning context and aims of the instructor. In particular, instructors in the Science, Technology, Engineering and Math (STEM) disciplines and introductory survey courses tended to focus on content development and follow degree program course curriculum closely. These instructors more commonly explored flipped classroom models using their MOOC materials. In other cases, MOOCs addressed goals related to a research program or advocacy activities. These instructors, mainly from the social sciences, incorporated design plans that focused on the learning process framed by community interaction and outreach. A review of content and assessment methods shows that our STEM and intro survey MOOCs tended to use a mastery learning pedagogy focusing on content production (video and assessment tools). In comparison, research and outreach-oriented MOOCs tended to focus on process to engage students socially as a learning community (lecturettes in combination with active learning strategies). See distribution of graded assessment types and forum participation rates across all MOOCs in Appendix 1 and 2.

Learning Design

New MOOCs offered and in development during 2013-14 were notable for the diversification of discipline area, with more specialized topics and advanced offerings as follows:

Title

Instructor

Division

Platform

Bioinfomethods I

Nicholas Provart

Faculty of Arts & Science

Coursera

Bioinfomethods II

Behavioural Economics in Action

Dilip Soman

Rotman School of Management

EdX

Our Energetic Earth

Bryan Karney

Faculty of Applied Science and Engineering

EdX

Library Advocacy Unshushed

Wendy Newman

Faculty of Information

EdX

In Development

 

 

 

Measuring Global Causes of Death

Prabhat Jha

Dalla Lana School of Public Health

EdX

Alternative Energy Systems

Jim Wallace

Faculty of Applied Science & Engineering

Coursera

In addition to new MOOCs, two of the earlier offerings were repeated in summer of 2013. Data on registration and completion during the second year of the MOOC initiative has been aggregated in the table below.

Aggregated Data for Year 2 of the UToronto MOOC Initiative

 

Total MOOC registrants

174571

Unique persons who watched at least one video or completed a quiz

109581

Completed for certificate

13395

Percentage of active registrants who completed for certificate

8%

Percentage video watchers /quiz takers who completed

15%

Note: Cumulative detailed data for Year 1 and 2 are provided in Appendix 3 and 4.

A research study is currently underway to provide deeper understanding of the MOOC instructors’ experience of designing and instructing in these new environments. MOOC instructors are being asked to share reflections on their motivations, design processes, instructional strategies and any impacts on their teaching practice in general. The study will be published in fall 2014.

Priority 2: Provide design support and team-based development strategies

Our early MOOC instructors launched their projects with short timelines and limited institutional experience to inform development of instructional strategy and production methods. While basic MOOC development guidelines related to resourcing strategies, work flow and design considerations were drafted in the first year, the second cohort of instructors who developed MOOCs for the EdX platform benefited considerably from the work of those first pioneers. The EdX instructor community formed around shared interests and concerns, with a 2013 MOOC workshop providing principles of online learning design and a framework for establishing alignment of learning outcomes, activities and assessments. The MOOC Design Round Table, coordinated by the Online Learning Strategies portfolio, was repeated again in 2014 for the most recent cohort of new instructors, representing teams interested in both EdX and Coursera platforms.

The team processes have been supported by library experts as well as educational technology professionals. We have learned that it “takes a village” to build a MOOC, which in turn requires that instructors make their design goals and plans visible and open to input from their colleagues. Anecdotal reports suggest the instructors appreciate the opportunity to tackle the design phase by participating in a guided workshop process. Elements of the Course Re/Design Institute offered annually by the Centre for Teaching Support & Innovation have been adapted for the MOOC context. Our experience has been that tried and true methods can effectively inform exploration in the new scaled environment. Introduction of foundational concepts such as “backward design” (Wiggins, 1998) and “significant learning” (Fink, 2003) has proved a successful strategy for pushing the boundaries beyond video presentation elements to effectively engage learners. The MOOC design process provides an entry point to applying principles of effective course design in a highly exploratory context, resulting in the twin benefits of increased awareness of course design principles, plus innovation in the online learning environment at scale.

Many of the MOOC instructors are award-winning faculty members who have considerable influence among their peers, both within their academic programs and across divisions. The spotlight on the MOOC initiative in combination with our other online learning projects drew considerable interest from many advisory committees and governance bodies within the institution. The internal audiences for presentations by these outstanding instructors included the Academic Board – Committee on Academic Policy and Programs; Teaching, Learning and Technology Advisory Committee; and the Open UToronto Advisory Committee. The formation of a network of instructors, supported by an informal community of practice, represents a significant gain in expertise and capacity within the sphere of online instruction at the University of Toronto.

Priority 3: Engage in research and evaluation activities

Scholarly research activity related to the learner characteristics, participation and outcomes accelerated during our second year of MOOC exploration, as several projects proposals received external funding. The Introduction to Psychology MOOC, as well as Making Sense of Data had already benefited from grant resourcing through the Bill & Melinda Gates Foundation that included a research component. In the fall of 2013, three additional Gates research projects were funded as part of the MOOC Research Initiative, coordinated by Athabasca University:

·      Beyond and Between “Traditional” MOOCs: Agile and Just-in-Time Learning
Jennifer Campbell and Alison Gibbs

·      Secondary School Students and MOOC’s: A Comparison between Independent MOOC Participation and Blended Learning
Dilip Soman and Rosemary Evans (UTS)

·      Hatch, match, and dispatch: Examining the relationship between student intent, expectations, behaviours and outcomes in six Coursera MOOCs at the University of Toronto
Carol Rolheiser, Chris Teplovs, Stian Haklev, Hedieh Najafin and Bodong Chen

These and other research initiatives were supported through development of an institutional MOOC data repository service. As part of the Open UToronto initiative, in collaboration with the Academic and Collaborative Technology portfolio, infrastructure was developed to support data analytics, research and evaluation projects using MOOC databases and clicklog archives. The service includes provision of cleaned and formatted data samples to researchers, while ensuring ethics protocols are addressed. Securelong-term storage is available for the purpose of research and evaluation.

As a result of the increased levels of research activity that occurred during the 2013-2014 academic year, community interest converged in a coordinated event to showcase this scholarly work. In April of 2014, Online Learning Strategies portfolio and the Centre for Teaching Support & Innovation co-hosted a MOOC Research Symposium featuring presentations of findings by five research teams. The event drew approximately 70 attendees, with a high level of discussion and engagement occurring throughout the day.

In addition to the Gates-funded research, our MOOC instructors and their colleagues have published a number of other papers. A collection of these MOOC research publications is posted on the Open UToronto web site, including two papers examining the use of MOOC materials in the inverted classroom model. Of particular interest was the finding from the Computer Science discipline indicating that learning as measured by final exam performance increases significantly when this format is used (Horton, Craig, Campbell, Gries, & Zingaro, 2014). A full list of publications is provided in Appendix 5.

Resourcing of institutional research support, including data storage and curation has been essential to our scholarly achievements related to use of the MOOC model. Several additional research projects are currently in progress, on topics including:

·      Evidence of development of higher order statistical skills in statistical literacy, reasoning, and thinking using broadly accepted evaluation frameworks, specific to the discipline area;

·      Impact of MOOC experience on perceptions and sense of efficacy in the sphere of library advocacy

·      Understanding the experiences and impact of instructors participating in the development and running of MOOCs; and

·      Use of analytics to better understand the relationship between student intent, expectations, behaviours and outcomes. 

Looking ahead to the Coming Year

As we look ahead to the coming year, we see the opportunity for continued exploration of a number of facets of online learning through the Open UToronto MOOC initiative. A key area where the MOOC model provides a catalyst for reflection and innovation is the use and re-use of digital content, including:

o Creation of on-demand open course content resources for flexible access;

o Re-use of MOOC content in differentiated fully online degree-creditcourses (enhanced through additional interactivity and instructor contact)

o Continuing exploration of a range of models beyond the “traditional” MOOCs, including flipped classrooms, outreach initiatives and research.

In order to support UofT’s broader strategic targets, several target actionables have been identified for the coming year:

o Piloting of University of Toronto alumni cohort MOOCs in collaboration with Alumni Relations.

o Using MOOC content to support additional flipped classroom active learning opportunities in gateway courses.

o Partnering with peer institutions in design and development efforts, either through community of practice activities or complementary MOOC offerings.

o Exploring the possibility of a revenue stream through non-degree program verified certificates.

In summary, we anticipate further extension of the types and uses of MOOCs through our continued partnership with Coursera and EdX in the 2014-2015 academic year. Our activities will include development of new MOOCs, as well as exploration of the potential for re-use of existing resources. The University of Toronto will continue to take a leadership role in the emerging research on scalable online learning, re-use of content in a range of instructional contexts, and application of learning analytics to improve the effectiveness of pedagogical design.

For more information on Open UToronto MOOC initiatives contact Laurie Harrison

References:

Stewart, B., 2013. What’s Next? After the MOOC Hype. Presentation at MOOC Research Initative Conference, December 2013. Retrieved from:http://www.slideshare.net/bonstewart/mooc- research-conference

Fink, L. D. (2003). Creating significant learning experiences: An integrated approach to designing college courses. San Francisco, CA: Jossey-Bass.

Horton, D., Craig, M., Campbell, J., Gries, P., & Zingaro, D. (2014). Comparing outcomes in inverted and traditional CS1 (pp. 261–266). ACM Press.

Pappano, L. (2012). The Year of the MOOC. The New York Times. Retrieved fromhttp://www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html?pagewanted=al

Wiggins, G. (1998). Educative Assessment: Designing Assessments to Inform and Improve Student Performance. San Francisco, CA: Jossey-Bass.

 


Appendix 1: Assessment Types as Percentage of MOOC Grading Scheme

MOOC

Quizzes,
Tests and Auto-graded Assignments

Peer

assessed

Self

assessed

Participation

Total

Possible

Learn to Program: The Fundamentals

100%

0

0

0

100%

Neural Networks for Machine Learning

100%

0

0

0

100%

Learn to Program: Crafting Quality Code

80%

20%

0

0

100%

 Introduction to Psychology

90%

10%

0

0

100%

 Our Energetic Earth

85%

15%

0

0

100%

Statistics: Making Sense of Data

70%

20%

0

0

100%

Aboriginal Worldviews and Education

40%

50%

0

10%

110%

Behavioural Economics in
Action

40%

40%

20%

0

100%

 Library Advocacy Unshushed

20%

0

80%

0

100%

The Social Context of Mental Health and Illness

10%

100%

0

0

110%

 

Appendix 2: Number of Forum Posts

 

MOOC

Total Posts

Active 
Participants*

# Posts per Active Participant

Bioinformatic Methods 1

2796

13396

0.2

Statistics: Making Sense of Data

5224

19757

0.3

Learn to Program: Crafting Quality Code

4709

17224

0.3

Bioinformatic Methods 2

1133

4058

0.3

Learn to Program: The Fundamentals

23055

75450

0.3

Neural Networks for Machine Learning

5192

15903

0.3

Our Energetic Earth

3003

4732

0.6

Introduction to Psychology

29461

41708

0.7

Behavioural Economics in Action

22364

20019

1.1

The Social Context of Mental Health and Illness

13289

8193

1.6

Aboriginal Worldviews and Education

34112

8860

3.9

Library Advocacy Unshushed

6278

1519

4.1

Appendix 3 – Cumulative MOOC Data for Year 1 & 2

Data on registration and completion during the first and second year of the MOOC initiative is as follows:

Activity and Completion Stats

Total MOOC
Registrants

Active
Participants*

Completed
for certificate

Registrants
completing certificate

Active
participants
completing certificate

Learn to Program: The Fundamentals

80,000

75450

8240

10%

11%

Neural Networks for Machine Learning

49550

15903

1398

3%

9%

The Social Context of Mental Health and Illness

23491

8193

1423

6%

17%

Aboriginal Worldviews and Education

20966

8860

3381

16%

38%

Learn to Program: Crafting Quality Code

53974

17224

3352

6%

19%

Statistics: Making Sense of Data

48687

19757

2825

6%

14%

Introduction to Psychology

77608

41708

3691

5%

9%

Behavioural Economics in Action

43042

20019

1154

3%

6%

Our Energetic Earth

10857

4732

458

4%

10%

Bioinformatic Methods I

21072

13396

1540

7%

11%

Bioinformatic Methods II

9,529

4058

1036

15%

26%

Library Advocacy Un-shushed

5,197

1519

363

7%

24%

Second Offering

 

 

 

 

 

Learn to Program: The Fundamentals

66510

56397

7839

11%

14%

The Social Context of Mental Health and Illness

18364

9460

1005

6%

11%

Total

528,847

296,676

37,705

8%

16%

 * Watched video or took quiz

Appendix 4 – Cumulative MOOC Data on Archive Usage Year 1 & 2

Data on registrations in MOOCs available in archive mode during the first and second years of the MOOC initiative is as follows.

Title

Registration at end of Session

Total Registration
to Date

Archive Users

Learn to Program

80,000

139852

59852

Neural Networks for Machine Learning

49550

97077

47527

The Social Context of Mental Health and Illness

23491

34060

10569

Aboriginal Worldviews and
Education

20966

31634

10668

Learn to Program: Crafting Quality Code

53974

87995

34021

Statistics: Making Sense of Data

48687

82721

34034

Introduction to Psychology

77608

93495

15887

Behavioural Economics in Action

43042

50349

7307

Our Energetic Earth

10857

13266

2409

Bioinformatic Methods I

21072

24415

3343

Bioinformatic Methods II

9529

11905

2376

Library Advocacy Unshushed

5197

5208

11

Second Offerings

 

 

 

Learn to Program: The Fundamentals

66510

97019

4847

The Social Context of Mental Health and Illness

18364

19862

1498

Total

528847

788858

234349

Appendix 5 – MOOC Publications and Research

Publications

Horton, D., Craig, M., Campbell, J., Gries, P., & Zingaro, D. (2014). Comparing Outcomes in Inverted and Traditional CS1. In Proceedings of the 2014 conference on Innovation & technology in computer science education (ITiCSE 2014). ACM, New York, NY. (pp. 261- 266). DOI 10.1145/2591708.2591752

Campbell, J., Horton, D., Craig, M., & Gries, P. (2014). Evaluating an Inverted CS1. In Proceedings of the 45th ACM technical symposium on Computer science education (SIGCSE 2014). ACM, New York, NY. (pp. 307–312). DOI 10.1145/2538862.2538943

Gibbs, A. (2014). Experiences Teaching an Introductory Statistics MOOC. In Proceedings of the Ninth International Conference on Teaching Statistics (ICOTS9, July, 2014). Voorburg, The Netherlands: International Statistical Institute. Access the paper.

Restoule, J.P. (2013) . Massive Open Online Courses and the Future of Adult Education. In C. Kawalilak & J. Groen (Eds.) Proceedings from Canadian Association for the Study of Adult Education (CASAE) 2013 (pp. 514-520). Access the paper.

MOOC Research Initiative Presentations

Campbell, J., Gibbs, A., & Haklev, S. (2013, December). Beyond and Between “Traditional” MOOCs: Agile and Just-in-Time Learning. Paper presented at the MOOC Research Conference, University of Texas, Arlington.
Access the paper.

Rolheiser, C., Harrison, L., Haklev, S., & Teplovs, C. (2013, December). Hatch, match, and dispatch: Examining the relationship between student intent, expectations, behaviours and outcomes in six Coursera MOOCs at the University of Toronto. Paper presented at the MOOC Research Conference, University of Texas, Arlington.
Access the paper.

Soman, D., Evans, R., & Federico, C. (2013, December). Secondary School Students and MOOC’s: A Comparison between Independent MOOC Participation and Blended Learning. Paper presented at the MOOC Research Conference, University of Texas, Arlington.
Access the paper.