EVALUATION OF E-LEARNING OF ANIMATION COURSES USING ATTITUDE, BEHAVIOUR AND OUTCOMES
AARATI PRAKASH KHARE
INSTRUMENT DESIGN AND DEVELOPMENT CENTRE INDIAN INSTITUTE OF TECHNOLOGY DELHI
AUGUST 2019
© Indian Institute of Technology Delhi (IITD), New Delhi, 2019
EVALUATION OF E-LEARNING OF ANIMATION COURSES USING ATTITUDE, BEHAVIOUR AND OUTCOMES
by
AARATI PRAKASH KHARE
INSTRUMENT DESIGN AND DEVELOPMENT CENTRE
Submitted
In partial fulfilment of requirements of the degree of Doctor of Philosophy to the
INDIAN INSTITUTE OF TECHNOLOGY DELHI
AUGUST 2019
Dedicated to the goddess of knowledge and to my family.
i
CERTIFICATE
This is to certify that the thesis titled, “Evaluation of e-learning of animation courses using attitude, behaviour and outcomes”, being submitted by Mrs. Aarati Prakash Khare to the Indian Institute of Technology Delhi for the award of the degree of Doctor of Philosophy has been carried out under my supervision.
The research work contained in this thesis has not been submitted in part or full to any other university or institute for the award of any degree or diploma.
Prof. Jyoti Kumar Assistant Professor Department of Design, Indian Institute of Technology Delhi
ii
ACKNOWLEDGEMENTS
My gratitude to Almighty for his blessings. I would like to thank my supervisor, Prof. Dr. Jyoti Kumar for his help through the years. I am also thankful to Prof. P.V.M. Rao, Prof. G.S. Khan, Prof. Sumer Singh and Prof. P.M. Pandey for continuous feedback and guidance.
My thanks are due to my lab mates Abhishek Dahiya, Naveen Bharadwaj, Surbhi Pratap, Sunny Bairisal and Abhijit Kujur. My special thanks to Jyotish Kumar and Naveen Bharadwaj for their assistance in conducting the experiments.
My gratitude cannot be complete without thanking my family. My passion for gaining knowledge was inculcated by my late grandmother Smt. Rukmini Mainkar and my late father Shriram Vitthal Mainkar. Further, my younger brother late Shantanu Mainkar and my mother- in-law late Smt. Kushala Davi nurtured my zeal through their support. I am much thankful to them today as I submit my Ph.D. thesis. I also wish to thank my mother Smt. Jayshree, for her immense help and taking care of the family when I had to devote my time for research work.
I would like to thank my husband Sri Rajeev Prakash for supporting me to pursue a Ph.D.
Special thanks to my niece Shubhra Prakash for help and dearest daughter Soumya Prakash, for being my strength, courage and support. Thanks also to all my relatives and extended family for their help and support.
Mrs. Aarati Prakash Khare
iii
ABSTRACT
Among the numerous usages of computers today, e-learning is emerging as an important use for mass education at low costs (Mason, 1998). Internet-based online courses have started facilitating all levels of education, including elementary, secondary and technical, etc. Even online design courses in general and animation design courses in particular, have started. Due to heavy demand from the animation industry, online courses in animation are highly sought after by youngsters seeking job opportunities in the animation industry (Ferdin, 2019, July 12). There have also been reports of high drop off rates of online courses (Woodley, 2004).
This thesis has identified the need to evaluate online learning courses, especially for design courses like animation where tacit knowledge is involved (Belas, 2017). This thesis has identified a gap in the literature that there is a lack of learning evaluation models for online courses and has proposed a new model for the evaluation of online courses. The evaluation model proposed in this thesis consists of six stages. The model starts with an evaluation of attitude towards e-learning, proposes to measure initial reaction towards the e-learning platform design, measure the ease of use of the learning platform, measure the engagement a learner has felt while exploring and learning from the e-learning platform, measure the learning the learner has undergone and finally evaluate the skill the learner has acquired. The proposed model has then been used to investigate learners' attitudes, behaviours, and outcomes through experimental studies.
The Learner’s attitude was evaluated through semantic differential scales of 20 bipolar adjectives with 482 participants and it was found that there exists a difference in learner’s attitude towards online and offline methods of learning. Next, the e-learning platform was designed and developed with seeded usability problems. Participants were invited in lab experiments to undergo learning using the platform. During the usage of the e-learning platform, Galvanic Skin Response (GSR) and behavioural data were collected, also before and after samples of drawing skills were collected. The findings suggest that difficulty in the usage of the e-learning platform directly affects the engagement levels of the participants and learning outcomes from the e-learning exercise.
So far the models of e-learning evaluation have not explicitly argued for the need to evaluate the quality of online learning platforms along with the learning content. The proposed evaluation model in this thesis has been the first attempt reported in the literature.
iv
The findings of this thesis have illustrated how the usability of the e-learning websites directly affects the learning of the students from the website. The findings of this thesis have further established the need to evaluate e-learning platforms differently than the existing learning evaluation methods due to this dependence of learning about the learning media itself.
Keywords: e-learning, evaluation model, Information System Design (ISD), HCI, Usability, User Experience Design.
सार
आज कंप्यूटर के अनेक इस्तेमाल के बीच इ लर्निंग माध्यम सार्वजर्नक र्िक्षा के र्लए कम लागत में
सबसे बेहतरीन र्र्कल्प के रूप में सामने आया है (Mason, 1998). इंटरनेट र्िक्षा पर आधाररत ऑनलाइन पाठ्यक्रम ं ने हर स्तर पर र्िक्षा सुर्र्धा देने की िुरुर्ात कर दी है र्जसमे प्राथर्मक, माध्यर्मक और तकनीकी र्िक्षा िार्मल है. यहााँ तक र्क सामान्य तौर पर ऑनलाइन र्िज़ाइन क सेज और र्र्िेष तौर पर एनीमेिन र्िज़ाइन क सेज की िुरुर्ात ह भी चुकी है. एनीमेिन क्षेत्र में कुिल युर्ाओं की
बढ़ती मांग की र्जह से एनीमेिन के ऑनलाइन क सेज उन युर्ाओं के द्वारा ज्यादा तलािे जा रहे हैं ज एनीमेिन क्षेत्र में र ज़गार की तलाि कर रहे हैं (Ferdin, 2019, July 12). कुछ ररप टटवस के अनुसार ऑनलाइन क सेज के दाम ं में भारी र्गरार्ट भी देखी गयी है (Woodley, 2004). इस थीर्सस ने ख़ास तौर पर एनीमेिन जैसे उन ऑनलाइन क सेज़ की आव्यिकता की पहचान की है र्जनमें गूढ़ ज्ञान र्नर्हत है (Belas, 2017). इस थीर्सस ने इस क्षेत्र के सार्हत्य में उस अंतर क भी र्चन्हित र्कया है र्क ऑनलाइन क सेस में पढाई के मूल्ांकन आधाररत मॉिल की कमी है और साथ ही ऑनलाइन क सेस के मूल्ांकन के र्लए एक नया मॉिल भी प्रस्तार्र्त र्कया है. इस थीर्सस में प्रस्तार्र्त मूल्ांकन मॉिल में 6 चरण हैं.
इस मॉिल की िुरुआत इ लर्निंग के प्रर्त रर्ैये के मूल्ांकन से ह ती है, यह इ लर्निंग प्लेटफामव र्िज़ाइन की ओर प्रारंर्भक प्रर्तर्क्रया मापने की प्रस्तार्ना करती है, लर्निंग प्लेटफामव के सुर्र्धाजनक इस्तेमाल का आंकलन करती है, यह इ लर्निंग प्लेटफामव पर पढ़ते समय और उसक गंभीरता से समझते समय पढ़ने र्ाले के इसके प्रर्त रुझान का आंकलन करती है, पढाई के दौरान पढ़ने र्ाले के अनुभर् ं क मापती है और अंत में पढ़ने र्ाले ने क्या सीखा इसका मूल्ांकन करती है. इसके बाद प्रस्तार्र्त मॉिल का इस्तेमाल र्कया गया तार्क पढ़ने र्ाले के रर्ैये, व्यव्हारऔर प्रय गात्मक र्िक्षा के नतीज ं का पता
लगाया जा सके.
पढ़ने र्ाले के रर्ैये का आंकलन 482 प्रर्तभार्गय ं र्ाले 20 बायप लर एिजेन्हिव्स के साथ र्समेंर्टक र्िफरेंर्ियल स्केल्स (GSR) द्वारा र्कया गया और यह पाया गया र्क पढाई के ऑफलाइन और ऑनलाइन माध्यम ं के प्रर्त पढ़ने र्ाले के रर्ैये में अंतर है. अगले इ लर्निंग प्लेटफामव क इसके इस्तेमाल में ह ने र्ाली परेिार्नय ं के साथ बनाया और र्र्कर्सत र्कया गया. प्रर्तभार्गय ं क इसके लैब एक्सपेररमेंट के र्लए बुलाया गया तार्क इस प्लेटफामव के इस्तेमाल क परखा जा सके. इस इ लर्निंग प्लेटफामव के इस्तेमाल के दौरान गैल्वेर्नक न्हस्कन ररस्ांस और व्यार्हाररक िाटा जमा र्कया गया, इसके
अलार्ा प्रय ग से पहले र् बाद में प्रर्तभार्गय ं की कला में कुिलता के नमूने भी जमा र्कये गए. ख ज में
सामने आया र्क इ लर्निंग प्लेटफामव के इस्तेमाल में आयी मुन्हिल ं ने प्रर्तभार्गय ं के इस प्लेटफामव के
प्रर्त झुकार् और इ लर्निंग एक्सरसाइज के नतीज ं क प्रभार्र्त र्कया. अभी तक इ लर्निंग मॉिल्स के
मूल्ांकन क ले कर मुखर रूप से बात नहींकी गयी थी र्क लर्निंग कंटेंट के साथ ऑनलाइन लर्निंग प्लेटफॉर्म्व की गुणर्त्ता का मूल्ांकन भी ह ना चार्हए. इस थीर्सस में प्रस्तार्र्त मूल्ाङ्कन मॉिल इस र्दिा में पहला प्रयास है.
इस थीर्सस की ख ज यह रेखांर्कत करती है र्क र्कस तरह इ लर्निंग र्ेबसाइटस का प्रय ग र्ेबसाइटस द्वारा पढ़ने र्ाले के सीखने की क्षमता क सीधे तौर पर प्रभार्र्त करता है. इस थीर्सस की ख ज इ लर्निंग प्लेटफॉर्म्व का मूलयांकन अलग तरीक ं से करने की आर्श्यकता पर ज़ र देती है क्य ंर्क मूल्ांकन के
मौजूदा तरीके लर्निंग्स और लर्निंग मीर्िया पर र्नभवर हैं.
क ुंजी शब्द (Key word): ई-लर्निंग, इवैल एशन मॉडल, इन्फॉमेशन र्िस्टम र्डजाइन (आइएिडी), एचिीआइ, यूजेर्िर्लटी, यूजर एक्सपीररएुंि र्डजाइन.
v
TABLE OF CONTENTS
CERTIFICATE ... i
ACKNOWLEDGEMENTS ... ii
ABSTRACT ………iii
TABLE OF CONTENTS... v
LIST OF FIGURES ... viii
LIST OF TABLES ... ix
GLOSSARY ... x
Chapter 1: Introduction ... 1
1.1 HCI and E-learning Systems ... 3
1.2 Pedagogy of Tacit Knowledge and Design Skills... 6
1.3 Research Questions and Research Objectives ... 7
1.4 Research Plan ... 8
1.5 Research Summary ... 8
1.6 Chapter Summaries ... 9
Chapter 2: Literature Review ... 10
2.1 Instructional Design ... 10
2.2 Learning Theories ... 12
2.2.1 Implications of Learning Theories on E-learning ... 14
2.2.2 E-Learning and Learning Styles ... 15
2.2.3 Attitude and Learning ... 17
2.3 E-Learning Pedagogy ... 18
2.4 Evaluation of Learning ... 21
2.5 Animation Industry, Academics and Pedagogy ... 23
vi
2.5.1 Curriculum of Animation Courses ... 25
2.6 Usability and User Experience measures ... 27
2.7 Summary of Literature Review ... 28
2.8 Gaps in the literature ... 29
Chapter 3: Proposal of an E-learning Evaluation Model ... 31
3.1.1 Comparison of Kirkpatrick Model and Proposed Model ... 34
3.1.2 Description of Proposed Evaluation Model ... 35
3.1.3 Attitude ... 35
3.1.4 Reaction ... 36
3.1.5 Usage ... 36
3.1.6 Engagement ... 37
3.1.7 Learning ... 38
3.1.8 Skill ... 39
Chapter 4: Experiments and Results ... 40
4.1 Experiment 1: Measurement of Attitude ... 41
4.1.1 Experiment Design ... 43
4.1.2 Participants ... 44
4.1.3 Data Analysis ... 44
4.1.4 Results ... 45
4.1.5 Discussion ... 52
4.2 Experiment 2: Measurement of Usage, Engagement and Skills ... 56
4.2.1 Experiment Design ... 56
4.2.2 Stimuli ... 57
4.2.3 Participants ... 59
4.2.4 Experimental Setup ... 59
4.2.4 Results ... 60
vii
4.2.5 Discussion ... 69
Chapter 5: Discussions and Conclusions ... 71
Research Questions and Their Answers ... 75
Research Objective and its fulfilment ... 76
5.1 Thesis contributions ... 77
5.2 Limitations of this thesis ... 78
5.3 Scope for future work ... 78
References ... 80
Appendices ... 89
APPENDIX A: SD Scale ... 90
APPENDIX B: Participant brief ... 92
APPENDIX C: Consent Form... 93
APPENDIX D: Protocol for Sketching ... 94
APPENDIX E: Lab Experiment Protocol... 95
APPENDIX F: List of Publications ... 96
Brief profile of the author ... 97
viii
LIST OF FIGURES
Figure 1: A behavioural model of Human-Computer Interaction (HCI) ... 4
Figure 2: E-learning as User Centred Design (UCD) or Learner Centred Design (LCD)? ... 5
Figure 3: ADDIE model of Instruction System Design ... 11
Figure 4: The Three Psychological Perspectives of Learning ... 12
Figure 5: Learning theories and their proponents ... 13
Figure 6: Kolb's learning style model ... 16
Figure 7: Bloom's Taxonomy in the cognitive domain, a) original b) revised ... 19
Figure 8: Bloom's taxonomy in the affective domain ... 19
Figure 9: Bloom’s taxonomy in the psychomotor domain ... 20
Figure 10: Kirkpatrick's learning evaluation model ... 21
Figure 11: Proposed e-learning evaluation model ... 33
Figure 12: Comparison of the Kirkpatrick model (a) and proposed model (b) ... 34
Figure 13: Overview of the research plan. ... 40
Figure 14: Error plot for online vs. offline semantic differential ratings ... 46
Figure 15: Plot of mean ratings for the semantic differential scale ... 48
Figure 16: Participant sketching two objects after learning from the website ... 57
Figure 17: Screenshot of e-learning website home page ... 58
Figure 18: Embedded video on the website created to teach scale and proportion ... 58
Figure 19: Seeded navigation difficulty on the e-learning website ... 59
Figure 20: Experimental setup for e-learning website usage ... 60
Figure 21: Participants’ usage behaviour during the navigation of websites ... 61
Figure 22: User behaviour difference in two websites during learning ... 63
Figure 23: Difference in skin conductance level between the two websites ... 64
Figure 24: Markers of quality of sketch and their visual illustration ... 66
Figure 25: Expert ratings on the quality of sketches pre and post-learning ... 67
Figure 26: Proposed e-learning evaluation model ... 72
ix
LIST OF TABLES
Table 1: Participant details for attitude measurement using semantic differential ... 44
Table 2: Semantic differential scores for online and offline modes of learning ... 47
Table 3: Gender-related differences in attitude towards online vs. offline mode ... 48
Table 4: Age-related differences between online and offline mode preference ... 49
Table 5: Qualification wise differences between a preference for online and offline mode. 50 Table 6: Factor analysis results for offline and online semantic differential data ... 51
Table 7: Correlated SD scales with factors for online medium (in order of correlation) ... 52
Table 8: Correlated SD scales with factors for offline medium (in order of correlation) ... 53
Table 9: ANOVA on users’ behaviour during navigation between the two sites ... 61
Table 10: ANOVA on behavioural markers during learning between the two sites ... 62
Table 11: Details of the three experts ... 65
Table 12: Inter-Rater reliability between three experts for pre-learning sketches ... 67
Table 13: Inter-Rater reliability between three experts for post-learning sketches ... 67
Table 14: ANOVA of expert ratings from the website with usability problems ... 68
Table 15: ANOVA of expert ratings from the website without usability problems ... 68
x
GLOSSARY
Animation Animation is a method of creating moving pictures through the illusion of movement by a successive display of hand-drawn or computer made images. The images could be 2 dimensional (length and breadth on paper/screen) or 3 dimensional (length, breadth and depth on paper/screen) thus giving the name 2D and 3D animations.
Attitude A belief or feeling about something or someone. In the context of learning, it is the predisposition towards the medium of learning, like the predisposition towards e-learning, the person mediated traditional learning, etc.
Behaviour The way in which one reacts. In the context of e-learning, it is the reactions towards the e-learning medium, the e-learning content or any of the steps involves in the process of e- learning.
Cell Animation Animators drawing by hand on the cell or acetate sheets, multiple drawings for each frame of the animated movies and feature-length film is known as Cell animation.
Cognitive Load Cognitive load is the amount of working memory used. In the context of e-learning, the working memory can be used by either the content of the learning or the e-learning platform itself.
E-learning E-learning or electronic learning is electronically mediated learning where ‘E’ stands for ‘Electronic’. E-learning is an overarching term for all different formats of learning where electronics has a role. The usage of storage devices like CD, DVD, VCD, cloud, etc., or audiovisual medium (multimedia) internet, intranet, satellite broadcast, interactive TV, websites, computer, tables, phones, etc. Literally, any electronic media
xi
used in the process of delivery of learning comes under e- learning.
Gestalt Psychology A movement in psychology which studied perception in terms of the ‘whole’ rather than its constituents.
Human-Computer Interaction Human-computer interaction or HCI in short, is the field of study that studies how human interacts with computers and how computers should be designed to match the human capabilities and needs. In the context of e-learning, HCI discipline and its theories would naturally become relevant along with learning theories as the interaction with the computer as well as the learning content together determine the learning efficiency.
Instructional Design Instructional design is the process of planning, designing and producing the learning content. Online courses, instructional manuals, simulations, exercises developed for print or online, web-based program or video tutorials, all come under instructional design field.
Interaction design Interaction design is the process of designing the interaction between the human and computer. The focus of design could be efficiency, visual delight, engagement, learnability etc.
depending upon the type of interface and the type of user.
Learnability (usability) Usability metrics include learnability along with efficiency, effectiveness, ease of use tec. As one of the ways to measure the usability of an interactive system. The ‘learnability’ aspect of usability attempts to measure the design quality of the interactive system because of which it can be learned easily by the user.
Learnability (learning) In the context of learning, ‘learnability’ is about the quality of the content because of which learning can happen easily. It is about how to design the learning content such that it can
xii
facilitate better understanding, retention, and application of the learned content.
Learning Outcome Learning outcome is the end result of any course or training.
The learning outcome can be evaluated based on performance of learning post-learning exercise and a difference between pre-learning performance and post learning performance can be used to assess the quality of the learning content and learning media.
Offline learning In this thesis, offline learning implies learning where no electronic media is used. It is used to imply traditional, instructor-led, classroom-based learning.
Online learning In this thesis, online learning implies learning where electronic media is used. For experiments in this thesis, online learning has been used to mean specifically learning through a live website and not stored content on the computer.
Semantic Differential Scale Semantic Differential Scale is a seven-point rating scale used in this thesis with bipolar adjectives on two ends of the scale. The respondents mark any of the 7 markers on the scale showing distance from the two opposite adjectives on two ends of the scale. This scale has been used to measure the attitude of learners towards the learning media.
Skill Skill is the expertise to do some meaningful act of profession.
In this thesis for the experiments that validate the proposed e- learning evaluation model, skill has been used to mean the competency of delivering a sketch in proportion and scale by the learner of the animation course.
Tacit Tacit means what can be understood without being stated.
Tacit knowledge forms a substantive part of the design profession in general and animation design profession in particular where a piece of knowledge is used during the professional practice and is appreciated but cannot be
xiii
explicitly stated or taught in theory. Often thus requiring the need to learn by practice, reflection, and imbibing.
Usability It is the quality of an interactive product because of which it is fit to be used. Ease of use, efficiency, effectiveness, learnability, and memorability are some of the parameters that are used to measure usability.
Usability Testing Testing with the users of the interactive products like websites, mobile phones etc. to understand the extent to which the product is understood, easy to learn, easy to use, etc. This testing of the ‘usability’ of the product determines the fitness for use by the users of the product.
User experience design The process of designing for the experience of the users of a product based on its interactivity, its emotional experience, its aesthetics, etc. While usability focuses to match the cognitive model of the user, the user experience aims to fulfil its affective and aesthetic needs.
Visceral Reaction Visceral reaction is a gut feeling or an instinctive response to a stimulus or an experience. An initial response towards the look and feel of the e-learning website may influence the actual learning behaviour and the learning outcomes of the learner.
Vocational Courses Vocational courses refer to different forms of courses designed to cater to specific needs of a career. Often the goal of a vocational course is to prepare an individual to directly suit the needs of industry. An example in the context of this thesis is online animation courses which are created to meet the needs of the growing demands of the animation industry.
