Development of user's cognitive load centric methodology for HCI based control panel design

(1)

DEVELOPMENT OF USER’S COGNITIVE LOAD CENTRIC METHODOLOGY FOR HCI BASED

CONTROL PANEL DESIGN

NAVEEN KUMAR

CENTRE FOR INSTRUMENT DESIGN AND DEVELOPMENT

INDIAN INSTITUTE OF TECHNOLOGY DELHI

OCTOBER 2018

(2)

©Indian Institute of Technology Delhi (IITD), New Delhi, 2018

(3)

DEVELOPMENT OF USER’S COGNITIVE LOAD CENTRIC METHODOLOGY FOR HCI BASED

CONTROL PANEL DESIGN

by

Naveen Kumar

Centre for Instrument Design and Development

Submitted

in fulfilment of the requirements for the degree of Doctor of Philosophy

to the

Indian Institute of Technology Delhi

October 2018

(4)

Dedicated to my loving parents and family

(5)

i

CERTIFICATE

This is certified that the work contained in this thesis titled “Development of user’s cognitive load centric methodology for HCI based control panel design” is submitted by Mr Naveen Kumar to the Indian Institute of Technology Delhi for the award of the degree of Doctor of Philosophy has been carried out under my supervision.

Mr. Naveen Kumar has fulfilled the requirements for the submission of this thesis, which to our knowledge has reached the requisite standard. This research work has not been submitted any other University/

Institute for the award of any degree.

Dr. Jyoti Kumar Professor Centre for Instrument Design and Development, Indian Institute of Technology Delhi, New Delhi-110016, Delhi INDIA

(6)

ii

ACKNOWLEDGMENTS

I am truly indebted to God ‘Almighty’ for blessing me through my supervisor Prof. Jyoti Kumar.

He has sincerely guided me throughout my PhD programme. Without his consistent support and encouragement my research work could not have been possible. It has been an honour to work with him as a full-time PhD student. I will cherish experience of working under his guidance throughout my life. His both personal and research advices help me a lot in taking important decisions in my PhD duration. I am very grateful to him for my entire life.

My sincere thanks to Prof. P V M Rao, Head, IDDC, IIT Delhi for his academic support. I thank to Prof. Chandra Shakher and Prof. A. L. Vyas for their advices and encouragement. I thank to Prof. Srinivasan Venkataraman (SRC member) for his valuable suggestions on my research work.

Also, I thank to Prof. P M Pandey (SRC external expert) for his valuable inputs in my research work. My sincere thanks to Prof. Sumer Singh and Mr S K Atreya for their advices. I am thankful to all the faculty and staff members of IDD Centre, IIT Delhi, for help and support. My special thanks to Prof. Gufran Syed Khan and Prof. S K Dubey for timely reminding course related activities.

I am also very thankful to all my co-researchers Ms. Aarati Prakash Khare, Mr. Jyotish Jaiswal, Mr.

Sunny, Mr Abhishek, Mr Jitesh, Ms Surbhi and Ms Pooja Sahni. They all are very special to me.

They helped me in many ways. I thank Mr Jyotish and Mr Sunny to provide their consent to publish their photos in my research articles. I thank Mr Abhishek who has helped me on conducting design workshops. Special thanks to the interns of UX-LAB, Mr Kuber and Mr Abhishek Rajpurohit who has helped me to collect participant’s physiological data.

I am wholeheartedly thankful to my mother Mrs Dhaneshwari Devi, father (Late) Shri Vidya Datt Bhardwaj, Brother Mr Praveen Bhardwaj, and my sisters Preeti and Kalpana for blessing me at every moment of my life. My heartfelt thanks to my wife Mrs. Preetam Dhoundiyal for her unconditional support, immense patience and understanding. Her love gave me strength to persevere and continue further.

Last but not the least sincere thanks to all the employees of IIT Delhi for maintaining curriculum, infrastructure, student activities, etc. Words are not enough to thank the countless people who have been a part of my journey.

Place: New Delhi (Naveen Kumar)

Date:

(7)

iii

ABSTRACT

The world we live in has become permeate with computing technologies. Today, computing machines are not only on our desktops but almost in all walks of life. Human Computer Interaction (HCI) in workplaces have started affecting productivity and efficiency. HCI systems are presenting complex information to human information processing system and is often causing excessive cognitive load in workplaces. The next industrial revolution called Industry 4.0 is likely to further add to the information complexity as technologies like Cyber-Physical Production Systems (CPPS), Network Cloud Computing (NCC), Internet of Things (IoT) and Wireless Embedded Network Systems (WENS) will enable the use of small handheld, mobile and Internet-based interactive devices to control manufacturing plants remotely. This thesis argues that if the design of HCI based control panels do not take care of information richness (intrinsic cognitive load), presentation of information on user interfaces (extraneous cognitive load) and learning efforts of users (germane cognitive load) through appropriate HCI design methodology then efficiency of operators and thereby of industry may suffer.

This thesis has proposed a theoretical framework for human efficiency measure as a factor of industry efficiency from cognitive load perspective. Then a design methodology with focus on cognitive load assessment for HCI based control panel designs has been proposed. Using the methodology, a few control panel designs were developed and evaluated. Experimental data using EEG (electroencephalography), GSR (Galvanic Skin Response), task performance measures (Response time and errors) and subjective self-reports were gathered for assessment of cognitive loads as a part of the methodology.

Results from experiments on cognitive load evaluation on different designs showed a. good congruence in cognitive load data gathered from the three measures namely, physiological, task performance and subjective measures, b. different design options for same tasks caused different cognitive loads, for example, display designs with both analog and digital displays caused less cognitive load compared to only analog and only digital designs c. auditory communication in HCI designs caused higher cognitive load compared to visual communication and d. software prototypes caused higher cognitive load compared to paper prototypes during testing. Based on the findings, this thesis has argued the need to use cognitive load focussed design methodology for design and evaluation of control panels in information rich environments like that of Industry 4.0.

(8)

सार

जिस दुजिया में हम रहते हैं वह कंप्यूज ंग प्रौद्योजगजकयों के साथ पारगम्य हो गया है। आि, कंप्यूज ंग मशीि

ि केवल हमारे डेस्क ॉप पर बल्कि िीवि के सभी क्षेत्ों में हैं। काययस्थलों में मािव कंप्यू र इं रैक्शि

(एचसीआई) िे उत्पादकता और दक्षता को प्रभाजवत करिा शुरू कर जदया है। एचसीआई जसस्टम मािव सूचिा प्रसंस्करण प्रणाली को िज ल िािकारी पेश कर रहे हैं और अक्सर काययस्थलों में अत्यजिक संज्ञािात्मक भार पैदा कर रहे हैं। इंडस्टरी 4.0 िामक अगली औद्योजगक क्ांजत में सूचिा िज लता में आगे

बढ़िे की संभाविा है क्ोंजक साइबर-जिजिकल प्रोडक्शि जसस्टम (सीपीपीएस), िे वकय क्लाउड कंप्यूज ंग (एिसीसी), इं रिे ऑि जथंग्स (आईओ ी) और वायरलेस एंबेडेड िे वकय जसस्टम (डब्ल्यूएिएस) िैसी तकिीकें जवजिमायण संयंत्ों को दूरस्थ रूप से जियंजत्त करिे के जलए छो े हैंडहेल्ड, मोबाइल और इं रिे -आिाररत इं रैल्किव जडवाइसों के उपयोग को सक्षम करें। यह थीजसस का तकय है

जक यजद एचसीआई आिाररत जियंत्ण पैिलों का जडजाइि सूचिा समृल्कि (आंतररक संज्ञािात्मक भार), उपयोगकताय इं रिेस (बाहरी संज्ञािात्मक भार) और उपयोगकतायओं के सीखिे के प्रयासों (िमयिी

संज्ञािात्मक भार) पर उजचत एचसीआई जडजाइि के माध्यम से िािकारी की प्रस्तुजत िहीं लेता है

काययप्रणाली तो ऑपरे र की दक्षता और इस प्रकार उद्योग का सामिा करिा पड़ सकता है।

इस थीजसस िे संज्ञािात्मक लोड पररप्रेक्ष्य से उद्योग दक्षता के कारक के रूप में मािव दक्षता उपायों के

जलए सैिांजतक रूपरेखा का प्रस्ताव जदया है। जिर एचसीआई आिाररत जियंत्ण पैिल जडिाइि के जलए संज्ञािात्मक भार मूल्ांकि पर ध्याि देिे के साथ एक जडिाइि पिजत का प्रस्ताव जदया गया है। पिजत का

उपयोग करके, कुछ जियंत्ण पैिल जडिाइि जवकजसत और मूल्ांकि जकए गए थे। ईईिी

(इलेिरोएन्सेफ्लोग्रािी), िीएसआर (गैल्वेजिक ल्कस्कि ररस्ांस), कायय जिष्पादि उपायों (प्रजतजक्या समय और त्ुज यों) और व्यल्किपरक आत्म-ररपो य का उपयोग कर प्रायोजगक डे ा पिजत के एक जहस्से के रूप में संज्ञािात्मक भार के आकलि के जलए एकत् जकए गए थे।

जवजभन्न जडजाइिों पर संज्ञािात्मक लोड मूल्ांकि पर प्रयोगों के पररणामों से पता चला है। तीि उपायों से

ए) एकजत्त संज्ञािात्मक भार डे ा में अच्छी संगतता, अथायत् शारीररक, कायय प्रदशयि और व्यल्किपरक उपायों, बी) जवजभन्न कायों के जलए अलग-अलग जडजाइि जवकल्ों के कारण जवजभन्न संज्ञािात्मक भार होते

हैं, उदाहरण के जलए, एिालॉग और जडजि ल जडस्ले दोिों के साथ जडस्ले जडजाइि केवल एिालॉग और केवल जडजि ल जडजाइि की तुलिा में कम संज्ञािात्मक लोड का कारण बिता है। सी) एचसीआई जडिाइिों में श्रवण संचार िे दृश्य संचार और डी की तुलिा में उच्च संज्ञािात्मक भार उत्पन्न जकया।

सॉफ्टवेयर प्रो ो ाइप परीक्षण के दौराि पेपर प्रो ो ाइप की तुलिा में उच्च संज्ञािात्मक भार का कारण बिता है। जिष्कर्षों के आिार पर, इस थीजसस िे इंडस्टरी 4.0 की तरह समृि वातावरण में जियंत्ण पैिलों

के जडिाइि और मूल्ांकि के जलए संज्ञािात्मक भार केंजित जडिाइि पिजत का उपयोग करिे की

आवश्यकता पर तकय जदया है।

(9)

iv

S. No. Table of Content Page No.

A. Certificate i

B. Acknowledgements ii

C. Abstract iii

D. Table of Content iv

E. List of Figures vii

F. List of Tables x

G List of Abbreviation and Terminology xii

Chapter1: Control panel design in Industry 4.0 context ... 1

1.1 Introduction ... 1

1.2 Changing context of control panel design in Industry ... 2

1.3 HCI and CPPS in Industry 4.0 ... 6

1.4 Cognitive load perspective in control panel design... 7

1.5 Research objectives ... 9

1.6 Research summary ... 9

1.7 Chapter summaries ... 10

Chapter 2: Literature survey on cogntive load and HCI based control panel design ... 12

2.1. Introduction ... 12

2.2. Changing role of control panels in Industry ... 12

2.3. Evolution of user interfaces in Industry ... 13

2.4. Design of human computer interaction ... 15

2.5. Human information processing models for HCI design ... 16

2.6. Human information processing and cognitive load ... 18

2.7. Cognitive load measurement methods for HCI design ... 19

2.8. Human brain and EEG measures ... 21

2.9. Gap analysis in literature ... 25

2.10. Chapter summary ... 25

Chapter 3: Development of a framework for measurement of cognitive load in Industry 4.0 ... 27

3.2 Prevalent production efficiency measures ... 29

3.3 A proposal for smart factory efficiency measure ... 31

3.3.1 Human cognitive efficiency (HCE) ... 32

3.3.2 Communication system efficiency (CSE) ... 37

(10)

v

3.3.3 Smart machine efficiency (SME) ... 41

3.3.4 Smart factory efficiency measure ... 42

3.4 Cognitive load heuristics for HCI design ... 43

3.4.1 Design heuristics for cognitive load consideration ... 44

3.5 Chapter summary ... 47

Chapter 4: Proposal of a cognitive load centred methodology for control panel design ... 48

4.2 Prevalent control panel design methods ... 48

4.3 Proposal of a new methodology ... 50

4.3.1. User research ... 52

4.3.2. Task analysis ... 53

4.3.3. Design ... 61

4.3.4. Validation of design ... 62

Chapter 5: Application of UCLCD4 for cognitive load evaluation of designs ... 67

5.2 Development of designs using UCLCD4 ... 67

5.3 Validation of control panel designs using physiological measurements ... 77

5.3.1 Which causes less cognitive load: analog, digital or hybrid? ... 77

5.3.1.1 Introduction ... 77

5.3.1.2 Material and methods ... 78

5.3.1.3 Results and discussions ... 82

5.3.1.4 Conclusions... 90

5.3.2 Which communication method causes more cognitive load: auditory, visual or hybrid ... 91

5.3.2.3 Results and discussions ... 94

5.3.2.4 Conclusion ... 97

5.4.3 Which prototyping methods is better for measuring cognitive load caused by designs: software prototype or paper prototype? ... 98

5.3.3.3 Results and discussion ... 105

5.3.3.4 Conclusion ... 112

(11)

vi

Chapter 6: Conclusion and discussion ... 114

6.1 Conclusion ... 114

6.2 Discussion ... 115

6.3 Thesis contribution ... 116

6.4 Limitations of the study ... 118

6.5 Scope of future work ... 119

Bibliography 120

Appendix A 132

Appendix B 144

Brief Profile of the Author 145

(12)

vii

LIST OF FIGURES

FIGURES DISCRIPTION PAGE NO.

Fig. 1.1 Industrial revolution and operators changing context from industry 1.0 to industry 4.0

3

Fig. 1.2 Operator role in cyber physical production systems (CPPS) 7

Fig. 2.1 User interface evolution in the factory automation 14

Fig. 2.2 Human information processing model by (Card, Moran, and Newell 1986)

17

Fig. 2.3 Workingmemory model (Atkinson & Shiffrin, 1968) 18

Fig. 2.4 Human brain: cerebrum, cerebellum and brain stem (Gazzaniga, 2014) 22

Fig. 3.1 Man and machine communication scenario in CPPS shop floor 38

Fig. 3.2 Machine to human communication via network nodes 39

Fig. 3.3 Pharmaceutical manufacturing HCI module with different production processes and parameters

39

Fig. 4.1 User Centred Design (UCD) methodology by (Norman, 1986) 51

Fig. 4.2 UCLCD4 methodology for control panel design 52

Fig. 4.3 User research process for control panel design 53

Fig. 4.4 Task cognitive demand level v/s Task frequency 57

Fig. 4.5 Task and scenario mapping for first quadrant 59

Fig. 4.6 Task flow diagram 60

Fig. 4.7 Process for designing control panels 61

Fig. 4.8 EEG data acquisition and analysis technique 63

Fig. 4.9 Combine method-experiment design for cognitive load measurement using NASA TLX & EEG

64

(13)

viii

Fig. 4.10 Evaluation of control panel design using cognitive load measurement through EEG & NASA TLX

65

Fig. 5.1 User research data (a) Persona (b) Scenario 68

Fig. 5.2 Design process 70

Fig. 5.3 Concept sketching 70

Fig. 5.4 Three interactive prototypes with three different display types 76 Fig. 5.5 Stimuli design (a) analog, (b) digital (c) hybrid display design 79-80

Fig. 5.6 Experimental methodology 81

Fig. 5.7 Experimental setup 82

Fig. 5.8 Beta band power difference across frontal lobe electrodes for each stimuli designs: (a) low-level task difficulty (b) high-level task difficulty

85

Fig. 5.9 Response time (milliseconds) for ANALOG, DIGITAL & HYBRID design: comparison between high and low difficulty task level

87

Fig. 5.10 Errors recorded (in counts) mean valve for each stimuli (a) high task difficulty level (b) low task difficulty level

88

Fig. 5.11 Post task rating on 10 point scale in NASA TLX parameters for each stimuli

90

Fig. 5.12 Electrode positions on scalp based on 10/20 international system 92

Fig. 5.13 Experimental setup 94

Fig. 5.14 Mean and Standard Deviation differences in task difficulty ratings by 10 participants among auditory, visual and hybrid stimuli (a) Low complex task (b) High complex task

95

Fig. 5.15 Spectral power at alpha band (8-13 Hz) across parietal & occipital regions for visual, auditory and hybrid stimuli (a) For low level complex task (b) For high level complex task

97

(14)

ix

Fig.5.16 Prototype design: (a) Starting screen (b) Intermediate screen (c) Main screen

100-101

Fig. 5.17 Experimental set up (a) software prototype (b) Paper prototype 102 Fig. 5.18 Experimental procedure for both software and paper prototype data

collection

103

Fig. 5.19 EEG data analysis method 104

Fig. 5.20 Mean response time (msec.) between software and paper prototype 105 Fig. 5.21 Mean difference in total no. of error per participant between software

and paper prototype

106

Fig. 5.22 Mean differences of NASA TLX parameters ratings for both software and paper prototype

106

Fig. 5.23 GSR Mean value differences between paper and software prototype 108 Fig. 5.24 Topography of correlation between frequency band powers with NASA

TLX parameters

110

(15)

x

LIST OF TABLES

TABLES DISCRIPTION PAGE NO.

Table. 2.1 Limits on processors (Card, Moran, and Newell 1986). 18

Table 2.2 Literature survey on information complexity in Industry 4.0 context 26

Table 3.1 CL heuristics for Intrinsic Cognitive Load 44

Table 3.2 CL heuristics for Extraneous Cognitive Load 45

Table 3.3 CL heuristics for Germane Cognitive Load 45

Table 3.4 CL evaluation heuristics 46

Table 4.1 Existing control panel design evaluation methods 49

Table 4.2 Example of list of user’s and tasks in CPPS. 55

Table 4.3 Cognitive demand between users: Mean and SD 56

Table 4.4 Operators tasks cognitive demand with task frequency 56-57

Table 5.1 Design concepts of control panels and their CL heuristic evaluations 71-73 Table 5.2 Heuristic evaluation of three design concepts having similar scores 75 Table 5.3 F values at P<0.01 for Beta band power using two-tailed T-test for

difference in high and low task difficulty level for stimuli designs (*P<0.01 significant values)

83

Table 5.4 Beta band power difference in high difficulty level task between analog, digital and hybrid design using ANOVA (*P<0.05 significant values)

84

Table 5.5 Beta band power difference in low difficulty level task between analog, digital and hybrid design using ANOVA (*P<0.05 significant values)

84

Table 5.6 Task design for measuring mental activity for auditory, visual and hybrid stimuli

93

(16)

xi

Table 5.7 Correlations between the NASA TLX parameters for both the prototypesCorrelations between the NASA TLX parameters for both the prototypes (*significance p>0.05).

107

Table 5.8 EEG electrodes correlations (R) with spectral powers and NASA TLX parameter

109

Table 5.9 Frequency band power across EEG channels and its mean difference between software and paper prototype (ANOVA table)

109

Table 5.10 EEG Entropy differences (p<0.05, p<0.01) in different EEG channels for both software and paper prototype

112

(17)

xii

LIST OF ABBRIVATIONS AND TERMINOLOGIES

CPPS: Cyber Physical Production System.

A new technology of production system where smart sensors and smart machines are integrated with wireless embedded network systems and internet of things.

CL: Cognitive Load. Load on human information processing system, specifically on

working memory, due to task related mental work that a human user is required to accomplish.

Control Panel: A systematic arrangement of two or more input and output components (e.g. push buttons, knobs, dials etc) at one place.

Design Heuristics: Rules of thumb to aid designers in creative ideation and quick assessment.

Design Validation: Process of verification of a creative design idea with or without the help of users

EEG:

Electroencephalography.

Measure of electrical activity of brain cells.

HCI: Human Computer Interaction.

Field of study that studies how humans interact with computers and how to design interactive interfaces for humans users.

Information Complexity: Increase in amount of relevant and irrelevant data in user interfaces that a human user needs to process in order to complete the requite industrial tasks.

Mental Model: An internal representation of external artefacts and procedures that human mind stores and uses as reference to during meaning making process.

Persona: A rich description of representative control panel user based on user research and depicting user needs, goals and behaviour patterns.

Scenario: A temporal description of the context of task on control panels illustrating users’ motivations, environment, circumstances and devices.

Task Analysis:

A systematic study of tasks that does characterization, prioritization and sequencing of task in order to identify dependencies and redundancies.