Editors Barbara Kump, Kristin Knipfer, Daniel Wessel, KMRC

Link to the work package WP1 Reflection model and user studies

Delivered M6 (December 2010)

Link to the deliverable (pdf) D1.1 Specification of Research Methodology and Research Tooling

Executive summary

The vision of MIRROR is to empower and motivate employees to learn by reflection of tacit work practices and personal experiences. This will be achieved by complementing personal and organisational learning environments (which mainly rely on knowledge being explicitly available) with highly personal MIRROR applications for individual, social, creative, game-based as well as organisational reflection and real-time learning. MIRROR Apps will be seamlessly integrated in the work situation of their intended users.

Reflection in the context of learning refers to “those intellectual and affective activities in which individuals engage to explore their experiences in order to lead to new understandings and appreciations” (Boud et al., 1985, p. 19).

Within MIRROR, we consider reflection as circulating among the individual, inter-individual and organisational level. Conceptually, the starting point for the design of MIRROR Apps is the model of Work Place Reflection (WPR) enriched by the individual perspectives of partners from WP3 to WP9. The model points to the potential of tools to mutually inform work and reflective processes (individual and collaborative) by capturing data from work processes and making them available as a resource for reflection. However, this model needs to be further refined.

User studies will be carried out in all MIRROR testbeds to ensure that the AS-IS situation in the testbeds before introducing MIRROR Apps is well-understood. The main purpose of the user studies is to find out which personal and context factors influence reflection in the testbeds, and how technology can support this reflection. The WPR model will also be iteratively refined based on results from the user studies. In our view, there is a two-way relationship between theory development and user studies: On one side, clearly the design of the user studies and applied methodology will be theory-driven; on the other side, findings from the user studies will be fed back into theory thus enabling us to further refine our theoretical concepts. In parallel to the user studies, requirements elicitation will take place. Requirements elicitation will focus on how MIRROR Apps can solve the needs, e.g. the desired to-be situation of reflective learning in the test beds. Therefore, user studies will build the basis for requirements elicitation activities.

Five different testbeds are part of MIRROR, the Registered Nursing Home Association (RNHA), the Neurological Clinic in Bad Neustadt (NBN), Regola, British Telecom (BT), and Infoman. This will ensure that the MIRROR system is usable in a variety of different scenarios.

The RNHA testbed comprises a sample of Care Homes in the UK. The relationship between carer and resident provides challenges particularly to inexperienced carers. Current knowledge of care for persons with dementia shows that the carer must actively take into account the unique history of the resident to adequately handle situations occurring in day-to-day interaction.

The NBN testbed are the medical staff in the stroke unit of the Neurological Clinic in Bad Neustadt. A big issue for care professions lies in coping with the amount of workload and emotional stress, although this is mostly neglected by hospitals. A key to preventing burn-out syndromes or similar problems lies in turning demanding situations into learning experiences by reflection on what was going on, how staff reacted to it, and if the reaction was beneficial in terms of outcomes.

The Regola testbed for MIRROR are the Regola headquarters and the Civil Protection Organisation in Turin (Torino). The Civil Protection is responsible for coordinating the effort of personnel from several organisations with respect to disaster management in the Turin area and in collaboration with other Civil Protection units. Simple and flexible resources are needed to handle events. Some reflection and training (e.g. field trials) is happening within the associated organisations. A major challenge for the Civil Protection is to achieve learning from their experiences of handling the cases of disaster prevention and management. A goal of introducing MIRROR solutions is to help the Civil Protection improve learning from experience among their volunteers on an individual, team and organisational level.

In the BT testbed, the main target group will be service technicians within BT and the contract teams which they are part of. The participating branch of BT in UK is a testbed for similar scenarios in the BT group. BT currently has a large number of service technicians working on customer sites. The contract team managers need knowledge about issues that have a negative impact on the contract process (e.g., delays in service technicians’ work) and want the team to learn from experience to avoid mistakes. There is a potential to learn from experience across contract teams and on the level of the organisation which may decide to implement changes to work processes.

The Infoman testbed will be the staff of the Infoman headquarters in Stuttgart. The targeted end users of MIRROR are sales people. Currently, systematic knowledge sharing and collaborative experience-based learning between sales consultants at Infoman happens only sporadically or face-to-face in a personal relationship. Explicit sharing of learning material does not take place at the moment.

The overarching research questions for the user studies are: (i) How do reflection and reflective learning currently take place within the testbeds, and how does technology support these processes?, and (ii) What are current needs and ideas for supporting reflective learning in the future?

The following aspects shall be addressed in the user studies:

• Organisational context and work practice
• Current technology usage and available data 
• Reflection (individual, collaborative, and organisational perspective)
• Learning at the workplace
• User experiences with, and attitude towards technology

User studies will be carried out by WP3 to WP9, that is, researchers from all work packages will visit different testbeds and collect data. The process of who collects data at which testbed will be coordinated by WP1 (Task T1.2). Outcomes from all user studies will inform theory work in WP1.

A toolbox was developed, which is a collection of research instruments for data collection (questionnaires, interview guidelines etc.), that covers all research questions of the different work packages for the user studies. The toolbox was developed in close cooperation with the research partners taking into account the specific situation in the testbeds. A Testbed Wiki was set up where information about each testbed was collected in a way that everyone in the project could access that information easily. Moreover, Testbed Visits were organized, in the course of which all research partners had the opportunity to visit the testbed organisation, see the real work situation and talk to potential users of the MIRROR Apps (the target group). Based on the research questions of the different work packages, and in close cooperation with the researchers from WP3 to WP9, research instruments for data collection were then designed by KMRC. Some of these materials, namely the staff interview, a short version of the reflection questionnaire, and the reflection diary have already been applied in pilot studies.

It is worth noting that it is not intended that each tool of the toolbox is applied in each single testbed, and that prior to implementation, the tools need to be customized for the respective testbed.
Altogether, 15 research instruments plus additional materials are included in the toolbox:

Organisational context and work practice

• Job Description Interview: questions about duties and responsibilities, current work practice, learning and training issues, team work, communication and coordination
• Work Observation Scheme: observation form for work practice with a special focus on team meetings, coordination, and communication

Reflection (individual, collaborative and organisational perspective)

• Reflection Interview: individual, collaborative and organisational aspects of reflection
• Reflection Diary: explorative tool for examining reflection AS-IS during daily work
• Reflection Questionnaire: AS-IS situation of individual and collaborative reflection
• Organisational Reflexivity Questionnaire: to be answered by management staff; examines the AS-IS situation of organisational learning, change and reflexivity

Learning at the workplace 

• Needs and Requirements Analysis for Organisational Learning and Intelligence: interview guideline examining the end users’ needs and requirements with respect to organisational learning and respective technology
• Organisational Learning Management Interview: interview questions for management staff to examine organisational learning and intelligence practice
• Learning at Work Questionnaire questions on the status of and relationship between individual learning, team learning and organisational learning

Current technology usage and available data

• Available Data Checklist: checklist for data available within the testbed that could be used for reflection purposes
• IT Checklist: structured interview guideline for an interview with a system administrator about the IT infrastructure of the target organisation

User experiences with and attitude towards technology

• IT Attitudes & Usage Questionnaire: questions about general attitudes towards and usage of different kinds of technology (including sensors and serious games)
• Privacy Questionnaire: questions on sharing of information, trust within teams, trust in the organisation, and the organisation's handling and use of personal data
• Serious Games Experience with In-Depth Interview: interview guideline to acquire requirements with regard to serious games
• Capturing Pilot Study Guidelines: description of how to carry out a pilot study to test the applicability and usability of sensor data; diary with follow-up interview cover the acceptance of sensor data of the users

 The following principles will be applied for the user studies:

(a) minimal risk for participants,
(b) comprehensive (de-)briefing of participants,
(c) evaluation of participant comprehension of information received,
(d) voluntary nature of participation,
(e) obligation to observe confidentiality,
(f) waivers and permissions for audio and video recordings and their transcripts, and
(g) documentation that adheres to such policies.

A detailed discussion of ethical guidelines for data collection and data sharing is also part of the Deliverable. DI.1: Specification of Research Methodology and Research Tooling 9. On-Site user studies will take place between February and April 2011. Each of the work packages will report on results of their user studies in June 2011 (M12). Findings will then be integrated and reported in September 2011 (M15) in Deliverable DI.2 by WP1.