Technologies for Learning and Rehabilitation

Summary:

The challenge of 3D Tune-In is to facilitate the successful exploitation of existing, overlooked or neglected functionalities of hearing devices to optimise their potential and thus greatly improved people’s quality of life and their interactions with other people and their surrounding environment. Bringing together the relevant stakeholders from traditional gaming industries (Reactify, Vianet, XTeam, Nerlaska), academic institutes (imperial College London, De Montfort University, University of Nottingham, University of Malaga); a large European hearing aid manufacturer (GN Resound); and hearing communities through Associations (Extra Care, Hearing Link, Action Deafness, Accesibilidad y Personas Sordas, Ente Nazionale Sordi), 3D Tune-In will create a toolkit and different games tailored to different target audiences and contexts to help in the understanding of hearing loss and hearing aids.

Key contribution factors:

• Stakeholder engagement and elicitation of user requirements
• Evaluation and validation of game concepts
• User evaluation and validation of the technologies and apps

Publications:

• Eastgate, R., Picinali, L., Patel, H., & D´Cruz, M. (2016). 3D Games for Tuning and Learning About Hearing Aids. The Hearing Journal, 69 (4), 30-32 (link?)
• Picinali, L., D’Cruz, M. & Simone, L. (2015) 3D-Tune-In: the use of 3D visuals and sound to facilitate use of hearing devices. Conference paper at EuroVR 2015, Lecco, Italy, 15 – 16 October
• Picinali, L., D’Cruz, M. & Simone, L. (2015) 3D-Tune-In: the use of 3D visuals and sound to facilitate use of hearing devices. Poster and Demonstration paper at EuroVR 2016, Athens, Greece, 22 – 24 Nov [see ppt attached]
• Levtov, Y., Picinali, L., D'Cruz, M. & Simeone, L. (2016) 3D Tune-In: The Use of 3D Sound and Gamification to Aid Better Adoption of Hearing Aid Technologies. Conference paper at 140th Audio Engineering Society Convention. Paris, France, 4-7 June.
• Patel, H., Cobb, S., Hallewell, M., D'Cruz, M., Eastgate, R., Picinali, L., & Tamascelli, S. (2016, October). User involvement in design and application of virtual reality gamification to facilitate the use of hearing aids. In Interactive Technologies and Games (iTAG), 2016 International Conference on (pp. 77-81). IEEE. Link http://ieeexplore.ieee.org/abstract/document/7782518/ 

3D Tune-In online leaflet

3D Tune-In

Project duration: 2016-2019                       

Sponsor: EU H2020 ICT-20-2015 Technologies for better human learning and teaching

Budget: €7.6 Million

Summary:

Mathisis aims to assist the educational process for learners and their tutors and caregivers by creating a novel and continuously adaptable "robot/machine/computer"-human interaction ecosystem to enhance vocational training, workplace learning and mainstream education for learners. The novelty of the MaTHiSiS system is that it will learn from student behaviour and feedback to adapt selection of new learning material suited to the individual learner.  Making use of shared knowledge within the system to support new methodologies in adaptive learning technology. Use case examples will be evaluated for a variety of target learner groups including; mainstream education, education for people within the autistic spectrum, profound and multiple learning disabilities cases and workplace-oriented activities.

HFRG members:

• Sue Cobb
• Marisé Galvez Trigo
• Penny Standen (School of Medicine)

• ATOS, SPAIN  - Project Coordinator
• DIGINEXT SARL, France
• Centre for Research and Technology Hellas (CERTH), Greece
• National Center for Scientific Research “Demokritos” (NCSR “Demokritos”), Greece
• Nottingham Trent University, UK
• The University of East London (UEL), UK
• Maastricht University (UM), Netherlands
• Vrije Universiteit Brussel (VUB), Belgium
• Fondazione Mondo Digitale, Italy
• IMOTEC, Lithuania
• La Cometa del Sud, Italy
• Nurogames GmbH, Germany
• General Directorate of Educational Innovation and Teacher Training, Spain
• EOPPEP is the National Organization for the Certification of Qualifications and Vocational Guidance, Greece
• Europole, Italy
• Aerospace Valley (AV), France
• OTE Academy, Greece

MaTHiSiS website

Project duration: 2009-2012

Budget: €1.6M

Sponsor: EU FP7 ICT – Digital Libraries and Technology-enhanced Learning

Summary:

Children with autism often experience difficulties with social interactions. The COSPATIAL project explored the use of innovative technologies for supporting social competence for children on the autism spectrum. 

The project focused on the use of two main types of technologies: Collaborative Virtual Environments (CVE) and Shared Active Surfaces (SAS). Critically, these technologies specifically foster collaborative interactions, meaning that children play and learn with their peers, unlike typical technological interventions which are often undertaken individually. The COSPATIAL applications were developed to provide safe, comfortable and structured support guided by principles of Cognitive-Behavioural Therapy. The COSPATIAL research team is multidisciplinary, comprising psychologists, occupational therapists, computer scientists, human factors experts and educators. The applications were designed with close involvement from teachers, children and young people, parents and other professionals, to ensure that they take the complex needs of children on the autism spectrum into account, along with the practical needs from the educational context.

Publications:

• Parsons, S. and Cobb, S. (2014). Reflections on the role of the ‘users’: challenges in a multidisciplinary context of learner-centred design for children on the autism spectrum. International Journal of Research & Method in Education, DOI:10.1080/1743727X.2014.890584
• Millen, L., Cobb, S., Patel, H. and Glover, T. (2014). A collaborative virtual environment for conducting design sessions with students with autism spectrum disorder. Int J Child Health Hum Dev 2014;7(4):367-376
• Parsons, S. & Cobb, S. (2013) Who chooses what I need? Child voice and user-involvement in the development of learning technologies for children with autism. EPSRC Observatory for Responsible Innovation in ICT: http://torrii.responsible-innovation.org.uk/resource-detail/1445 [last accessed 8th September 2013].
• Parsons, S. and Cobb, S.  (2011). State-of-the art of Virtual Reality technologies for children on the autism spectrum.  European Journal of Special Needs Education (EJSNE). 26, (3), 355-366.
• PhD Thesis: Collaborative technologies for children with autism. Laura Millen (2014)

Applying Participatory Design to Develop Technology for Autism Evaluation and Treatment. Online course in collaboration with Autism Speaks

Summary:

Virtual Environments can be powerful tools in supporting education. They allow unique fidelity of representation and learner interaction, providing users with experiences that are not possible in the real world. The ability to display and also to interact with 3D models can support learning about abstract concepts that are traditionally difficult to teach, while the use of immersive technologies can be highly engaging and can motivate learners.

However, Educational Virtual Environments (EVEs) can be difficult to design and there is insufficient guidance available in the literature. The creation of 3D prototypes is often difficult, costly and time-consuming, and therefore 3D features are often not considered until late in the development process when many critical design decisions have already been made. Additionally, it is critical that educators and other relevant stakeholders and involved in the design process to ensure that the applications produced effectively meet learning objectives, yet it can be difficult for non-technical design partners to understand the affordances and capabilities of EVE technologies. The combination of complex development issues means that it is difficult to design educational EVE applications that are useful, usable and effective.

Across a broad range of educational contexts, HFRG have investigated tools and methods to facilitate the involvement of education professionals in EVE development projects, to improve the efficiency of development, the educational effectiveness of the developed systems, and increased acceptance of solutions into real classroom practice.

Publications:

• Cobb, S.V.G., Hawkins, T., Millen, L., and Wilson, J.R. (2014) Development and Evaluation of Virtual Environments for Education. In K. Hale and K. Stanney (Eds.), Virtual Environment Handbook, 2nd Edition, Boca Raton: CRC Press, 1075-1108.
• Parsons, S. & Cobb, S. (2013) Who chooses what I need? Child voice and user-involvement in the development of learning technologies for children with autism. EPSRC Observatory for Responsible Innovation in ICT: http://torrii.responsible-innovation.org.uk/resource-detail/1445 [last accessed 8th September 2013].
• Parsons, S., Millen, L., Garib-Penna, S. and Cobb, S. (2011). Participatory design in the development of innovative technologies for children and young people on the autism spectrum: the COSPATIAL project. Journal of Assistive Technologies (JAT), Vol. 5, Issue 1, March, 29-34.
• Millen, L., Cobb, S.V.G. & Patel, H (2011). Participatory design approach with children with autism. International Journal on Disability and Human Development (IJDHD), 10(4), 289-294.
• PhD Thesis: The role of computer-mediated supported for design teams. Tessa Roper (2015)

Project duration: 2009-2012       

Budget: €3.4M 

Sponsor: FP7 – Accessible and Assistive ICT

Summary:

The project aimed to improve the accessibility of ICT products for older users through the implementation of adaptive personalised interfaces, designed around individual user profiles.  An inclusive design approach was taken involving older users in a series of activities to identify user behaviour when they found screen-based information difficult to interact with and user preferences for system adaptation to alleviate this (e.g. when the system detects that a user is leaning towards the TV screen, it infers a problem with visibility and increases text size). A prototype demonstration of an interactive TV system was produced by the project, demonstrating: Adaptive user interfaces for interactive TV and mobile devices; Automatic adaptations to special user needs and perceptual, motor and cognitive impairments; Tools for the development of adaptive user interfaces.  The project deliverable reports provide guidance for designers and developers of accessible applications and service. 

Key contribution factors:

• Stakeholder engagement and elicitation of user requirements
• Evaluation and validation of game concepts
• User evaluation and validation of the technologies and apps

Publications:

• Edlin-White, R; Cobb, S; Floyde, A; Lewthwaite, S; Wang, J. & Riedel, J. (2012) From guinea pigs to design partners – involving older people in technology design, Chapter 16. In: Langdon, P; Clarkson, J; Robinson, P; Lazar, J. & Heylighen, A. (Eds) Designing Inclusive Systems - Designing Inclusion for Real-world Applications, Springer Verlag, London, pp 155-164. ISBN 978-1-4471-2866-3. DOI: 10.1007/978-1-4471-2867-0_16 .Link: http://www.springerlink.com/content/m1250617l51rpj82/
• PhD Thesis: Design and evaluation of adaptive interfaces for elderly users. Rob Edlin-White (2014)

Project duration: 2003-2004                       

Budget: £89,000

Sponsor: The Stroke Association

Summary:

This project explored the use of virtual and mixed reality technologies to provide guidance and training for rehabilitation of stroke survivors. A common stroke rehabilitation task requires patients to demonstrate independence in making a hot drink. However, doing this task for real involves handling heavy objects and hot liquid. In this project users first practised the task in a virtual environment simulating the task sequence within a virtual kitchen scenario.  Users could then practice the task sequence using physical objects tethered to the computer interface: interacting with each object correctly activated the virtual simulation until task completion. This enables users to practice the task in safety. An extension of the project used a mixed reality system in which the VR simulation provided the user with task sequence instructions and camera vision technology observed user performance with untethered physical objects. The results showed that stroke survivors could use the MR system and found evidence of learning in some individuals.  However, this project also highlighted the importance of task relevance to individuals.

Publications:

• Hilton, D., Cobb, S., Pridmore, T., Gladman, J. and Edmans, J. (2011). Development and evaluation of a mixed reality system for stroke rehabilitation. In: Advanced Computational Intelligence Paradigms in Healthcare 6 (eds. S. Brahnam & L.C. Jain), Springer-Verlag Berlin Heidelberg, chapter 10, 193-228.
• Pridmore, T., Cobb, S., Hilton, D., Green, J. and Eastgate, R. (2007). Virtual Technologies as learning tools in stroke rehabilitation: Interfaces across the real/virtual divide. International Journal on Disability and Human Development, 6 (1), 87-95.
• Edmans, JA., Gladman, J., Cobb, S., Pridmore, T., Hilton, D., Sunderland, A. and Walker, M., (2006). Validity of a virtual environment for stroke rehabilitation.  Stroke, 37, 2770-2775.
• PhD Thesis: Virtual Reality for assessment and rehabilitation of stroke patients. Dave Hilton (2007)

Project: Experiential Virtual Environments (1992)

Project: Makaton Language Teaching (1993)

Project: Health and Safety at work (1997)

Project: Life skills training for students with learning disabilities (1997-98)

In 1992, the Virtual Reality Applications Research Team (VIRART), a sub-group of HFRG, was founded to explore use of virtual reality technology for a variety of application domains.  At the time there literature suggested that VR technology could provide powerful learning environments not available through other means (Bricken, 1991).  In collaboration with local special schools in Nottinghamshire, most notably Shepherd School (now Oakfield School), VIRART developed a number of desktop virtual environments to explore use and utility in special education. These applications intended to support training for individuals with learning needs and communication difficulties in preparation for more independence in their everyday activities and communications in the community.  A review of these projects is given in Cobb (2007).  During this time, a growing VR research community was established through the conference series International Conference on Disability, Virtual Reality and associated Technologies (ICDVRAT), established by Professor Paul Sharkey at the University of Reading in 1996, to examine the benefits of using this technology in mainstream and special needs education, as well as rehabilitation and other learning environments.  At the 10-year anniversary, Cobb and Sharkey published a review of all research conducted to date (Cobb and Sharkey, 2007).  The 20-year anniversary conference will be held in Nottingham 5th September 2018. A key feature of the first conference was the involvement of users themselves in giving the Keynote address to describe their role in the user-centred design process. To mark the importance of user involvement in design and development of virtual rehabilitation technologies and applications, members of the original user group who presented in 1996 will again present their story at ICDVRAT2018. 

HFRG members:

• Dave Brown
• Sue Cobb
• Helen Neale
• Steve Kerr
• Rick Barnes
• Richard Eastgate
• John R. Wilson

Publications:

• Cobb, SVG (2007). Virtual environments supporting learning and communication in special needs education. Topics in Language Disorders, 27 (3), 211-225. 
• Cobb, SVG and Sharkey, PM. (2007). A decade of research and development in disability, virtual reality and associated technologies: review of ICDVRAT 1996-2006. International Journal of Virtual Reality, 6 (2), 51-68.
• Cobb, SVG, Brooks, T, Sharkey, PM and Merrick, J. (2006).  (Eds) International Journal on Disability and Human Development, Special Issue on virtual reality technologies for disability and rehabilitation. 5 (2), April-June 2006. ISSN 1565-012X  EDITORIAL: Virtual Reality and Disability, pp 95-96.
• Cobb, S.and Stanton Fraser, D (2005). Multimedia Learning in Virtual Reality.  In: R. Mayer (Ed.)  The Cambridge Handbook of Multimedia Learning.  New York: Cambridge University Press. pp525-548.
• Cobb, S.V.G., Neale, H.R, Crosier, J.K. and Wilson, J.R. (2002). Development and Evaluation of Virtual Environments for Education. In K. Stanney (Ed.), Virtual Environment Handbook, Chapter 47, Lawrence Erlbaum. 911-936.
• Brown, D. J., Neale, H. R., Cobb, S. V. G., & Reynolds, H. (1999). Development and evaluation of the virtual city. International Journal of Virtual Reality, 3(4), 27-38.
• Brown, D.J., Cobb, S.V.G. and Eastgate, R.M. (1995). Learning in Virtual Environments (LIVE). In: Virtual Reality Applications (eds. RA Earnshaw, JA Vince and H. Jones).  London: Academic Press, 245-252.
• PhD Thesis: Structured evaluation of virtual environments in special needs education. Helen Neale (2001)

Project duration: 2000-2003       

Budget: £400,000            

Sponsor: The Shirley Foundation

Summary:

This project allowed us to conduct leading edge research into use of 3D interactive virtual environments for ASD social skills training. The AS Interactive project aimed to find out whether people with ASDs could use desktop VEs successfully, and what they understood about them. We used a combination of experimental and qualitative approaches to investigate three main topics: use and understanding of virtual environments and the potential they offer for learning specific social skills. The same students with ASDs were included in all three studies and ranged in age from 13 to 18 years. All participants had a full scale IQ of 70 or greater. Users completed tasks in different scenarios including a virtual cafeteria and a virtual bus. In both scenarios, the task given to users involved finding a place to sit. This required understanding of social conventions regarding queueing and sitting near to other people. These tasks required them to make decisions about how to socially interact with virtual characters. They could ask questions or find out what the characters were thinking. The project found that ASD users do interpret virtual environments as real-world simulations and so can be used to assess and support social interaction skills. A teaching resource pack was produced to provide guided notes and printable material to support training activities in the classroom or community.

Publications:

• Rutten, A., Cobb, S., Neale, H., Kerr, S., Leonard, A., Parsons, S., and Mitchell, P. (2003). The AS Interactive Project: Single-User and Collaborative Virtual Environments for People with High-Functioning Autistic Spectrum Disorders.  Journal of Visualization and Computer Animation, 14, (5), 233-241.
• Cobb, S., Beardon, L., Eastgate, R., Glover, T, Kerr, S., Neale, H., Parsons, S., Benford, S., Hopkins, E., Mitchell, P., Reynard, G., & Wilson, JR. (2002). Applied Virtual Environments to support learning of Social Interaction Skills in users with Asperger’s Syndrome, Digital Creativity, Vol. 13, No.1, 11-22.
• PhD Thesis: Scaffolded learning in virtual environments. Steven Kerr (2005)

Sponsor: Particle Physics and Astronomy Research Council (PPARC)

HFRG members:

• Joanna Crosier
• Sue Cobb
• Mirabelle D’Cruz
• Richard Eastgate
• John R. Wilson

Publications:

• Crosier, J.K., Cobb, S.V. and Wilson, J.R. (2002) Key Lessons for the Design and Integration of Virtual Environments in Secondary Science, Computers and Education, 38, 77-94.
• Crosier, J.K., Cobb, S.V.G. and Wilson, J.R. (2000). Experimental comparison of virtual reality with traditional teaching methods for teaching radioactivity, Education and Information Technologies, 5(4), 329-343.

PhD Thesis: Evaluation of VR as a tool for secondary science education. Joanna Crosier (2000)

Project duration: 1998-2001                       

Sponsor: The European Commission, ESPRIT i3

Summary:

The KIDSTORY project produced two new collaborative storytelling technologies for children in the age group three to eight years. KidPAD is a shared drawing surface featuring a zooming interface, that allows children to create simple graphical stories and bring them to life by animated zooming. KidDIVE, is a formless three-dimensional object that acts like modelling clay. A group of children can pull it around, colour it and texture it in order to create characters and other story objects. Both of these technologies demonstrate an approach to interface design that encourages children to collaborate without forcing them to do so.

Another aspect of the KidStory project was the pioneering approach of 'children as inventors'. Instead of treating children as experimental subjects or as informants, they were involved as design partners and their design ideas incorporated. The project worked closely with children in two schools in Stockholm and Nottingham. Activities ranged from low technology prototyping sessions using cardboard, string, glue, crayons and other familiar materials, through to intensive technology immersion sessions using KidPAD and KidDIVE.