AHA Project

The Augmented Human Assistant proposes a set of ICT and Robotics based solutions to provide users, therapists and institutions with tools and methods to promote a healthy lifestyle in older adults, either for the prevention of sedentary behavior in elderly care institutions or for rehabilitation of motor deficits (e.g. stroke survivors). The approach of the project is based on the use of social assistive robots, interactive feedback and gamification of the exercise, to keep the users engaged in the exercise through motivational techniques (coaching, scores, levels, achievements, competition with peers).


With the aging of the population, many older persons fall into sedentary behavior either due to disease or lack of family/social support. Even when institutionalized in elderly care centers (nursing homes, daycare), these people spend most of their time sitting or lying down, and nursing staff does not have the tools and motivation to engage them in an active lifestyle. Many rehabilitation and maintenance exercise programs fail due to lack of adherence by the users because they do not have the proper motivation and support.


The primary goal of the “AHA: Augmented Human Assistance” project is to develop novel assistive technologies to promote exercise among the elderly and patients of motor disabilities. For exercise programs to be effective, it is essential that users and patients comply with the prescribed schedule and perform the exercises following established protocols. The deployment of automated systems that remind users of exercises, keep them engaged in game-like experiences through motivational techniques, and control the intensity and correctness of exercise execution, can provide a significant improvement in the physical and therapeutic benefits of the prescribed exercise programs. Moreover, they can help clinical staff or caretakers in monitoring and quantifying the progress of their patients and better customize the exercise programs for their needs without requiring actual consultation.

Augmented Assistance

The systems will be integrated on a mobile robotic platform with indoor navigation capabilities that will interact through a virtual coach system to monitor patients, provide reminders on tasks, guide patients through exercises, and assist them in daily routines.

Increasing self-awareness

Monitoring of user state by means of biosensors, computer vision systems and exercise performance data. User state will be assessed in a transparent manner and data will be visualized through friendly user interfaces, and shared with patients, clinicians and/or relatives.

Physical (re)training

We propose to develop adaptive Augmented Reality physical training tools that deliver online feedback on performance to prevent sedentarism, support active aging and provide personalized tools for function re-training in motor impaired patients


IST-ID – Computer and Robot Vision Lab (VisLab) of the Institute for Systems and Robotics (ISR-Lisbon) of Instituto Superior Técnico.

CMU – The Quality of Life Technology Center of the School of Computer Science (SCS) at Carnegie Mellon University (CMU).

M-ITI – NeuroRehabLab of the Madeira Interactive Technologies Institute (M-ITI) of the University of Madeira.

FCTUNL – The Foundation of the Faculty of Sciences and Technology of “Universidade Nova de Lisboa” (FFCT/UNL).

FMH – The Interdisciplinary Centre for the Study of Human Performance (CIPER) of the Faculty of Human Kinetics-FMH.

PLUX – PLUX www.plux.info creates innovative products for Physiotherapists and Researchers, by developing advanced biosignal monitoring platforms.

YDR – YDreams Robotics (www.ydreamsrobotics.com/) is a spin-out of YDreams specializing in robotics, mechatronics, and the Internet of Things.

Advisory Committee

The project has set up an Advisory Committee (AC) to assess and ensure quality, results and provide guidance. The AC consists of 6 external members distributed in the following way:

Fernando Lopes da Silva, M.D., Ph.D.
Emeritus Professor and at the Center of Neuroscience of the Swammerdam Institute for Life Sciences, Amsterdam, The Netherlands.

Paolo Dario
Professor of Biomedical Engineering and Director of The BioRobotics Institute of the Scuola Superiore Sant’Anna (SSSA), Pisa, Italy, the Founding Coordinator of the Center of Micro-BioRobotics@SSSA of the Italian Institute of Technology (IIT).

Steve Kelly, Chairman and COO of Myomo Inc. (Boston, USA)
Also advisor at MIT for the Deshpande Center for Technological Innovation and Venture Mentoring Service where he has led the commercialization of late stage research projects.

Kynan Eng, PhD
Co-founder and chair of scientific advisory board, YouRehab AG (Zurich, Switzerland). Technology transfer coordinator at the Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland

Scott Bleakley, PhD
Director of Physical Therapy at the HealthSouth Sewickley Rehabilitation Hospital, Pittsburgh area, USA.

Maria Vânia Silva Nunes, PhD
Professor at the Health Sciences Institute of the Universidade Católica Portuguesa and responsible for the dependent and elderly population area at the Portuguese Red Cross (Cruz Vermelha Portuguesa).


Committee members will:

a) Provide guidance on research, education/training and innovation programs and their strategic directions

b) Act as liaisons between the project and the stakeholders they represent, guiding translation of research into innovation

c) Share best practices, challenges, and successes


The integrated AHA system will be composed of:

a) A mobile robotic platform with advances in perception, navigation and control skills, leveraged with an extended set of sensors for human sensing and emotional state estimation.

b) Serious gaming abilities through novel augmented reality methods yielding extended feedback modalities for physical exercising and motor rehabilitation.

c) A virtual coach system with technologies and techniques that assist and encourage users while they perform rehabilitation exercises, and instills better compliance with their prescribed exercise regimen.


The AHA project brings together innovation and research in a cross-disciplinary consortium with expertise in such diverse areas such as Human Functioning and Performance, Augmented Reality (AR) technologies, serious games for health, physiological signal acquisition systems, computer vision systems, robot navigation and intelligent scene assessment.

The contributions of this complementary expertise are organized in the following work packages:

WP1 – Augmented Reality Training;

WP2 – Human State Awareness;

WP3 – Virtual Coach. Each of these work packages is responsible for a so-called AHA module, which are the core technology components of the AHA static platform.

WP4 – Robotic Assistance Platform results from the integration of the static platform
with a mobile robotic device.

WP5 – End user evaluation is a transversal work package to evaluate the
separate components as well as the static and mobile platforms in realistic scenarios.

Dissemination and exploitation are addressed in WP6.


Augmented Reality Serious Games

Through a diversity of exergames with gamification components to keep users engaged, we hope to break sedentary cycles more often and reduce the associated health risks. The Serious Games take advantage of AR technology to implement virtual tasks with cultural and historical relevance for the target population. These exercises are designed to have a positive impact on daily life activities, promoting autonomy and independence while providing entertainment and sociability to the elderly.

  • The main objective is the user to collect each musical note.
  • Lower limb motor coordination exercises (collect musical notes)
  • Arm extension (catch the stars - optional )
  • The main goal is to paddle on the river and collect as many barrels as possible while avoiding rocks
  • Upper limbs exercise (circular arm movements simulating rowing gestures to move the boat forward)
  • Balance / side gait / lateral rotation of the trunk - Adaptive interaction mechanics, for the lateral displacements of the boat, users can be standing and use lateral movements or trunk leaning or they can be seated using trunk leaning.
  • The main goal of this Exergame is to stomp on grapes, pulled from the conveyor belt into the tank.
  • Flexion-extension arm movements (drag the grape bunches into the tanks)
  • Static Gait (stomping grapes process)
  • Aerobic training
  • The main goal in this Exergame is to successfully collect as much objects as possible in a pre-established time avoiding crashes with cars and pedestrians.
  • Trunk postural training (flexion accelerate sledge and extensions to deaccelerate)
  • Balance (directions of sledge - optional)
  • Lateral gait (directions of sledge – optional)
  • The main objective it is avoid the ball fall for not hit the paddle dimension limits
  • Lateral gait (paddle side movements)
  • Balance (when played standing)
  • Upper limbs movements (control the paddle when played seated)
PEPE - Portable Exergame Platform for Elderly People

Pepe is an easily deployable and configurable platform for physical training with exergames.

Value Propositions Technological Innovations
  1. Older people playing through floor/ wall augmented reality projections
  2. Physical (re)training through serious games while playing
  3. Automatic game intensity and difficulty concerning each senior
  4. Get real-time seniors physiological signals values
  5. Allows automated assessment of seniors physical fitness levels
  6. Health professionals can easily configure and operate PEPE
  1. An augmented reality (AR) component through video projection in the environment
  2. Serious games for physical training
  3. A hand-grip device to measure physiological signals
  4. Real-time feedback and game adaptation
  5. Automated assessment of physical fitness levels
  6. Easy operation and configuration by health professionals in care homes

PEPE exercises are designed to have a positive impact on daily life activities, promoting autonomy and independence while providing entertainment and sociability to the elderly.

Such platform will define a new class of assistive devices for healthy, elderly and patient users, allowing new modalities of interaction and engagement not yet available in the state-of-the-art.

Virtual Coach

Virtual coach techniques are able to monitor the correctness of the exercise and provide immediate guidance to the user, as well as providing reports for therapists. During the project we made significant progress on developing novel scoring functions to evaluate performance, that learn from the evaluations of therapists from captured datasets. Also, an automatic goal adjustment method is on continually updating and evolving the model with the most recent observations.

Vizzy as Coach

Vizzy is a robot for the promotion of physical exercise among the elderly. It will be a personal assistant robot, designed to teach exercise routines and provide physical therapy support, while also ensuring proper exercise form and monitoring physiological responses.

The aim of Vizzy is to be introduced to institutions of the elderly and to captivate them and motivate them to exercise. Both Vizzy and the Portable Exergame Platform for Elderly (PEPE) are used to elicit users and professionals for exergaming.

The last goal is to be Vizzy eliciting people to exercise and be himself to project the exergames on the floor while being a personal trainer during the user performance.


The Handgrip has the function of being a complementary device to PEPE. It will be used before the start of each game, during the game and also after the end of the game. The goal is to collect and monitor physiological data such as heart rate, sweating levels on the skin and manual strength.

The Handgrip has the function of being a complementary device to PEPE. It will be used before the start of each game, during the game and also after the end of the game. The goal is to collect and monitor physiological data such as heart rate, sweating levels on the skin and manual strength.

The Handgrip was developed taking into account the manual limitations of some elderly people, namely regarding arthrosis or muscolo-skeletal atrophies.

The goal is to collect and monitor physiological data such as heart rate, sweating levels on the skin and manual strength.