HEALTHSIGN - Sign language analysis on mobile devices for interpretation with emphasis on health services

Sign Languages (SLs) are the main means of communication for deaf people. The access to SL is essential for the fulfillment of basic Human Rights, however there is a shortage of interpreters, which undermines these rights and often endangers the lives of the deaf, especially in cases of emergency or serious health incidents.

HealthSign proposes to develop an application for the automated interpretation of the Greek Sign Language (GSL) over internet with focus on the health services, which are the most common reason to seek for an interpreter. The high demand for interpreters is often not met or requires long waiting. Furthermore, the availability of interpreters facilitates the integration of the deaf community.

Vision is probably the only sensor modality that could be of practical use because (a) only vision can capture manual and non-manual cues, which provide essential information for SLR, (b) camera-equipped hand-held devices with powerful processors are a commodity nowadays and (c) recent advances in computer vision and machine learning render mainstream visual SLR a realistic option.

The HealthSign project aims to fulfil the following goals:

G1:Develop a database of GSL from native speakers with emphasis on health services
G2: Implement an internet-based platform for synchronous communication and interpretation with health professional
G3:Develop in parallel a lightweight version which will be able to run on an embedded platform.
G4: Develop algorithms for recognition of SLs, using computer vision and deep learning using the hand and body/facial cues
G5: Implement the algorithms on embedded platforms using FPGAs

Financial goals: The main goal is the viability of the proposed application which will be achieved by (a) simple off-the-shelf equipment for the users (b) efficient implementation of the proposed algorithms (c) simple installation (d) development of a business plan to facilitate the longevity of the proposed product.

The consortium is composed of (a) The Signal Processing and Telecommunications Lab of the University of Patras as expert in machine learning which is necessary for the recognition of GSL (b) the Computer Vision and Robotics Lab of ICS-FORTH, which will adapt their 3D hand model for tracking (c) the Bioassist company, which specializes in assistive technologies and develops an internet based platform (d) the IRIDA S.A. to develop the embedded application and Deaf Studies Unit at the University of Patras, which will bring the users, the interpreters and the GSL experts.

This work is partially supported by the Greek Secretariat for Research and Technology, and the EU, Project HealthSign: Analysis of Sign Language on mobile devices with focus on health services T1EΔK-01299 within the framework of “Competitiveness, Entrepreneurship and Innovation” (EPAnEK) Operational Programme 2014-2020

Dimitrios Kosmopoulos: Principal investigator and project coordinator.

http://xanthippi.ceid.upatras.gr/HealthSign/EN/HealthSign.html