Computer-assisted teaching of Sign Language using Computer Vision and Machine Learning

The CAT-SL aims to fill the skill gap in SL teaching in higher education, due to scarcity of available expertise. It is related to the social inclusion priority, since it aims to educate educators of children with special needs. Finally, it aims to develop innovative technologies for higher education, thus justifying the innovative practices in the digital era priority.

Context
According to the European Union of Deaf, more than 750.000 deaf SL users live in EU, while only 12.000 interpreters are registered. This fact highlights the barriers that deaf and hard-of-hearing (HoH) people are facing, specifically when it comes to communication with hearing people and equal opportunities to proper education. For instance, in Greece and Portugal more than 60% of the deaf children have extremely limited reading and writing skills. Hence, it is no coincidence that deaf children, initially educated in SL, usually stop their studies at the elementary school level. Education as a fundamental right is mentioned not only in the Preamble of the UN Convention on the Rights of Persons with Disabilities (UNCRPD), ratified by the EU as a whole in January 2011, but also in a dedicated article on education (Article 24).

Deaf children must be able to become active citizens who are employed and contribute to the society, being taxpayers rather than living on social benefits. Those children obviously need a lot of support at school. Therefore, the educators need to receive appropriate training in Sign Languages, the native languages of the Deaf, so that they are able to communicate with them and to teach them how to develop linguistically. Sign Languages in the curricula of Primary Education University Departments or Departments dealing with Special Education definitely contribute towards this goal. However, it is hard for Higher Education Institutes to find and employ as tutors experts in Sign Language, so many educators are not adequately trained as students to face the challenges of communicating with deaf children in the classroom.

Objectives
a. To develop an innovative and affordable CAT-SL system for interactive SL teaching for students in Special Education or Pedagogical departments. The system will be based on computer-vision, machine-learning, linguistic technology and avatars developed by all the involved partners in several national and EU-funded projects that are ongoing or recently completed. Thus we aim at an exploitation of reserarch results at European level that would not be possible without the contribution of the EU.
b. To set up the CAT-SL infrastructure in four Higher Education Institutes, based on open standards, to enable its practical use by tutors and university students (the future teachers of the deaf children).
c. To develop curricula for teaching SL using the CAT-SL system for at least three university courses, in Greece, Cyprus and Portugal. This way we aim to support the social inclusion of the deaf children.
d. To actively disseminate the project results through the promotion of workshops with stakeholders in the EU to raise awareness on the main challenges faced by the Deaf.

Target groups
a. The students of Special Education university departments, aiming to learn how to use and teach SL to deaf children.
b. The university professors teaching SLs in Special Education departments.
c. Educators in public or private organizations involved in SL teaching or Deaf education.
d. Any organization, institute, company or individual interested in promotion of the use of SLs and related technology.
e. Local, regional, national and international bodies related to inclusion and citizenship.

The needs
SL courses are essential in Special Education University departments, the Deaf being one of the largest target groups. The universal lack of SL tutors may undermine the quality of the provided courses. Therefore the CAT-SL aims to complement the practice sessions of related curricula with automated exercises to be performed by students and automatically evaluated by the system. Therefore the needs to be covered are:
a. An automated system for interactive SL teaching in various national languages, which will be affordable and easy to deploy and maintain.
b. Related curricula in the SL courses conforming to EU-wide standards.
c. A community to adopt and support the related technology and motivate further technological developments using open standards.

Transnational project
The efforts for SL analysis systems are fragmented over Europe. There are different groups focusing on machine learning, computer vision and avatar design, SL linguistics, SL didactics etc, however it is very hard for single national groups to encompass all the related know-how. Furthermore, the problem is not of local importance since there are many national SLs. Approaches that apply to a single SL may not be applicable to other ones. Therefore, a more global approach is necessary to develop a system and IOs conforming to EU-wide standards. In CAT-SL we gather together HEIs with significant track-record in the development of methods for SL analysis as proved by recent projects and related publications. This track-record would not be possible without the EU funding. In CAT-SL we aim to leverage the acquired knowledge from at least four different projects and disseminate the results internationally.

Funding
Funded by the Erasmus+ programme
Project code: 2020-1-EL01-KA203-079232