Project description
The PhD project will contribute to the development of robust and generalizable sign-to-text translation for low-resource sign languages, with a focus on LSF and DGS. Rather than concentrating only on the final translation model, the PhD will address the upstream components that are essential for reliable sign-to-text translation: data collection and curation, privacy-aware data processing, automatic video-text alignment, and sign-language representation learning.
A first objective will be to identify, collect, structure, and curate sign-language video resources for LSF and DGS, for which available resources remain more limited than for some other sign languages and domains [2,3]. The work will exploit existing and newly identified video-text resources, including open repositories and potential data sources from associations or broadcasters. Since sign-language data typically consists of videos of identifiable signers, the PhD will also address privacy and ethical challenges, including consent, anonymization, secure data handling, metadata quality, and linguistic diversity.
A second objective will be to develop automatic methods for aligning sign-language videos with corresponding written transcriptions, subtitles, or other textual material. Such alignment is essential for transforming raw or weakly structured video-text data into resources that can be used for machine learning. The PhD will therefore investigate multimodal approaches that connect visual sign-language information with textual representations under low-resource conditions, using techniques from computer vision, natural language processing, and representation learning [4,5].
A third objective will be to study how sign-language videos can be represented in a way that captures the visual and temporal complexity of signed communication, including manual signs, body posture, facial expression, mouthing, spatial structure, and timing. The PhD will explore how sign-language-specific pre-trained and multimodal models can be adapted to LSF and DGS data, in order to support video-text alignment, representation learning, and downstream sign-to-text translation [9,10].
Implementation plan
The first stage of the PhD will focus on data collection and curation. The candidate will identify and gather relevant LSF and DGS video-text resources from open repositories and other potential sources. This will include the analysis of available metadata, text-video correspondence, recording conditions, signer variation, and data quality. Particular attention will be paid to privacy-aware processing, including anonymization strategies, secure data handling, and the possible use of pose-based or derived representations to reduce identifiability.
The second stage will focus on automatic alignment between sign-language videos and written text. The candidate will investigate methods for aligning videos with subtitles, transcriptions, or other textual material, to transform weakly structured data into usable training and evaluation resources. This work may involve temporal segmentation, visual feature extraction, sentence-level semantic representations, multimodal similarity learning, and Transformer-based or CLIP-like architectures adapted to sign-language data [4,5].
The third stage will explore the use of pre-trained and foundation models for sign-language representation learning. The candidate will study how computer vision models, multimodal models, and large language models can be adapted or fine-tuned for sign-language data. This may include keypoint-based, video-based, or hybrid representations, as well as weakly supervised or contrastive learning strategies for connecting visual and textual information.
In the final stage, the developed data-processing, alignment, and representation-learning components will be assessed in relation to downstream sign-to-text translation. The work will include quantitative and qualitative analysis of alignment quality, robustness across data sources and signers, and the usefulness of the learned representations for translation-oriented tasks. While the PhD is not primarily centred on evaluation methodology, the candidate will use appropriate automatic and human-informed analyses to validate the proposed methods [6].
The PhD candidate is expected to contribute to both scientific knowledge and practical resources for the sign language technology community. Expected outcomes may include:
- Curated or enriched LSF and DGS video-text resources;
- Improved automatic alignment methods for sign videos and written text;
- Multimodal representations of sign-language videos suitable for downstream translation;
- Adaptation of pre-trained computer vision, multimodal, and language models to sign-language data;
- Open-source implementations of data-processing, alignment, and modelling components;
- Experimental analyses demonstrating the usefulness of the proposed methods for sign-to-text translation;
- Publications in leading conferences and workshops in NLP, computer vision, or sign language technology.
[1] N. C. Camgöz, S. Hadfield, O. Koller, H. Ney, and R. Bowden, “Neural Sign Language Translation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[2] J. Forster, C. Schmidt, O. Koller, M. Bellgardt, and H. Ney, “Extensions of the Sign Language Recognition and Translation Corpus RWTH-PHOENIX-Weather,” in Proceedings of the International Conference on Language Resources and Evaluation (LREC), 2014.
[3] J. Halbout, D. Fabre, Y. Ouakrim, J. Lascar, A. Braffort, et al., “Matignon-LSF: A Large Corpus of Interpreted French Sign Language,” in Proceedings of the LREC-COLING Workshop on the Representation and Processing of Sign Languages, 2024.
[4] H. Bull, T. Afouras, G. Varol, S. Albanie, L. Momeni, and A. Zisserman, “Aligning Subtitles in Sign Language Videos,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[5] Y. Hamidullah, J. van Genabith, and C. España-Bonet, “Sign Language Translation with Sentence Embedding Supervision,” in Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Short Papers, 2024.
[6] M. Müller et al., “Findings of the Second WMT Shared Task on Sign Language Translation,” in Proceedings of the Conference on Machine Translation (WMT), 2023.
[7] G. Fauré, M. Sadeghi, S. Bigeard, and S. Ouni, “Towards Skeletal and Signer Noise Reduction in Sign Language Production via Quaternion-Based Pose Encoding and Contrastive Learning,” in Proceedings of SLTAT 2025: 9th Workshop on Sign Language Translation and Avatar Technologies, 2025.
[8] G. Fauré, M. Sadeghi, S. Bigeard, and S. Ouni, “The Impact of VAE Design on Latent Pose Representations for Diffusion-based Sign Language Production,” in Proceedings of GenSign: Generative AI for Sign Language, CVPR 2026 Workshop, 2026.
[9] Z. Jiang, G. Sant, A. Moryossef, M. Müller, R. Sennrich, and S. Ebling, “SignCLIP: Connecting Text and Sign Language by Contrastive Learning,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9171–9193, 2024.
[10] P. Jiao, Y. Min, and X. Chen, “Visual Alignment Pre-training for Sign Language Translation,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 349–367, 2024.
