Research Engineer - Domain Adaptation in NMT

2023-06317 - Research Engineer - Domain Adaptation in NMT

Contract type : Fixed-term contract

Level of qualifications required : Graduate degree or equivalent

Fonction : Temporary scientific engineer

Context

The chosen applicant will carry out research in the context of the DadaNMT Emergence project funded by Sorbonne Université. The aim of the project is to explore approaches to training MT models (i) for low-resource domains by using out-of-domain data (domain adaptation) and (ii) that are capable of handling texts from different domains (multi-domain models). We will apply our approaches across three domains: film subtitles (using different film genres), news (from different countries) and biomedical data. We will also test across different language pairs, focusing on at least the three following pairs: English-French, English-German and English-Romanian, chosen because of the availability of training and test data across the domains and because they represent three different scenarios in terms of the amount of data available. The three chosen domains have freely available parallel data available: film subtitles from OpenSubtitles (Lison et al. 2018), news data from the WMT news translation shared tasks (Barrault et al. 2019) and biomedical data from the WMT biomedical MT shared tasks (Neves et al. 2019).

Assignment

The aim of the topic will be to explore directions for the adaptation of machine translation (MT) to low-resource domains and/or the development of MT models capable of handling several domains, while maintaining performance of the model on those domains that are well represented (i.e. avoid catastrophic forgetting). There are several possible directions that could be explored depending on the interest and experience of the candidate, including:

Curriculum learning: how to best introduce training examples such that the model can learn better (e.g. from simpler to more complex examples, transitioning from one domain to another, etc.) (Platanios et al., 2019; Zhan et al., 2021)

Meta-learning: how to best initialise a model in order for it to be able to robustly adapt to new domains, particularly those that have few training examples. (Sharaf et al., 2020; Zhan et al., 2021)

Using large-scale pretrained LLMs to see how well they can be adapted to translation for specific domains. This could either involve conversational LLMs or traditional LLMs. In the first case, potential approaches could include the decomposition of the translation of new examples into composite tasks via chain-of-thought prompting (Wei et al., 2022; Wang et al., 2022) or the training of composite tasks (Bursztyn et al., 2022) each of lower simplicity (e.g. word translation, formulation, reformulation) and also translation refinement using adapted prompts. In the second case, the approach could follow prompt selection and example selection.

References

Victor Bursztyn, David Demeter, Doug Downey, and Larry Birnbaum. 2022. Learning to perform complex tasks through compositional Fine-Tuning of language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1676–1686, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. Competence- based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1162–1172, Minneapolis, Minnesota.

Amr Sharaf, Hany Hassan, and Hal Daum ́e, III. 2020. Meta-Learning for Few-Shot NMT adaptation. In Proceedings of the Fourth Workshop on Neural Generation and Translation.

Boshi Wang, Xiang Deng, and Huan Sun. 2022. Iteratively prompt pre-trained language models for chain of thought. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2714–2730, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. Adv. Neural Inf. Process. Syst., 35:24824–24837.

Runzhe Zhan, Xuebo Liu, Derek F Wong, and Lidia S Chao. 2021. Meta- Curriculum learning for domain adaptation in neural machine translation. In Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence (AAAI-21), pages 14310–14318, Online.

Main activities

The main activities include carrying out reading on the proposed topic, experimenting with baselines (previous work) and proposing and implementing new solutions. The work carried out will be presented both within the team and internationally (should the work be accepted as a peer-reviewed conference or workshop).

Skills

We are looking for an applicant with a good experience in machine learning, natural language processing and a strong interest for linguistics. The applicant must have a good level of written and spoken English.

Benefits package

Subsidized meals

Partial reimbursement of public transport costs

Leave: 7 weeks of annual leave + 10 extra days off due to RTT (statutory reduction in working hours) + possibility of exceptional leave (sick children, moving home, etc.)

Possibility of teleworking

Professional equipment available (videoconferencing, loan of computer equipment, etc.)

Social, cultural and sports events and activities

Access to vocational training

Social security coverage

General Information

Theme/Domain : Language, Speech and Audio

Town/city : Paris

Inria Center : Centre Inria de Paris

Starting date : 2023-08-01

Duration of contract : 4 months

Deadline to apply : 2023-06-15

Contacts

Inria Team : ALMANACH

Recruiter : Bawden Rachel / [email protected]

About Inria

Inria is the French national research institute dedicated to digital science and technology. It employs 2,600 people. Its 200 agile project teams, generally run jointly with academic partners, include more than 3,500 scientists and engineers working to meet the challenges of digital technology, often at the interface with other disciplines. The Institute also employs numerous talents in over forty different professions. 900 research support staff contribute to the preparation and development of scientific and entrepreneurial projects that have a worldwide impact.

Instruction to apply

Defence Security : This position is likely to be situated in a restricted area (ZRR), as defined in Decree No. 2011-1425 relating to the protection of national scientific and technical potential (PPST).Authorisation to enter an area is granted by the director of the unit, following a favourable Ministerial decision, as defined in the decree of 3 July 2012 relating to the PPST. An unfavourable Ministerial decision in respect of a position situated in a ZRR would result in the cancellation of the appointment.

Recruitment Policy : As part of its diversity policy, all Inria positions are accessible to people with disabilities.

Warning : you must enter your e-mail address in order to save your application to Inria. Applications must be submitted online on the Inria website. Processing of applications sent from other channels is not guaranteed.