Phd Position In Natural Language Processing: Statistical Models Of Disagreement Between Annotators (0.8 – 1.0 Fte)

Utrecht University
May 29, 2023
Offerd Salary:€2,541
Working address:N/A
Contract Type:Other
Working Time:Full time
Working type:N/A
Ref info:N/A
PhD position in Natural Language Processing: statistical models of

disagreement between annotators (0.8 – 1.0 FTE)

Job description

In Natural Language Processing (NLP), human annotators are frequently needed to tell researchers what a given expression “means”, by assigning the expression a label. When human judges disagree about a label (e.g., whether an utterance is offensive or not), it is important that these disagreements be taken into account, as opposed to simply aggregating the values e.g., using reconciliation or majority voting. Such disagreements are now generally recognized to provide information rather than being noise. We also need to recognize that uncertainties may originate from different sources: in particular, they may be due to semantic ambiguity or to subjective bias.

  • whether the differences between various sources of disagreement (e.g., noise, ambiguity, and subjective bias) can be detected using statistical models.
  • how to use datasets containing different types of variation to train and evaluate NLP models, starting from datasets such as those created in DALI and LeWiDi. It may also leverage other datasets that have recently appeared, e.g., for natural language inference.
  • whether the obvious candidates for soft evaluation metrics (cross- entropy, Kullback-Leibler divergence) apply to all these tasks.
  • to what extent variations in one person's verbal behaviour can be understood mathematically in the same way as variations between different speakers.
  • This PhD position is one of six inter-connected PhD positions focussing on uncertainty in NLP, under Utrecht University's AiNed project “Dealing with Meaning Variation in NLP”, which will be led by Professor Massimo Poesio. We are simultaneously recruiting for another of the six PhD position in this project We invite you to also check out this interesting vacancy on our website.

    Qualifications You are a motivated researcher with a curious and critical mindset. You hold a Master's degree in an area relevant to this project; this could be Artificial Intelligence, Computational Cognitive Science, Computing Science, Linguistics, or Statistics. A good mastery of statistics and deep learning is essential; experience with NLP would be beneficial. Excellent English communication skills are necessary.


    A position for 4 years;

  • A full-time gross salary that starts at €2,541 and increases to €3,247 per month (scale P of the Collective Labour Agreement Dutch Universities (CAO));
  • 8% holiday bonus and 8.3% end-of-year bonus;
  • A pension scheme, partially paid parental leave, and flexible employment conditions based on the Collective Labour Agreement Dutch Universities.
  • In addition to the employment conditions laid down in the CAO for Dutch Universities, Utrecht University has a number of its own arrangements. For example, there are agreements on professional development, leave arrangements and sports. We also give you the opportunity to expand your terms of employment via the Employment Conditions Selection Model. This is how we like to encourage you to continue to grow.

    About the organization

    Sharing science, shaping tomorrow. A better future for everyone. This ambition motivates our scientists in executing their leading research and inspiring teaching. At Utrecht University, the various disciplines collaborate intensively towards major strategic themes. Our focus is on Dynamics of Youth, Institutions for Open Societies, Life Sciences and Sustainability.

    Additional information

    If you have questions about this position, don't hesitate to contact Professor Massimo Poesio at [email protected] or Professor Kees van Deemter at [email protected].

    Everyone deserves to feel at home at our university. We welcome employees with a wide variety of backgrounds and perspectives.

  • your letter of motivation
  • your curriculum vitae
  • scans of BSc and/or MSc transcripts (incl. grade list), with a certified translation in English if the degree qualification is not in German, Dutch, or English
  • your MSc-thesis (if not available, then BSc-thesis)
  • the contact details of at least two references
  • If this specific opportunity isn't for you, but you know someone else who may be interested, please forward this vacancy to them.Some connections are fundamental – Be one of them

    The application deadline is 29/05/2023

    From this employer

    Recent blogs

    Recent news