Few-shot learning for motion inbetweening techniques in character animation

2023-05938 - Few-shot learning for motion inbetweening techniques in character animation

Contract type : Internship agreement

Level of qualifications required : Master's or equivalent

Fonction : Internship Research

Context

This internship is part of a joint collaboration between the company Mercenaries Engineering which develops the Rumba animation tool, and the IRISA / Inria Center at Rennes University. Motion in-betweening in animation is the task that consists in creating intermediate animations between two given keyframes while preserving the naturalness of the motion. This is a classical task in the creation pipeline of artists, and recent contributions in the field have been proposing ways to automatically generate these in-between motions using deep-learning techniques. Harvey etal. 1 rely on conditioned Recurrent Transition Networks, extended with time-to-arrival information, to create in-between motions even with sparse keyframes for animated characters. The work has been applied to problems of in-betweening in virtual cinematography 2, to improve pose estimation of occluded characters 3 and have even inspired the most recent motion diffusion models 4. Most approaches require large datasets to ensure natural motions, which may be sparse or hard to access when considering more cartoon- style animations.

Assignment

The objective of this internship is to design a motion in-betweening technique for character animation inside the Rumba animation tool. Given a small dataset of existing animations ( e.g. representing sample motions of a character available in a studio), our purpose is to design a tool capable of exploiting this sparse data to generate automatically in-between motions. Additional specification constraints (speed / feet contacts) may also be considered to improve the editing capacity of the technique.

The work will start by reproducing the results of classical motion in- betweening techniques such as 156. We will then explore how existing datasets may be augmented with dedicated sparse data provided directly by users of the system, to partially retrain the model and generate results which are visually similar to the sparse data. Dedicated noise generators may also need to be designed to ensure that the keyframe constraints can be reached even with low amounts of data by using the delta- interpolator 5. We will finally explore how additional editing constraints can be added on the in-betweening process to provide creative designers with more high-level controllers (speed, trajectory, amplitude, style).

The research will be closely conducted with Mercenaries Engineering who will provide access to their tools and datasets. Results will be integrated in the Rumba animation tool with the support of the Rumba R&D team.

Harvey, F. G., Yurick, M., Nowrouzezahrai, D., & Pal, C. (2020). Robust motion in-betweening. ACM Transactions on Graphics (TOG) , 39 (4), 60-1.

Jiang, H., Christie, M., Wang, X., Liu, L., Wang, B., & Chen, B. (2021). Camera keyframing with style and control. ACM Transactions on Graphics (TOG) , 40 (6), 1-13.

Yuan, Y., Iqbal, U., Molchanov, P., Kitani, K., & Kautz, J. (2022). GLAMR: Global occlusion-aware human mesh recovery with dynamic cameras. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11038-11049).

Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., & Bermano, A. H. (2022). Human motion diffusion model. arXiv preprint arXiv:2209.14916.

Oreshkin, B. N., Valkanas, A., Harvey, F. G., Ménard, L. S., Bocquelet, F., & Coates, M. J. (2022). Motion Inbetweening via Deep $Delta $-Interpolator. arXiv preprint arXiv:2201.06701.

Kim, J., Byun, T., Shin, S., Won, J., & Choi, S. (2022). Conditional motion in-betweening. Pattern Recognition , 132 , 108894.

Main activities

analyse existing contributions in the field

extend existing work with few-shot learning techniques

propose an implementation in the Rumba engine

Skills

Languages: Python scripting, C/C++ (for Rumba integration)

Technology: Deep Learning, Rumba

General Information

Theme/Domain : Interaction and visualization Scientific computing (BAP E)

Town/city : Rennes

Inria Center : Centre Inria de l'Université de Rennes

Starting date : 2023-05-01

Duration of contract : 4 months

Deadline to apply : 2023-04-30

Contacts

Inria Team : VIRTUS

Recruiter : Christie Marc / [email protected]

The keys to success

5 to 6 months Research or Research and Development internship

Good background knowledge on Deep Learning and Computer Graphics techniques

Master student or Engineering school

About Inria

Inria is the French national research institute dedicated to digital science and technology. It employs 2,600 people. Its 200 agile project teams, generally run jointly with academic partners, include more than 3,500 scientists and engineers working to meet the challenges of digital technology, often at the interface with other disciplines. The Institute also employs numerous talents in over forty different professions. 900 research support staff contribute to the preparation and development of scientific and entrepreneurial projects that have a worldwide impact.

Instruction to apply

Defence Security : This position is likely to be situated in a restricted area (ZRR), as defined in Decree No. 2011-1425 relating to the protection of national scientific and technical potential (PPST).Authorisation to enter an area is granted by the director of the unit, following a favourable Ministerial decision, as defined in the decree of 3 July 2012 relating to the PPST. An unfavourable Ministerial decision in respect of a position situated in a ZRR would result in the cancellation of the appointment.

Recruitment Policy : As part of its diversity policy, all Inria positions are accessible to people with disabilities.

Warning : you must enter your e-mail address in order to save your application to Inria. Applications must be submitted online on the Inria website. Processing of applications sent from other channels is not guaranteed.