Post-Doctoral Position: Synthesis Of Videos Driven By Text And Audio
Description of the research project
In recent years, voice interaction with computers has made significant progress. Virtual agents offer a user-friendly human-machine interface while reducing maintenance costs. Speechbased interaction is already effective, as proved by Siri, Google Assistant or Alexa virtual agents, however, their visual counterpart is still far behind. The level of user engagement for audiovisual interactions is much higher than for purely audio interactions. Therefore, it is desirable to be able to associate face visual animations with the generated audio.
A notable advancement in video generation was made by a team at Stanford University in 2019 in partnership with Adobe 1. Their work is aimed at enabling a video editing technology of a person's face-to-face scene to revise its speech script and adapt the rendering automatically based simply on this revised text.
The latest advances in the field of audio-driven face video synthesis were presented in 2. The proposed approach generalizes across different people, to synthesize videos of a target actor with the voice of any actor from an unknown source or even synthetic voices that can be generated using standard text-to-speech approaches.
During this post-doctoral contract, we want to work on the development of a prototype of a Text to Speech to Video technology with a sufficient level of accuracy.
Duration, place to work and supervisors
The funding covers 18 months of post-doc, the desired start is April-May 2022. The postdoctoral fellow will be attached to the LIRIS (Laboratory of Computer Science in Image and Information Systems) on the campus of the University Lyon 2 in Bron. Some stages of work can be conducted in the office of Mon Petit Placement in Lyon.
The post-doc will be supervised by IuliiaTkachenko and Serge Miguet (LIRIS). The project manager on the side of Mon Petit Placement is the technical director of the startup Thibault Jaillon.
References
1 O. Fried, A. Tewari, M. Zollhöfer, A. Finkelstein, E. Shechtman, D. Goldman, K. Genova, Z. Jin, C. Theobalt, M. Agrawala, “Text-based editing of talking-head video”, ACM Transactions on Graphics (TOG), Vol.38, 2019.
2 J. Thies, M. Elgharib, A. Tewari, C. Theobalt, M. Niessner, “Neural Voice Puppetry: Audio-driven Facial Reenactment”, ECCV 2020 (https: // justusthies.github.io/posts/neural-voice-puppetry/ )
Offer Requirements Specific Requirements
The candidate must have a PhD in computer science, specializing in image and video processing
Programming languages: Python/C++
Neural network libraries: PyTorch/Keras/Tensorflow
Programming tools for image analysis: OpenCV
Scientific knowledge: machine learning and deep learning, video analysis and processing
Good writing skills and proficiency in written and spoken English (French is not a requirement)
Contact Information
Organisation/Company: CNRS LIRIS
Organisation Type: Small Medium Enterprise, Start-up
Website: https: // perso.liris.cnrs.fr/itkachenko/
Country: France
为防止简历投递丢失请抄送一份至：boshijob@126.com（邮件标题格式：应聘职位名称+姓名+学历+专业+中国博士人才网）
中国-博士人才网发布
声明提示：凡本网注明“来源：XXX”的文/图等稿件，本网转载出于传递更多信息及方便产业探讨之目的，并不意味着本站赞同其观点或证实其内容的真实性，文章内容仅供参考。
