Research Scientist - Video Generation Models

About Cantina Labs

Cantina Labs is a pioneering social AI enterprise focused on creating cutting-edge real-time models that enhance expression, personality, and realism. Our innovations bring digital characters to life, revolutionizing storytelling, connectivity, and creativity. Our flagship platform, Cantina, is merely the starting point.

Role Overview

We are expanding our Singapore team and seeking a Research Scientist to lead foundational research in video generation models. This role encompasses end-to-end management of the research process and spearheading advancements in post-training techniques. Close collaboration with data, infrastructure, and modeling teams is essential to translate research outcomes into impactful improvements in our models.

Key Responsibilities

• Develop and sustain scalable frameworks for processing and managing large video datasets to support training needs.
• Architect and grow distributed pipelines for tasks such as data preprocessing, dataset creation, and routine dataset updates.
• Manage workflow orchestration, schedule computational jobs, monitor processes, and handle failure recovery for extensive data operations.
• Implement and oversee containerized infrastructures utilizing Kubernetes or similar orchestration platforms.
• Optimize cloud data storage and transfer mechanisms across platforms like AWS, GCS, or Azure to balance costs, speed, and operational efficiency.
• Establish and apply best practices concerning dataset storage formats, version control, caching, retention policies, and access methodologies.
• Create tools for efficient large-scale deduplication, including near-deduplication processes over expansive video datasets.
• Conduct research and develop distillation approaches specific to large-scale diffusion and flow-based video generation models, focusing on maintaining or enhancing output quality while minimizing inference resource consumption.
• Build and refine reward models and preference-driven fine-tuning methods that align generated video quality with human evaluations on aesthetics, motion fidelity, and prompt compliance.
• Investigate how foundational model characteristics influence outcomes post-training, collaborating with the foundational model team to guide pretraining strategies.

Candidate Profile

• Proven expertise in constructing or scaling extensive data processing systems and pipelines tailored for machine learning workflows.
• Hands-on experience with distributed data processing technologies such as PySpark or Ray, alongside orchestration tools like Airflow.
• Familiarity with containerization and orchestration, notably Docker and Kubernetes.
• Proficiency with cloud storage and compute environments including AWS, GCS, or Azure, understanding the trade-offs related to cost, throughput, and storage configurations.
• Knowledge of video and media processing utilities such as FFmpeg, PyAV, DALI, or OpenCV.
• Experience handling multimodal datasets encompassing video, images, text, and audio.
• Strong research background in post-training methods for large-scale diffusion or flow-based generative models, with practical expertise in distillation optimizing inference efficiency and quality retention.
• Experience with reward modeling and preference-based tuning for generative models, including approaches like RLHF and DPO.
• Deep understanding of the relationship between pretraining and post-training stages affecting distillation and fine-tuning results.
• Proficiency in Python and modern ML frameworks, preferentially PyTorch or JAX.
• Independent research experience demonstrated by leading projects from conception to experimental validation.
• Publication history in premier conferences (e.g., NeurIPS, ICML, ICLR, CVPR, ICCV, ECCV) is advantageous.
• Insight into the practical complexities of developing reliable, scalable, and reproducible ML data workflows.

Perks and Benefits

• Competitive remuneration with generous equity participation.
• Paid personal leave and holidays.
• Comprehensive health insurance coverage.
• International travel insurance for global business trips.
• Monthly personal expenditure allowance of $500 (approximately S$635).
• Provision of all necessary equipment for an efficient home workspace.