Freelance Agent Evaluation Engineer

About Mindrift

Mindrift specializes in connecting skilled professionals with project-based opportunities in artificial intelligence, focusing on the testing, evaluation, and enhancement of AI systems for prominent technology firms. Participation is structured around specific projects rather than permanent employment.

Project Overview: AI Coding Agent Evaluation

This project involves the creation of a comprehensive dataset designed to assess the capabilities of AI coding agents. The goal is to determine how effectively these agents can handle authentic developer tasks.

Key Responsibilities

• Construct realistic developer environments, simulating a virtual company with a complete codebase, necessary infrastructure, and contextual information (including tickets, documentation, and communications) to establish a credible development history.
• Develop challenging tasks and define precise evaluation criteria within these simulated environments. This includes crafting effective prompts and establishing clear definitions of what constitutes a