Freelance Agent Evaluation Engineer

About the Role

Mindrift specializes in connecting experts with project-based AI initiatives for leading technology firms. This role involves part-time, project-oriented work centered on testing, assessing, and enhancing AI systems. This is not a permanent employment but an opportunity for specialists to contribute on defined projects.

Role Responsibilities

• Develop realistic simulated developer settings, such as virtual companies complete with codebases, infrastructure, and contextual elements like tickets, documentation, and discussions, to create believable development histories.
• Design evaluation tasks starting from intermediate points within these environments by crafting effective prompts, defining clear success criteria, and guaranteeing the tasks are manageable for AI agents.
• Develop testing procedures that accurately validate agent solutions by accepting all valid implementations and rejecting incorrect ones, maintaining a balance between strictness and flexibility.
• Continuously improve tasks and tests informed by quality assurance feedback through reviewing agent outputs, analyzing failures, and refining assessments for fairness and robustness.

What This Role Does Not Include

• This is not a data annotation or labeling role.
• It does not involve prompt engineering.
• Writing full code from scratch is not required as the AI agent primarily generates code; your responsibility is to direct and evaluate its output.

Qualifications

• Minimum of five years' experience in software development.
• Expertise in a core technology stack including Python (FastAPI), JavaScript/TypeScript (React), Docker, PostgreSQL, Kafka, and Redis.
• Proven experience in developing functional and integration tests.
• English language proficiency at or above B2 level.

Challenges

Creating tasks that challenge cutting-edge AI coding models requires a profound understanding of their limitations. Evaluating numerous valid solutions to ensure tests accept all correct answers while excluding incorrect ones demands meticulous attention to detail.

Working Process

Interested candidates apply and undergo qualification evaluations. Upon qualifying, they join projects, complete assigned tasks, and receive payment accordingly.

Workload and Schedule

Each task is estimated to require approximately 20 hours, with flexibility regarding when and how the work is conducted. Completion must align with deadlines and meet acceptance criteria.

Compensation

Compensation can reach up to $50 per hour, adjusted based on experience level and productivity. As a freelancer, you set your own working hours and pace.

Additional Information

Please ensure your CV is submitted in English and clearly states your English proficiency level.