Research Engineer, Benchmarks

Company Overview

HUD develops foundational infrastructure for generating reinforcement learning training data and evaluations for advanced AI agents. Their platform supports frontier research labs, large corporations, and startups, and the company has secured $16 million in funding from prominent venture capitalists. HUD participated in Y Combinator's Winter 2025 batch.

Position Summary

This role involves engineering high-quality benchmarks used to assess advanced AI agents on specific domain tasks. The ideal candidate will focus on building technically sound, practical, and widely respected benchmarks for frontier research labs.

Key Responsibilities

• Take full ownership of the design, development, and maintenance of internal agent benchmarks at HUD.
• Collaborate with experts in various fields to define relevant tasks and craft benchmarks that reflect actual workflow scenarios.
• Create infrastructure facilitating consistent execution of models and agents against benchmark challenges.
• Design and implement metrics and analyses to measure benchmark difficulty, consistency, and common failure points.
• Confirm that benchmark outcomes align with real-world evaluations, customer needs, and expectations of research laboratories.
• Produce clear, accessible documentation and detailed benchmark reports for technical audiences.

Candidate Profile

• Proficient with Python programming, Docker containerization, and working within Linux systems.
• Previous publications or authoritative technical writings on benchmarks, model shortcomings, or related subjects, with references requested in the application.
• Deep understanding of the essential qualities of effective benchmarks, including realism, reliability, and utility.
• Experience designing evaluation environments and performance assessments.
• Strong curiosity enabling thorough comprehension of workflows across diverse domains.

Desirable Attributes

• Excellent attention to detail with an ability to identify subtle discrepancies and edge cases.
• A rational, first-principles approach to task creation, scoring methods, and diagnosing failures.
• Comfort operating in ambiguous, less-structured problem spaces.
• Experience in early-stage startups, with self-motivation to work independently and adapt quickly.
• Effective communication skills suited for collaborating across multiple time zones remotely.

Additional Information

• Team currently has about 15 members, largely full-time onsite, with some remote contributors.
• The team comprises international competition medalists, serial AI entrepreneurs, and published researchers in leading conferences such as ICLR and NeurIPS.
• Company is well-funded with eight-figure backing and exhibits strong revenue growth, scaling profitably to meet growing demand.

Work Location and Process

• Full-time employment status.
• Primary locations are company offices in the San Francisco Bay Area or Singapore, with onsite presence required.
• Relocation and visa sponsorship available for qualified candidates moving to the US or Singapore.
• Recruitment involves two technical interviews followed by a 2-3 day work trial, with applications reviewed on a rolling basis.

Benefits and Perks

• Competitive salary package.
• Comprehensive medical, dental, and vision insurance covered entirely by the company for US employees through Blue Shield of California.
• Free lunch and dinner when working onsite.
• Company-wide holiday closure during Christmas Eve through New Year's Day, in addition to paid time off and holidays.
• Other perks for US-based staff include Equinox gym membership, 401(k) plans, and commuter benefits.
• Unlimited access to tokens for advanced AI tools like ChatGPT, Claude Code, and Cursor, with no practical usage limits observed.

Note

Due to a high volume of applications, responses might not be guaranteed, but candidates may reach out by email if their application is overlooked.