LLM Red Team Specialist — Failure Modes & Edge Cases

DESCRIPTION

A leading AI lab is building the next generation of agentic evaluation benchmarks for frontier models and needs specialists who can find where those models break. Working in a red-teaming setup, you will design and probe complex, multi-step tasks to expose vulnerabilities, edge cases, and failure modes in frontier AI systems — the places where a model looks competent but is quietly wrong. Each task represents one to two days of continuous, focused effort and spans multiple technical skills: coding, experimentation, and careful analysis. You will work in a tight feedback loop with the lab's researchers, turning the failure modes you find into stronger benchmark tasks. This is a full-time W-2 employment position with Cincinnatus LLC, with the opportunity to be placed at a leading AI lab as part of their extended workforce. This role is fully remote within the United States, at approximately 35 hours per week. 2. Key Responsibilities Probe models: Explore how frontier AI models behave on coding, ML, and analysis tasks, and find the spots where they quietly get things wrong. Design challenges: Turn the weaknesses you find into well-crafted tasks that are hard for models but fair to grade. Document findings: Write up what you discover clearly, with evidence and steps others can reproduce. Strengthen tasks: Team up with task authors to close loopholes, shortcuts, and grading gaps. Work as a team: Share insights with researchers and fellow experts so the benchmark keeps getting better. 3. Core

REQUIREMENTS

• ▸MSc or PhD in a STEM field, or equivalent practical experience in a research-heavy domain requiring data analysis and coding.
• ▸1+ years of experience in a research, research-engineering, security, or AI-evaluation role.
• ▸Demonstrated ability to identify vulnerabilities, edge cases, or failure modes in LLMs or ML systems — through red teaming, adversarial testing, security research, or rigorous model evaluation.
• ▸Working proficiency in Python and Git, with the ability to script your own probes and analyses.
• ▸Strong familiarity with LLM capabilities, limitations, and evaluation techniques.
• ▸Past experience in AI training, model evaluation, or benchmark/task authoring is preferred.
• ▸Ability to engage reliably for approximately 35 hours per week.
• ▸About Cincinnatus LLC

More Mercor contracts