Machine Learning Engineer, Evals

Nous Research is an open-source AI research lab known for its Hermes models, and also runs Psyche, a decentralized AI-training network coordinated on Solana, backed by a roughly $50M Series A led by Paradigm. This role focuses on agent capability evaluations, benchmark design, LLM-as-judge systems, failure analysis, and evaluation infrastructure.

The engineer will run the full evaluation pipeline end-to-end and own the recurring evaluation workflow that informs model development.

• Run the full evaluation pipeline end-to-end
• Build a judge calibration protocol for LLM-as-judge systems
• Extend benchmarks such as GAIA, tau-Bench, and SWE-bench
• Run failure analysis on model outputs
• Own the recurring evaluation workflow

• 3+ years of software engineering, machine learning, or data science experience with evaluation work
• Experience with LLM evaluation frameworks
• Hands-on experience working with LLMs
• Solid Python skills
• Understanding of evaluation statistics

A remote evaluation-focused ML role at a well-funded open-source AI lab that also runs a decentralized, Solana-coordinated training network.