Machine Learning Engineer, Evals
Job Description
Nous Research is an open-source AI research lab known for its Hermes models, and also runs Psyche, a decentralized AI-training network coordinated on Solana, backed by a roughly $50M Series A led by Paradigm. This role focuses on agent capability evaluations, benchmark design, LLM-as-judge systems, failure analysis, and evaluation infrastructure.
The engineer will run the full evaluation pipeline end-to-end and own the recurring evaluation workflow that informs model development.
Responsibilities
- Run the full evaluation pipeline end-to-end
- Build a judge calibration protocol for LLM-as-judge systems
- Extend benchmarks such as GAIA, tau-Bench, and SWE-bench
- Run failure analysis on model outputs
- Own the recurring evaluation workflow
What we are looking for
- 3+ years of software engineering, machine learning, or data science experience with evaluation work
- Experience with LLM evaluation frameworks
- Hands-on experience working with LLMs
- Solid Python skills
- Understanding of evaluation statistics
Additional Information
A remote evaluation-focused ML role at a well-funded open-source AI lab that also runs a decentralized, Solana-coordinated training network.
Perks and Benefits
- Remote role across Americas time zones
- Work on open-source models used broadly across the AI community
- Contribute to Psyche, a decentralized training network backed by Paradigm
Keywords: LLM Evaluation, Python, Benchmark Design, LLM-as-judge, RLHF/RLVR, Statistics
Nous Research
📍 Distributed (Americas time zones)Visit Company WebsiteNous Research is an open-source AI research lab known for its Hermes models, also running Psyche, a decentralized AI-training network coordinated on Solana. Backed by a ~$50M Series A led by Paradigm at a near-$1B valuation.
Apply To This Job<<>>
Support us by letting the company know you found them on our website.
Magica
Every AI You Need.Just $15/month.
Go To the Offer