Software Engineering, Data Science & Systems Design Experts (AI Model Evaluation)

Role Overview

As aTechnical AI Evaluator, you will assess and improve AI-generated responses to coding and software engineering tasks. Your work will directly impact how accurately and effectively AI systems solve problems, write code, and communicate technical concepts.

This is a hands-on, intellectually rigorous role suited for professionals with strong coding, analytical, and problem-solving skills.

Key Responsibilities

• Evaluate AI-generated responses foraccuracy, reasoning, clarity, and completeness

• Execute and test code to validate correctness and performance

• Identify bugs, inefficiencies, and logical flaws in model outputs

• Annotate responses with detailed feedback and improvement suggestions

• Assess code quality, readability, and adherence to best practices

• Fact-check technical claims using reliable and authoritative sources

• Apply structured evaluation frameworks and guidelines consistently

What We’re Looking For

• Bachelor’s, Master’s, or PhD in Computer Science or a related field

• 3+ years of professional experience in software engineering, data science, or related roles

• Strong proficiency inat least two programming languages (e.g., Python, JavaScript, C++, Java, SQL, Go, Rust, etc.)

• Ability to independently solvemedium to hard algorithmic problems

• Experience working with or alongsideLLMs in coding workflows

• Strong attention to detail and ability to evaluate complex technical reasoning

Nice to Have

• Experience withAI model evaluation, RLHF, or data annotation

• Background incompetitive programming

• Contributions toopen-source projects (merged pull requests preferred)

• Experience reviewing production-level code

• Ability to explain complex technical concepts clearly to non-experts

What Success Looks Like

• You consistently identify subtle bugs, flawed logic, and edge cases

• Your feedback improves the accuracy and clarity of AI-generated code

• You produce structured, reproducible evaluation outputs

• AI systems become more reliable for real-world engineering tasks

Compensation & Terms

• Independent contractor engagement

• Fully remote with flexible working hours (open to US and non-US candidates)

• Full-time or part-time contract options available

• Weekly payments based on services rendered

Equal Opportunity

We consider all qualified applicants without regard to legally protected characteristics and provide reasonable accommodations upon request.