IW
Product Analyst
Info Way Solutions LLC
🇺🇸 United States
On-site
2 months ago
- Jira
- Agile
- AI
- Machine Learning
- RAG
- Instructional Design
2 months ago
Location - Phoenix, AZ is preferred but remote can be considered
Duration - 6+ months
Skills -
Required Qualifications
- 3+ years experience drafting detailed Jira user stories, maintaining product backlogs, and defining acceptance criteria based on business requirements in an Agile product team or equivalent environment.
- 2+ years experience in data analysis with the ability to interpret AI performance metrics and translate findings into actionable product recommendations.
- 1+ year experience working with LLM-based applications, including understanding of hallucination patterns, output quality issues, grounding techniques, bias detection, and mitigation strategies.
- 1+ year hands-on experience executing structured AI agent evaluation plans, tracking iterations, maintaining golden datasets, and performing regression testing.
- 1+ year experience supporting the development, evaluation, and continuous improvement of AI-powered products, AI agents, copilots, or generative AI solutions.
- Experience translating AI product requirements into prompts, agent workflows, business rules, and measurable success criteria.
- Experience defining AI product success metrics and measuring improvements in output quality, user satisfaction, task completion, or productivity.
- Experience partnering with engineering, UX, data science, and business stakeholders to prioritize, evaluate, and refine AI product capabilities.
- Experience in HR technology, learning technology, content creation platforms, or knowledge management solutions as a Business Analyst, Product Analyst, Product Owner, or equivalent role.
- Hands-on experience building and executing specialized AI test cases, evaluating LLM outputs, prompt variations, and agentic workflows.
- Hands-on experience with prompt engineering, prompt optimization, and evaluation of AI-generated content quality, accuracy, completeness, and consistency.
- Experience designing evaluation rubrics, benchmark datasets, golden datasets, and quality measurement frameworks for AI-generated outputs.
- Experience conducting side-by-side model comparisons, experimentation, or A/B testing to improve AI product performance.
- Familiarity with Retrieval Augmented Generation (RAG), grounding strategies, hallucination mitigation, and content sourcing concepts.
- Familiarity with automated AI evaluation tools, frameworks, and testing methodologies.
- Experience with human-in-the-loop workflows and designing review, validation, and approval processes for AI-generated outputs.
- Experience working with AI governance, responsible AI practices, bias detection, content safety, and model quality evaluation.
- Experience evaluating content quality against defined business, compliance, instructional design, or organizational standards.
- Strong written and verbal communication skills, with the ability to document evaluation methodologies, findings, recommendations, and trade-offs for both technical and business audiences.
Product Analyst · Info Way Solutions LLC