AI QA Analyst

5+ years of experience

Ukraine, Europe

Fulltime, remote

Our client is a global health marketing, communications, and transformation platform specializing in Marketing, Comms, Transformation, Access, and Media, partnering with leading healthcare and life sciences organizations to drive growth through data-driven insights and a human-centered approach, supporting products across the full lifecycle from early development to commercialization.

We are currently looking for an AI QA Analyst.

The AI QA Analyst is responsible for validating the quality, reliability, and trustworthiness of outputs generated by an agentic AI platform across both ad-hoc and structured workflows. This role goes beyond traditional QA by focusing on AI-generated results, multi-agent workflows, execution traceability, and lifecycle integrity in complex and evolving environments.

You will ensure that AI systems produce accurate, explainable, compliant, and reproducible outcomes while adhering to key platform principles such as attribution, auditability, and lifecycle consistency.

The role involves close collaboration with Product and Business Analysis teams to integrate quality standards directly into the execution model.

Key Responsibilities

Agentic AI Output Validation

Design and execute QA strategies to validate AI-generated artifacts, recommendations, and decisions produced by autonomous and semi-autonomous agents.
Evaluate outputs for accuracy, relevance, completeness, hallucination risk, and alignment with provided context modules and knowledge sources.
Validate that agent outputs are attributable, traceable, and reproducible within our execution and memory model.

Multi-Agent Workflow & Lifecycle Testing

Test[MB5] end-to-end agent workflows across the full lifecycle:
Intent Objective Plan Execution Artifacts Memory Evolution[ST6] [MB7]
Validate correct behavior across both ad-hoc requests and structured deliverables, including seamless promotion between the two without loss of context or artifacts.
Ensure lifecycle state transitions, dependencies, and blocking logic behave deterministically and transparently.

Platform & Execution Quality Assurance

Develop test cases for plan creation, phase execution, step execution, retries, halts, and re-runs across human and AI agents.
Validate execution logs, artifacts, decisions, and audit events for completeness and consistency.
Test failure modes, edge cases, and recovery scenarios in agent execution paths.

Compliance, Governance & Explainability

Partner with Legal, Compliance, and Data teams to ensure QA coverage for regulated workflows, including healthcare-grade auditability and review readiness.
Validate that AI outputs meet governance requirements for explainability, source grounding, and decision rationale.
Ensure platform behavior aligns with responsible AI standards, including bias awareness and data usage constraints.

Continuous Quality Improvement

Contribute to the design of automated testing frameworks for agentic systems, including regression testing of AI outputs over time.
Define and track quality metrics related to AI performance, output stability, and user trust signals.
Provide structured feedback loops to Product and Engineering teams to improve agent behavior, prompts, and orchestration logic.

Required Qualifications

Professional Experience

5–8+ years of experience in Quality Assurance, Test Engineering, or Product Quality roles within complex software platforms.
Hands-on experience testing AI-enabled, data-driven, or workflow-oriented systems.
Proven ability to validate non-deterministic outputs where correctness is probabilistic rather than binary.

Skills & Competencies

Strong understanding of QA methodologies across functional, integration, regression, and UAT testing.
Experience designing test cases for LLMs, generative AI systems, or intelligent automation.
Ability to reason about agent behavior, execution traces, and output quality across multi-step workflows.
Familiarity with Agile delivery environments and tools such as JIRA, Confluence, or similar.
Exceptional attention to detail and comfort working with complex system states and abstractions.

Preferred Qualifications

Experience with agentic AI frameworks, orchestration systems, or multi-agent workflows.
Background in regulated industries such as healthcare, life sciences, or financial services.
Exposure to LLM evaluation techniques, prompt testing, or AI output benchmarking.
Familiarity with audit logging, compliance validation, or responsible AI practices.

Success Metrics

Reduction in AI-related defects, regressions, and post-release quality issues.
Demonstrated improvement in consistency, trustworthiness, and explainability of agent outputs.
High confidence from Product, Compliance, and Engineering stakeholders in platform readiness.
Successful validation of complex agent workflows across structured and ad-hoc use cases.

We offer