Job Title:  Associate Director | Automation | Bengaluru | Engineering | Quality Engineering

The Team

Deloitte’s Technology & Transformation: EAD: Engineering practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning.

Your work profile:

• We are looking for a highly experienced QA Automation Architect to lead the quality engineering vision for AI-driven and Generative AI applications.
• This role goes beyond traditional QA—focusing on LLM validation, AI safety, governance, and scalable automation architecture, including integrations with enterprise platforms such as Adobe Experience Manager (AEM).
• You will play a critical role in ensuring that AI systems are accurate, reliable, safe, and compliant, while driving enterprise-wide standards for AI quality engineering.
• Define and drive the end-to-end QA strategy for AI/LLM-based platforms
• Design and implement scalable automation frameworks using Playwright (TypeScript) and BDD (Python/Behave)
• Establish reusable test architecture, patterns, and best practices across teams
• Drive test strategies across UI, API, and CMS-driven applications (including AEM-based platforms)
• Champion shift-left testing and automation-first culture

AI/LLM Testing & Quality Validation

• Build frameworks to validate:
• Accuracy, relevance, and contextual correctness
• Bias, fairness, and ethical AI behaviour
• Safety, toxicity, and harmful prompt handling
• Fallback and edge-case scenarios
• Design and implement LLM evaluation strategies, including automated scoring and human-in-loop validation
• Create feedback loops and quality dashboards for continuous improvement

Automation & API Engineering

• Architect robust end-to-end and API test automation suites
• Drive high coverage across UI, API, backend, and CMS layers (including AEM components and workflows)
• Implement test data strategies for deterministic and repeatable AI testing

Governance, Compliance & Observability

• Define and enforce Responsible AI testing standards and governance frameworks
• Ensure traceability, auditability, and regulatory compliance
• Establish standards for prompt versioning, dataset control, and experiment tracking
• Validate content workflows and publishing pipelines in systems like AEM
• Leverage logs, traces, and observability tools to validate system behavior

CI/CD & DevOps Integration

• Integrate automation into CI/CD pipelines with quality gates
• Enable test orchestration, parallel execution, and environment consistency (Docker/Kubernetes)
• Define and track quality KPIs and SLAs

Performance & Reliability

• Drive performance testing (p50/p95 latency, throughput, scalability) for AI systems
• Validate performance across APIs, AI services, and CMS platforms (e.g., AEM)
• Identify bottlenecks in model inference and service layers

Leadership & Influence

• Mentor QA engineers and SDETs; elevate team capabilities
• Collaborate with Engineering, Product, Data Science, CMS/AEM teams, and Security teams
• Act as a thought leader in AI quality engineering and automation

Key skills required:

• 13+ years in QA/Automation with architect-level experience
• Strong expertise in Playwright + TypeScript (mandatory)
• Proven hands-on experience in AI/LLM testing and validation
• Deep experience in building scalable automation frameworks
• Strong experience with:
• API testing (Playwright, Postman, Rest Assured)
• BDD (Python + Behave)
• CI/CD tools (Jenkins, GitHub Actions, Azure DevOps, etc.)
• Solid understanding of Agile, DevOps, and modern engineering practices

Good to Have

• Experience with Generative AI systems in production
• Knowledge of Responsible AI / AI Governance frameworks
• Exposure to Cloud (AWS/Azure/GCP) environments
• Experience with Docker, Kubernetes, microservices
• Understanding of prompt engineering and versioning
• Experience working with ML/Data Science teams
• Hands-on experience or exposure to Adobe Experience Manager (AEM) (component testing, workflows, content validation)
• Background in regulated domains (Fintech, Healthcare, SaaS)

Why Join Us

• Work on cutting-edge AI/LLM platforms
• Opportunity to shape enterprise AI quality standards
• High-impact role with architectural ownership and visibility
• Collaborative environment with cross-functional innovation

Ideal Candidate Profile

A seasoned QA leader who:

• Thinks architecturally, not just functionally
• Has a passion for quality in AI systems and digital experience platforms (like AEM)
• Balances technical depth with strategic vision
• Drives innovation while ensuring safety and compliance

Qualifications:

• The candidate should have an educational background of BE / BTech