Job Title:  T&T | EAD | Analyst | Automation Tester | Mumbai | Engineering

The Team

Deloitte’s Technology & Transformation practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning.     

Your work profile

As an Analyst in AI-driven Quality Engineering, you will play a critical role in validating the reliability, accuracy, and ethical behavior of Large Language Model (LLM)-based applications. You will work at the intersection of AI, testing, and business workflows to ensure that AI systems are trustworthy, performant, and production-ready.

Key Responsibilities

1. LLM Functional  Testing

• Design and execute test scenarios for LLM-based applications (chatbots, copilots, generative AI systems)
• Validate response correctness, relevance, and contextual understanding
• Perform prompt-based testing to assess model behaviour under varied inputs
  
  

2. Prompt Engineering & Validation

• Create, refine, and optimize prompts to test different use cases and edge conditions
• Identify prompt vulnerabilities such as hallucinations, bias, and unsafe outputs
• Evaluate consistency and determinism across multiple runs
  
  

3. AI Quality Dimensions Validation

• Validate outputs across key AI quality pillars:
• Accuracy & relevance
• Coherence & fluency
• Bias & fairness
• Safety & compliance
• Perform adversarial testing to uncover unexpected or harmful responses
  
  

4. Test Data & Scenario Design

• Develop structured and unstructured datasets for AI validation
• Simulate real-world user interactions across domains (banking, telecom, healthcare, etc.)
• Design boundary, negative, and stress scenarios for LLMs
  
  

5. Automation & Tooling

• Leverage automation tools/frameworks for AI testing (Python, APIs, prompt libraries)
• Use AI testing platforms (e.g., Botium, LangChain testing tools, Azure AI Studio)
• Integrate tests into CI/CD pipelines for continuous validation
  
  

6. Performance & Scalability Testing

• Validate response latency, throughput, and system behaviour under load
• Assess performance of LLM APIs and integrated AI workflows
  
  

7. Governance, Risk & Compliance

• Ensure adherence to Responsible AI principles
• Validate compliance with data privacy, security, and regulatory standards
• Document test evidence for audit and governance reviews

8. Defect Analysis & Reporting

• Identify, log, and track AI-specific defects (hallucination, drift, inconsistency)
• Provide actionable insights and recommendations for model tuning
  
  

Required Skills & Qualifications

Core Skills

• Understanding of software testing fundamentals (STLC, defect lifecycle)
• Basic knowledge of AI/ML and LLM concepts (prompting, tokens, embeddings)
• Familiarity with API testing (Postman / REST clients)
• Exposure to automation (Python / Selenium / Playwright – basic level)

Good to Have

• Experience with AI/LLM tools (OpenAI, Azure OpenAI, Hugging Face)
• Knowledge of prompt engineering techniques
• Understanding of NLP concepts
• Exposure to tools like Botium, LangChain, or similar AI testing frameworks
  
  

Behavioral Competencies

• Analytical thinking with strong attention to detail
• Curiosity to explore AI behavior and edge cases
• Strong communication skills for articulating AI risks and findings
• Ability to work in agile and fast-evolving environments

What You Will Learn / Exposure

• Enterprise-scale AI testing frameworks and governance models
• Hands-on experience with cutting-edge LLM technologies
• Exposure to AI ethics, safety, and compliance standards
• Opportunity to work on multi-channel AI systems (voice, chat, email, agent assist)

Education 
Bachelor’s degree in computer science, Information Technology, or related field. 

Location and Way of Working 
         Base location: Mumbai