Job Title:  Technology and Transformation - Engineering - Manager - AWS Data Engineer - Bengaluru

The Team

Deloitte’s Technology & Transformation practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning. 

Your work profile

The Lead Engineer is responsible for designing, developing, testing, optimizing and maintaining scalable data engineering solutions using PySpark, SQL, ETL, AWS Glue, AWS cloud services, Unix and data warehousing concepts.This role requires hands-on engineering capability across AWS and Databricks-based data platforms, Spark-based distributed processing, AWS data analytics services, reusable data components, modern table formats and CI/CD practices for data pipelines.

Key Skills & Competencies

• Design, develop, test and maintain data engineering applications and pipelines on AWS Cloud.
• Build scalable ETL and ELT solutions using PySpark, SparkSQL, AWS Glue and SQL-based transformation patterns.
• Design and implement efficient data transformation and storage solutions using open table formats such as Delta, Iceberg and Hudi.
• Develop reusable data components and frameworks using AWS tools, technologies and Databricks.
• Apply Unix scripting and data warehousing concepts to support operational data processing and production workloads.

• Work hands-on with PySpark, Spark DataFrames, RDDs and SparkSQL for distributed data processing.
• Implement PySpark performance optimization techniques for scalable, reliable and efficient data workloads.
• Use AWS Data Analytics technology stack including AWS Glue, Amazon S3, AWS Lambda, AWS Lake Formation, Amazon Athena and Amazon EventBridge.
• Build and optimize data solutions across AWS, Databricks, Hadoop and data warehouse ecosystems.
• Write advanced SQL and PL/SQL programs for data extraction, transformation, validation and analytical processing.

• Use dbt with AWS and Databricks-based environments for ELT pipeline development and transformation management.
• Implement DevOps and CI/CD practices for data pipelines using GitLab Actions or similar tools.
• Support version control, automated deployment and Infrastructure-as-Code practices for AWS and Databricks data engineering solutions.
• Collaborate with engineering, DevOps and platform teams to improve reliability, maintainability and deployment efficiency.
• Support production readiness through testing, monitoring, issue resolution and continuous optimization.

• Strong hands-on experience in PySpark with good understanding of DataFrames, RDDs and SparkSQL.
• Hands-on experience in PySpark performance optimization techniques and distributed data processing patterns.
• Strong knowledge of SQL, advanced SQL and PL/SQL programming for data engineering and analytics workloads.
• Good understanding of AWS cloud, Hadoop and data warehousing concepts.
• Hands-on experience developing, testing and maintaining applications on AWS Cloud.
• Strong experience with AWS Data Analytics stack including AWS Glue, Amazon S3, AWS Lambda, AWS Lake Formation, Amazon Athena and Amazon EventBridge.
• Experience designing scalable data transformation and storage solutions using Delta, Iceberg and Hudi open table formats.
• Experience using dbt with AWS and Databricks-based environments for ELT pipeline development.
• Hands-on experience building reusable data components using AWS tools, technologies and Databricks.
• Experience using GitLab Actions or similar CI/CD tools for version control, automated deployment and Infrastructure-as-Code for data pipelines.

• Core skills: PySpark, SQL, ETL, AWS Glue, Unix and data warehousing concepts.
• AWS stack: AWS Glue, Amazon S3, AWS Lambda, AWS Lake Formation, Amazon Athena and Amazon EventBridge.
• Databricks skills: Databricks notebooks, jobs/workflows, clusters, Spark processing and lakehouse-based data engineering.
• Big data and Spark: Spark DataFrames, RDDs, SparkSQL, Spark performance tuning and distributed processing.
• Modern data platforms: AWS analytics services, Databricks, Hadoop ecosystem and data warehouse architecture.
• Open table formats: Delta, Iceberg and Hudi for scalable data transformation and storage solutions.
• ELT development using dbt with AWS and Databricks-based environments.
• DevOps and CI/CD for data including GitLab Actions, version control, automated deployments and Infrastructure-as-Code.
• Exposure to data governance or lineage tools such as Immuta and Alation is an added advantage.
• Experience with orchestration tools such as Apache Airflow or AWS-native scheduling services is an added advantage.
• Knowledge of Ab Initio ETL tool is a plus.
• Proficiency in English is required.

Preferred Qualifications

• Education - B. tech / BE in Computer Science or Information Technology.

Success Metrics

• Reliable and scalable AWS and Databricks-based data pipelines delivered on time and aligned with business requirements.
• Optimized PySpark, SparkSQL and SQL workloads with improved performance, cost efficiency and reliability.
• Reusable data engineering components and frameworks adopted across AWS and Databricks implementations.
• Successful implementation of dbt-based ELT pipelines and modern open table format solutions.
• Improved deployment quality through GitLab Actions or similar CI/CD, version control and Infrastructure-as-Code practices.
• Reduced production issues through strong testing, maintainability, monitoring and continuous optimization.

Location and way of working:

• Base location: Bengaluru / Pune
• This profile involves travelling to client locations