Job Title:  Technology and Transformation - Engineering - Senior Consultant - AWS Data Engineer - Bengaluru

The Team

Deloitte’s Technology & Transformation practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning. 

Your work profile

We are looking for Data Engineering professionals with strong expertise in PySpark, AWS Cloud, SQL, and ETL/Data Pipeline development. The role involves developing, optimizing, and maintaining scalable data pipelines and cloud-based data solutions across enterprise data platforms. Candidates should have hands-on experience in Data Engineering, AWS Data Analytics services, ETL frameworks, and big data technologies along with good understanding of ETL/Data Testing, data validation, and reconciliation processes to support end-to-end data quality assurance.

Key Skills & Competencies

• Data Engineering: PySpark, SparkSQL, DataFrames, RDD, Data Pipeline Development, Data Processing
• Cloud Technologies: AWS Glue, S3, Lambda, Lake Formation, Athena, Event Bridge
• SQL & Programming: Advanced SQL, PL/SQL, Python Programming, Shell Scripting
• ETL Tools: Ab Initio, Informatica, Talend, DataStage, dbt (Anyone)
• Big Data Technologies: Hadoop Ecosystem, Spark Performance Optimization
• Data Platforms: Databricks, Snowflake, Redshift, BigQuery, Synapse
• CI/CD & Tools: GitLab, Jenkins, Azure DevOps, GitHub Actions, Scheduling Tools
• Operating Systems: UNIX/Linux Environment & Commands
• Data Formats: Parquet, Avro, JSON, CSV, XML
• Testing Knowledge: ETL/Data Validation, Data Reconciliation, Source-to-Target Validation, CDC Validation, Automation Testing Basics
• Additional Skills: Agile/Scrum Methodology, Cloud Data Validation, Performance Tuning
• Develop, enhance, and maintain scalable ETL/ELT pipelines and cloud-based data processing applications.
• Design and implement data engineering solutions using PySpark, SparkSQL, Python, and AWS cloud services.
• Work on large-scale data processing using DataFrames, RDDs, and distributed computing frameworks.
• Perform PySpark performance optimization and tuning for enterprise data workloads.
• Develop and maintain applications on AWS cloud platforms using services such as Glue, S3, Lambda, Athena, Lake Formation, and Event Bridge.
• Write advanced SQL and PL/SQL queries, stored procedures, and data transformation logic.
• Work with structured and semi-structured data formats including Parquet, Avro, JSON, CSV, and XML.
• Utilize UNIX/Linux commands and shell scripting for deployment, monitoring, and operational activities.
• Collaborate with cross-functional teams including Data Engineering, QA, DevOps, and Business teams for successful project delivery.
• Participate in CI/CD implementation and code deployment activities using GitLab, Jenkins, or related tools.
• Support ETL/Data Testing activities including data validation, reconciliation, source-to-target validation, and data quality checks.
• Analyze production issues, identify root causes, and provide timely resolutions for data pipeline failures.
• Participate in Agile ceremonies, sprint planning, estimation, documentation, and status reporting activities.
• Contribute to continuous improvement initiatives, automation enhancements, and knowledge-sharing sessions.

Preferred Qualifications

• Education - B. tech / BE in Computer Science or Information Technology.

Success Metrics

• Scalable and reliable ETL/ELT pipelines delivered on time and aligned with project requirements.
• Improved data processing performance through PySpark, SparkSQL, AWS and Databricks workload optimization.
• High-quality data outputs supported by validation, reconciliation, source-to-target and CDC validation checks.
• Reduced production incidents through timely root cause analysis and pipeline issue resolution.
• Successful CI/CD adoption, automated deployments and improved operational efficiency for data pipelines.
• Effective collaboration across Data Engineering, QA, DevOps and Business teams in Agile delivery cycles.

Location and way of working:

• Base location: Bengaluru / Pune
• This profile involves travelling to client locations