Job Title:  Enabling Areas - National Office - Manager - Data Engineering

The team

The Office of the COO, Deloitte India is powered by a dynamic team known for its agility and results. Despite its size, the team manages critical projects, consistently delivering high-impact outcomes.

Collaboration is the key to the team’s success, with members working closely together while taking ownership of their responsibilities. They operate in a fast-paced environment, engaging directly with senior leadership (CXOs’) to implement key initiatives and drive strategic outcomes.

Your work profile

Data Lake Architecture & Engineering

• Design, develop, and maintain scalable data lake solutions using Microsoft Azure technologies.
• Build and manage data ingestion, transformation, and processing pipelines to integrate data from multiple enterprise applications and source systems.
• Develop and maintain structured data layers within the data lake, ensuring data is organized, accessible, and optimized for downstream consumption.
• Work extensively with Azure Data Lake Storage (ADLS Gen2), Azure Data Factory (ADF), Azure SQL, and other relevant Azure data services.
• Implement efficient data storage, partitioning, transformation, and processing strategies to improve performance and scalability.
• Support the development and maintenance of enterprise data models catering to financial, operational, and management reporting requirements.
• Evaluate existing data architecture and recommend improvements to enhance performance, reliability, and maintainability.

SQL Development & Data Transformation

• Develop, maintain, and optimize complex SQL queries, stored procedures, views, functions, and data transformation logic.
• Design and implement robust ETL/ELT processes for large-volume enterprise datasets.
• Perform SQL query tuning, performance optimization, and database troubleshooting to ensure efficient data processing.
• Develop reusable data transformation components and standardized data engineering frameworks.
• Translate complex business requirements into technical data structures, transformation rules, and data models.
• Ensure accurate handling of historical data, incremental loads, data reconciliation, and changes in source data structures.
• Troubleshoot complex data issues involving inconsistent source data, transformation logic, and downstream reporting discrepancies.

Data Integration & Pipeline Management

• Design and implement automated data pipelines to support integration between enterprise systems, data lake environments, and downstream applications.
• Work with structured and semi-structured data from databases, APIs, flat files, and enterprise applications.
• Develop and manage scheduled and event-driven data ingestion processes.
• Implement appropriate error handling, logging, monitoring, and alerting mechanisms across data pipelines.
• Ensure timely availability of data for business-critical reporting and analytics platforms.
• Investigate and resolve data pipeline failures, processing delays, and integration issues.
• Coordinate with source-system teams to manage changes in data structures, interfaces, and integration requirements.

Data Quality, Validation & Governance

• Establish and implement data quality frameworks covering completeness, accuracy, consistency, and integrity of information.
• Develop automated validation and reconciliation mechanisms between source systems, data lake layers, and downstream reporting applications.
• Identify root causes of data discrepancies and implement sustainable technical solutions.
• Maintain appropriate documentation of data pipelines, transformation logic, source-to-target mappings, and data lineage.
• Implement data access controls and security practices in alignment with organizational information security policies.
• Support data governance initiatives, including standardization of data definitions and master data across systems.
• Ensure engineering solutions comply with applicable data privacy, security, and retention requirements.

Enterprise Analytics & Reporting Enablement

• Collaborate with business and technology teams to understand enterprise reporting and analytics requirements.
• Develop and maintain data structures that support financial planning, business performance reporting, and operational dashboards.
• Support integration of data lake information with downstream applications such as Power BI, Anaplan, and other enterprise analytics platforms.
• Work with large datasets covering financial and operational metrics such as revenue, billing, collections, workforce, utilization, costs, and profitability.
• Optimize datasets and data transformation processes to improve reporting performance and data refresh timelines.
• Support the implementation of new reporting dimensions, business hierarchies, and changes in organizational structures.
• Provide technical support for data reconciliation and issue resolution during implementation, testing, and production operations.

Technical Delivery & Team Management

• Independently lead the technical design, development, testing, and implementation of data engineering solutions.
• Manage multiple development initiatives and ensure delivery within agreed timelines and quality standards.
• Review SQL code, pipeline designs, data models, and technical solutions developed by team members.
• Provide technical guidance and mentoring to data engineers and other members of the development team.
• Establish and enforce coding standards, documentation practices, version control, and deployment processes.
• Collaborate with IT infrastructure, application development, data architecture, and business teams to resolve technical dependencies.
• Proactively identify technical risks, performance bottlenecks, and areas requiring architectural improvements.
• Ensure appropriate testing, release management, and production support processes are followed.

Continuous Improvement & Automation

• Identify opportunities to automate manual data processing, reconciliation, and operational activities.
• Develop reusable frameworks and utilities to improve engineering productivity and reduce development effort.
• Implement improvements to reduce data processing time, infrastructure costs, and recurring operational issues.
• Evaluate emerging Azure technologies and data engineering capabilities for potential adoption.
• Drive continuous improvement in data pipeline reliability, scalability, observability, and maintainability.

Key skills required

• 8–12 years of overall experience in data engineering, database development, data integration, or related technology roles.
• Minimum 4–5 years of hands-on experience working with Microsoft Azure data engineering technologies.
• Strong experience in SQL development, query optimization, and enterprise database management.
• Proven experience designing and implementing end-to-end ETL/ELT pipelines and data lake solutions.
• Experience working with large-scale enterprise datasets and integrating information from multiple source systems.
• Prior experience managing technical deliverables and providing guidance to small engineering teams.
• Experience in financial services, professional services, or enterprise finance and operations reporting would be advantageous.
• Previous experience in a Big 4 organization or a large multinational enterprise would be an added advantage.

Mandatory Technical Skills

Microsoft Azure Data Engineering

• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Azure Data Factory (ADF)
• Azure SQL Database / SQL Server
• Azure data integration and pipeline orchestration
• Data lake architecture and data processing frameworks
• Pipeline monitoring, troubleshooting, and performance optimization

SQL & Database Development

• Advanced SQL and T-SQL
• Complex queries, joins, CTEs, and window functions
• Stored procedures, functions, and views
• Query performance tuning and execution plan analysis
• Database design, indexing, and optimization
• Data validation and reconciliation

Data Engineering & Integration

• ETL/ELT architecture and development
• Data ingestion and transformation
• Data warehousing and dimensional modelling
• Incremental loading and change data capture concepts
• Data quality management
• Source-to-target mapping and data lineage
• Integration with APIs, databases, and flat files
• Handling large datasets and complex transformation logic

Good-to-Have Technical Skills

• Azure Databricks
• Python and PySpark
• Apache Spark
• Microsoft Fabric
• Azure Synapse Analytics
• Delta Lake and Medallion Architecture
• Power BI datasets and semantic models
• Azure DevOps, Git, and CI/CD pipelines
• REST APIs and application integration
• Experience integrating data with SAP, ERP, or enterprise finance systems
• Knowledge of cloud security, access management, and infrastructure cost optimization