Job Title:  Consultant | Azure Databricks | Bengaluru | Engineering | Data Modernization & Migration

The Team

Deloitte’s Technology & Transformation: EAD: Engineering practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning.

Your work profile:

As a Consultant/Senior Consultant in our Technology & Transformation Team you’ll build and nurture positive working relationships with teams and clients with the intention to exceed client expectations: -

Key skills required:

Job Summary

We are seeking a skilled Azure Databricks Data Engineer with 3–6 years of experience in designing and developing cloud-based data engineering solutions. The ideal candidate should have hands-on experience with Azure Databricks, PySpark, Azure Data Factory, and Azure Data Lake Storage. The role involves building scalable ETL/ELT pipelines, optimizing data processing workloads, and supporting enterprise analytics initiatives.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines using Azure Databricks.
• Develop data transformation workflows using PySpark, Spark SQL, and Python.
• Build and orchestrate data pipelines using Azure Data Factory (ADF).
• Ingest data from various sources including databases, APIs, files, and cloud storage.
• Develop and optimize Delta Lake tables following Medallion Architecture (Bronze, Silver, Gold).
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Work with structured, semi-structured, and unstructured datasets.
• Implement data quality checks, logging, and monitoring for data pipelines.
• Collaborate with data architects, analysts, and business stakeholders to understand business requirements.
• Troubleshoot production issues and provide timely resolutions.
• Participate in code reviews and follow coding standards and best practices.
• Support deployment activities using Azure DevOps or Git-based CI/CD pipelines.
• Create and maintain technical documentation for data solutions.

Required Technical Skills

• 3–6 years of experience in Data Engineering.
• Hands-on experience with Azure Databricks.
• Strong expertise in PySpark, Apache Spark, and Spark SQL.
• Proficiency in Python and SQL.
• Experience with Azure Data Factory (ADF).
• Experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• Good understanding of Delta Lake concepts.
• Knowledge of ETL/ELT development and data integration.
• Experience in Spark performance tuning and optimization.
• Familiarity with Git and Azure DevOps.
• Understanding of data modeling concepts and relational databases.
• Experience working in Agile/Scrum environments.

Preferred Skills

• Azure Synapse Analytics
• Databricks Workflows
• Unity Catalog
• Azure Key Vault
• Azure Event Hubs or Kafka
• REST API integration
• Power BI integration
• Basic knowledge of Terraform or Infrastructure as Code (IaC)
• Experience with streaming data processing

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• Strong problem-solving and analytical skills.
• Good communication and collaboration skills.
• Ability to work independently and as part of a team.

Preferred Certifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Microsoft Azure Fundamentals (AZ-900)
• Databricks Certified Data Engineer Associate

Key Technologies

• Azure Databricks
• PySpark
• Apache Spark
• Spark SQL
• Python
• SQL
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Delta Lake
• Azure DevOps
• Git
• Databricks Workflows
• Unity Catalog
• Azure Synapse Analytics
• Power BI

Nice to Have

• Experience with CI/CD implementation.
• Exposure to data warehousing concepts.
• Understanding of cloud security and data governance.
• Experience with real-time or streaming data processing.
• Knowledge of Microsoft Fabric is an added advantage.

Preferred Qualifications

• Informatica IDMC or MDM certifications
• Experience in cloud migration programs or hybrid architectures
• Exposure to real-time streaming or event-driven architectures

Experience and Qualification:

• B.E./ B.Tech. or M.C.A. in Computer Science from a reputed University
• 4-7  years of relevant industry experience