Job Title:  T&T| Engg| Databricks| Mumbai| Consultant

The Team

Deloitte’s Technology & Transformation practice can help you uncover and unlock the value buried deep inside vast amounts of data. Our global network provides strategic guidance and implementation services to help companies manage data from disparate sources and convert it into accurate, actionable information that can support fact-driven decision-making and generate an insight-driven advantage. Our practice addresses the continuum of opportunities in business intelligence & visualization, data management, performance management and next-generation analytics and technologies, including big data, cloud, cognitive and machine learning.       

Your work profile

Location: Mumbai

Need minimum 3+ years of experience

Education: Btech/BE/MBA

Key Responsibilities

Data Engineering & Pipeline Development

• Design, develop, and maintain batch and streaming data pipelines using Databricks (Spark SQL, PySpark, Scala).

• Build and optimize data ingestion frameworks using Auto Loader, Delta Live Tables (DLT), and Structured Streaming.

• Implement scalable data solutions on cloud platforms (Azure / AWS / GCP) using Databricks Lakehouse architecture.

Databricks Platform & Lakehouse

• Work with Delta Lake for ACID transactions, schema enforcement, and time-travel.

• Develop and optimize ETL/ELT pipelines leveraging Spark best practices.

• Manage data stored across cloud object storage such as ADLS, S3, or GCS.

• Use Unity Catalog for governance, lineage, and access control.

Programming & Data Processing

• Develop data transformation logic using SQL, PySpark, Spark SQL, or Scala.

• Build reusable frameworks, libraries, and components for data processing workflows.

Orchestration & Automation

• Orchestrate workflows using Databricks Workflows, Airflow, or equivalent schedulers.

• Integrate Databricks jobs into CI/CD pipelines using Git, Azure DevOps, GitHub Actions, or Jenkins.

Quality, Monitoring & Performance

• Implement data quality checks using expectations (DLT expectations, QA frameworks).

• Optimize Spark jobs for performance and cost efficiency.

• Troubleshoot data pipeline failures, cluster issues, and performance bottlenecks.

Collaboration & Delivery

• Work with data scientists, analysts, and business teams to understand requirements.

• Support data migration projects, such as on-prem to cloud lakehouse migration.

• Create and maintain documentation, data flow diagrams, and runbooks.

Required Skills

• Strong hands-on experience with Databricks (notebooks, jobs, clusters, Workflows).

• Proficiency in PySpark, Spark SQL, and/or Scala.

• Strong SQL skills.

• Understanding of data modeling, lakehouse architecture, and ETL/ELT pipelines.

• Familiarity with cloud platforms (Azure / AWS / GCP).

• Working knowledge of DevOps, Git, and CI/CD practices.

Preferred Qualifications

• Databricks Data Engineer Associate / Professional Certification.

• Experience with Delta Live Tables, Unity Catalog, and MLflow.

   Exposure real-time streaming using Spark Structured Streaming.

• Experience working in Agile Scrum environments.