Job Title:  Senior Consultant | Databricks + Pyspark | Mumbai | Engineering | Data Modernization & Migration

Job Title: Databricks - Sr. Consultant

Role Overview: As a Senior Consultant, you will lead the design and optimization of high-performance data processing frameworks on Azure Databricks. You will be responsible for advanced Spark tuning, implementing robust data governance through Unity Catalog, and driving the transition toward a modern Medallion architecture. 

Experience: 

• 5–7 years of experience in data engineering, with deep expertise in Azure Databricks and Apache Spark. 
• Advanced experience with Delta Lake and Medallion Architecture (Bronze/Silver/Gold). 
• Proven track record in Spark Performance Tuning (handling data skew, shuffle tuning, etc.). 
• Strong experience with Unity Catalog for data governance and security. 

Key Responsibilities: Technical Skills: 

• Design and implement scalable Medallion Architectures using Delta Live Tables (DLT) and Databricks. 
• Lead the optimization of complex Spark workloads to improve execution speed and reduce cloud compute costs. 
• Implement enterprise-grade data governance and access control using Unity Catalog. 
• Architect automated CI/CD pipelines for Databricks using Databricks Asset Bundles (DABs) or Terraform. 
• Design sophisticated data integration patterns between Databricks, Azure Synapse, and various RDBMS. 
• Act as a technical mentor to junior engineers, conducting code reviews and enforcing best practices. 

Soft Skills: 

• Strong leadership and ability to drive technical delivery. 
• Advanced problem-solving skills for complex architectural challenges. 
• Excellent communication skills for interacting with stakeholders and technical leads. 

Preferred Qualifications: 

• Bachelor’s or Master’s degree in a technical field. 
• Databricks Certified Data Engineer Professional is highly desirable. 
• Experience with DataOps and Infrastructure as Code (Terraform). 

Job Details: 

• Location: Mumbai (Hybrid)