Job Title:  Consultant/Senior Consultant | Databricks |Coimbatore | Engineering | Data Modernization & Migration

Job Title: Senior Databricks Engineer

Experience: 6-9 Years

Location: Mumbai

Employment Type: Full-Time

Job Summary

We are seeking an experienced Databricks Developer with 6-9 years of expertise in designing, building, and supporting modern Data Lakehouse solutions. The ideal candidate will have strong hands-on experience in Databricks, PySpark, Delta Lake, cloud platforms, and large-scale data processing. The role involves building scalable data pipelines, optimizing distributed data workloads, and enabling advanced analytics and AI-driven initiatives.

Key Responsibilities

• Design, develop, and maintain scalable data engineering solutions using Databricks.

• Build and optimize batch and real-time data processing pipelines.

• Develop ETL/ELT frameworks using PySpark and Spark SQL.

• Implement Delta Lake architecture and data governance standards.

• Process large-scale structured, semi-structured, and streaming datasets.

• Optimize Spark jobs and cluster performance for maximum efficiency.

• Develop reusable frameworks, notebooks, workflows, and data products.

• Collaborate with business teams, data scientists, analysts, and architects to deliver enterprise-grade solutions.

• Implement monitoring, alerting, security, and compliance requirements.

• Participate in code reviews, deployment activities, and production support.

• Develop technical documentation and mentor junior engineers.

Required Technical Skills

Databricks & Big Data

• Azure Databricks

• Databricks Lakehouse Platform

• Delta Lake

• Spark SQL

• PySpark

• Databricks Workflows

• Unity Catalog

• Delta Live Tables (DLT)

• Structured Streaming

Data Engineering

• Data Lake Architecture

• Lakehouse Architecture

• ETL/ELT Development

• Data Modeling

• Data Integration

• Performance Tuning

• Data Quality Frameworks

Programming Languages

• Python

• PySpark

• SQL

• Scala

• Java

• Shell Scripting

Cloud Platforms

• Microsoft Azure

• Azure Data Factory (ADF)

• Azure Data Lake Storage (ADLS)

• AWS

• Amazon S3

• AWS EMR

• Google Cloud Platform (GCP)

Databases

• SQL Server

• Oracle

• PostgreSQL

• MySQL

• MongoDB

• Cassandra

Streaming & Messaging

• Kafka

• Event Hubs

• Spark Streaming

• Azure Stream Analytics

DevOps & Tools

• Git

• Jenkins

• Azure DevOps

• CI/CD Pipelines

• Terraform

• Unix/Linux

Required Qualifications

1. Bachelor's or Master's degree in Computer Science, Engineering, Information Technology, or related field.

2. 6-9 years of IT experience with at least 4+ years of hands-on Databricks and Spark development experience.

3. Strong expertise in distributed data processing and Lakehouse architecture.

4. Experience building enterprise-scale data engineering solutions.

5. Strong knowledge of PySpark, Spark SQL, and performance optimization.

6. Experience with Agile development methodologies.

7. Excellent analytical, troubleshooting, and communication skills.

Preferred Qualifications

1. Experience with cloud-native analytics platforms on Azure, AWS, or GCP.

2. Knowledge of Machine Learning workflows within Databricks.

3. Experience implementing Delta Lake and Unity Catalog.

4. Exposure to Data Governance and Data Security frameworks.

5. Relevant Databricks, Azure, or AWS certifications.

Key Competencies

• Data Engineering Excellence

• Performance Optimization

• Solution Design

• Stakeholder Management

• Collaboration and Leadership

• Ownership and Accountability

Nice to Have

• Snowflake

• Azure Data Factory (ADF)

• Delta Live Tables (DLT)

• Unity Catalog

• MLflow

• Airflow

• Kafka

• Data Mesh Architecture

• Generative AI / AI Engineering exposure

• dbt