Job Title:  Technology & Transformation - Engineering - Consultant/Senior Consultant - Azure Data Engineer

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines using Azure Data Factory and Azure Databricks.
• Work with structured and unstructured data from diverse sources such as SQL databases, APIs, SAP, flat files, and cloud storage.
• Implement Delta Lake, data ingestion frameworks, and real-time/streaming pipelines where required.
• Optimize Databricks notebooks and pipelines for performance, scalability, and cost efficiency.
• Ensure high data quality through validation, reconciliation, and automated checks.
• Collaborate with product owners, architects, and business teams to understand data requirements.
• Implement best practices in CI/CD using Azure DevOps, version control, and environment management.
• Support production pipelines, troubleshoot issues, and ensure timely resolution.
• Contribute to documentation, design specs, and operational playbooks.

Required Skills

• Strong hands-on experience with Azure Databricks and PySpark.
• Expertise in Azure Data Factory (ADF) pipeline development.
• Practical knowledge of Delta Lake, schema evolution, and data lake optimization.
• Experience with SQL, performance tuning, and data modeling.
• Working knowledge of Azure services such as ADLS, Azure Key Vault, Azure Monitor, etc.
• Familiarity with Azure DevOps (Repos, Pipelines, CI/CD).
• Strong debugging, problem-solving, and analytical skills.

Good to Have

• Experience with Lakehouse architecture.
• Exposure to Synapse, Unity Catalog, Databricks Jobs, and Cluster management.
• Knowledge of real-time streaming (Event Hub, Kafka, Auto Loader).
• Understanding of data governance, security, and access management.