Job Title:  Consultant | Azure Data Engineer | Bengaluru | Engineering

Manager : Sanjeev Heerorker

Data engineer

As a data engineer, candidate will have an opportunity to work on the universe of data and solve some very interesting problems by creating and maintaining scalable data pipelines dealing with petabytes of data.

Requirements and skills

Azure Cloud , Python/Scala ,spark with Databricks, SQL ,Power BI

• Hand-on expertise in Analyze and organize raw data and Good Understanding of data model is a must with good technical expertise on Azure Databricks, Spark, Python, SQL

• Good experience in Azure and Azure Data Factory, Storage accounts and Notebooks

• Good experience with DWH, ETL/ELT process, hands-on expertise ETL tools, relational DB or MPP, query tuning and performance tuning, and Apache Kafka

• Good knowledge in DevOps, CI/CD including deploying a range of data engineering pipelines into production, testing techniques, and Agile development methodology

• Good knowledge of data architecture patterns (Data lakehouse, delta lake, streaming, Lambda/Kappa architecture)

• Experienced in PowerShell scripting for orchestration

• Good presentation, communication (written and verbal English) skills, and interpersonal skills and comfortable in liaising with stakeholders independently

• Work with Business & IT teams to gather, understand and define requirements and break down Demands into EPICS and User Stories, where needed.

• Design and build end-to-end CI/CD data pipelines / deployments to get the relevant data and for the deliverables

• Reviewing design, code and other deliverables created by your team to guarantee high-quality results

• Define acceptance criteria, manage & perform testing of own pipelines to ensure expected quality

• Own the status of Change requirements throughout the lifecycle & adapt changes to the existing scripts, codes and pipelines.

• Own PoCs and deliver the results in reasonable time

• Ensuring appropriate use of MB IT project management tools, governance, processes, policies and methodologies

• Bring value adds by contributing to continuous improvement and innovation, encourage best practices, challenge current practices, provide feedback to colleagues

• Presents user stories in demand concept meetings together with respective stakeholders

• Align with architects if and as needed

• Detect dependencies to other systems/demands

• Conduct regular status meetings with all stakeholders, keeping the stakeholders needs and requirements continuously in view