Job Title:  Technology and Transformation - EAD - Engineering -Senior Consultant /Consultant - Data Engineer

Key Skills :

GCP, Spark, SQL, Scala , AIRFLOW, AUTOMIC , CICD , BIG QUERY , PySpark, Unix , Python , Hive , Hadoop, Kafka, Spark Streaming

· GCP : BigQuery, Cloud Storage, Dataproc, Dataflow, Pub/Sub, and Bigtable

· ETL : Experience in designing, building, and managing ETL/ELT pipelines

· Hadoop & Spark : Proficiency with Hadoop ecosystem tools and Apache Spark for large-scale data processing.

· HIVE : SQL Proficiency (HiveQL), Performance tuning & optimization Joins and Subqueries, Window functions, Partitions and Buckets

· DevOps CI/CD

Responsibilities

· Design, develop, implement and tune distributed data processing pipelines that process large volume of data; focusing on scalability, low -latency, and fault-tolerance in every system built

· Engage with Product Management and Business to drive the agenda, set your priorities and deliver awesome product features to keep platform ahead of market scenarios.

· Influence cross functional architecture in sprint planning

· Provide business insights, while leveraging internal tools and systems, databases and industry data

· Skillset:

· Proven work experience in Spark, Python/Scala, Java, SQL, Hive, Any RDBMS/NoSQL database

· Demonstrates expertise in writing complex, highly-optimized queries across large data sets

· Hands on experience with one of the cloud environments either on GCP , Azure or others

· Experience in Unix/Linux shell scripting or similar programming/scripting knowledge

· Understanding of the CI/CD framework

· Exposure to one or more tools such as apache Airflow, Nifi

· Develop, Test, deploy and maintain automation processes to extract data across various data systems, platforms and filetypes; creates data pipelines and transform data into structures that is relevant to the problem by applying appropriate techniques; and automate the end-to-end loading process into target databases.

· Perform detailed data analysis, data quality checks, data cleansing on extracted data, and perform source to target mapping exercises; and identify integration challenges and develop, test and deploy appropriate solutions.

· Analyze complex data elements, systems, data flows, dependencies, and relationships to contribute to conceptual, physical, and logical data models; define relational tables, primary and foreign keys, and stored procedures to create a data model structure; and evaluate existing data models and physical databases for variances and discrepancies.

· Interact with various data sources to perform data analytics and derive business insights, and communicate these through reports and dashboards.

· Participate in discussions and collecting business requirements from stakeholders and translate to design and development specifications.

· Troubleshoot business and production issues by gathering information (for example, issue, impact, criticality); engage support teams to assist in the resolution of issues; and perform actions as designated in the action plan.

· Produce well-tested code and participate in code reviews to ensure code base adhere to standards.

· Contribute to code documentation, maintaining client implementation playbooks and provide training to others on data models, data pipelines and the Extract-Transform-Load (ETL) processes.