Job Title:  Lead Senior Associate | Data Engineering | Bengaluru | Engineering as a Service/ Operate

Role Title: Sr Data Engineer I

Responsibilities: 

Pipeline Development & Operations:

•  Design, maintain, and optimize Spark and Flink pipelines running in Databricks and Airflow.

•  Ensure reliability and availability of data workflows in production.

•  Debug and resolve issues in notebooks, jobs, and orchestration workflows.

Performance Optimization:

•  Tune Spark jobs, clusters, and data pipelines for performance and cost efficiency.

•   Analyze logs, queries, and workloads to identify bottlenecks and implement improvements.

Monitoring & Observability:

•   Build and maintain dashboards, alerts, and metrics with Datadog, CloudWatch, or Prometheus.

•   Proactively monitor pipeline health and escalate issues as needed.

Incident Response & RCA:

•  Participate in incident response for pipeline issues, helping diagnose and resolve outages.

•   Contribute to RCA and preventative actions to improve long-term stability.

Collaboration & Knowledge Sharing:

•  Partner with data engineering, BI, and platform teams to support delivery and operations.

•  Document best practices, contribute to runbooks, and mentor junior engineers.

Basic Qualifications: 

•  4+ years of experience in data engineering or big data platform operations.

•  Hands-on experience with Databricks, Spark, Flink, or Airflow (MWAA) in production.

•  Strong SQL skills and experience with ETL/ELT pipeline development.

•  Familiarity with AWS services (S3, Glue, Athena, EMR, EKS) for pipeline management.

•  Working knowledge of observability tools (Datadog, CloudWatch, Prometheus).

•  Proficiency in Python, Scala, or Java for pipeline development and automation.

•   Familiarity with infrastructure-as-code (Terraform or similar).

•   Strong collaboration and problem-solving skills, with experience in distributed teams.

Preferred Qualifications: 

•  Experience with incident response and RCA in production data environments.

•   Familiarity with CI/CD pipelines (GitLab, Jenkins, or GitHub Actions).

•   Exposure to data governance, access control, and cost optimization practices.

•  Certifications such as Databricks Data Engineer Associate or AWS Solutions Architect – Associate.

Required Education:

•  Bachelor’s degree in Computer Science, Engineering, or related technical field (or equivalent experience).