Introduction
The role involves building and maintaining scalable data ingestion and transformation pipelines within Databricks, ensuring compliance with governance and security standards, and driving infrastructure automation using Infrastructure as Code (IaC) tools such as Terraform.
Required Skills & Qualifications
- 7 years of experience in Data Engineering and/or DataOps.
- Strong proficiency in Python and SQL.
- Hands-on expertise with Databricks, AWS Glue, Kinesis, Firehose, and other cloud-native services (AWS).
- Deep understanding of Delta Lake and Databricks Unity Catalog.
- Proven experience with Infrastructure as Code tools, especially Terraform.
- Strong knowledge of data governance, security, and compliance standards.
- Excellent communication, collaboration, and problem-solving skills.
- Prior work experience at client or in client's Industry.
<...