We are seeking a talented Data Engineer with over 5 years of experience in Databricks, Delta Lake, and Azure Data Services. The ideal candidate will have hands-on expertise in building scalable data ingestion, transformation, and AI-driven data pipelines using our agents, contributing to our data-driven decision-making processes.

Key Responsibilities:

Build and maintain data ingestion and transformation pipelines.

Implement ingestion, transformation, and validation agents.

Optimize Spark workloads for improved performance and efficiency.

Implement data quality measures and testing hooks to ensure data integrity.

Support Continuous Integration/Continuous Deployment (CI/CD) deployments.

Qualifications:

5+ years of experience in Databricks, Delta Lake, and Azure Data Services.

Proven expertise in building scalable data pipelines.

Strong understanding of data quality and testi...