Job Responsibilities:
Build and manage data pipelines (batch and real-time) using Databricks / Snowflake Implement data ingestion frameworks (API, streaming, batch) for external sources (e.g., Moody's) Develop data transformation, standardisation, and enrichment pipelines Implement data quality checks and CDQ rules within pipelines Integrate pipelines with metadata and lineage platforms Automate workflows and integrate with ServiceNow for issue management Support implementation of data governance controls and audit traceability Build reusable data engineering components for scalability
Job Requirements: Minimum 10 years of relevant experience Strong hands-on experience in Databricks (Spark, Delta Lake) / Snowflake Experience in ETL/ELT pipelines, API integrations, streaming (Kafka/Event Hub) Knowledge of data quality frameworks and validation techniques Familiarity with data lineage integration and metadata tagging Programming skills (Python, SQL, Spark) Experience with cloud pla...