Job Description:

• Design, develop, and maintain ETL pipelines using Informatica, Azure Data Factory (ADF), and Databricks
• Perform migration of legacy ETL workflows (Informatica) to Databricks using Python/PySpark
• Analyze existing ETL workflows and re-engineer into optimized Spark-based transformations
• Develop data processing and transformation solutions using Python and PySpark
• Apply AI/ML techniques for data enrichment, anomaly detection, and predictive insights (where applicable)
• Build and optimize SQL queries, data models, and transformations
• Schedule and monitor jobs using AutoSys
• Integrate data from multiple sources: Relational databases (SQL Server, DB2), Files (CSV, XML, JSON) and mainframe systems
• Streaming platforms like Kafka
• Perform data validation, reconciliation, and ensure data quality
• Troubleshoot ETL/pipeline failures and optimize performance