America's Job Portal
Duties and Responsibilities
Build and optimize scalable data pipelines using Databricks, PySpark, Spark, and Azure Data Factory.
• Analyse, clean, and organize raw data from multiple structured and unstructured sources.
• Develop and maintain ETL/ELT workflows following best engineering and DevOps practices.
• Implement CI/CD pipelines using GitHub and Azure DevOps.
• Deploy and orchestrate workloads using Docker and Kubernetes.
• Develop solutions aligned to enterprise data models, DataVault methods, and architectural standards.
• Ensure data quality, data governance, GDPR compliance, and metadata management.
• Use tools like QuerySurge for data testing and validation.
• Collaborate with teams using Agile/Scrum frameworks and tools like JIRA, Confluence, and SharePoint.
• Support production deployments, troubleshooting, and operational...