Responsibilities

• Build and design large scale, distributed crawling bots and infrastructure
• Develop and maintain data pipelines to extract data from web pages, documents, PDFs, and APIs
• Unify heterogeneous documents into a coherent data schema
• Preprocess and normalize raw data for downstream classification, ML/NLP, and search indexing
• Build APIs to expose structured, classified data via ElasticSearch/OpenSearch
• Collaborate with ML/NLP teams to integrate classification models into the pipeline
• Use LLMs to find precise answers in OEM repair procedures
• Automate workflows using Apache Airflow and deploy solutions in Kubernetes on AWS
• Optimize and scale data pipelines using Spark (EMR)

Requirements

• 5+ years of experience in Python with building crawling/scraping solutions at scale
• Experience working with APIs (REST)
• Experience with PDF processing (OCR, Te...