Senior Python Data Scraping Engineer (Freelance)

About the Role and Company

Mindrift seeks experienced Senior Python Data Scraping Engineers to participate in the Tendem project, focusing on specialized data scraping within a hybrid AI-human system. This role, known internally as AI Pilot, involves partnering with Tendem Agents who handle repetitive tasks, while providing expert analysis, domain knowledge, and quality assurance to produce precise and actionable data.

This position is part-time and remote, tailored for skilled technical professionals with hands-on experience in web scraping, data extraction, and data processing.

Mindrift's Mission

The Mindrift platform connects experts with AI-related projects from leading technology innovators, aiming to maximize the capabilities of Generative AI by incorporating real-world expertise globally.

Key Responsibilities

• Manage end-to-end data extraction workflows from complex websites to ensure thorough coverage, accuracy, and delivery of clean, structured datasets.
• Utilize in-house tools such as Apify and OpenRouter alongside custom developed workflows to speed up data gathering, validation, and execution of assigned tasks per requirements.
• Adapt approaches to reliably extract data from dynamic, interactive web sources including those with JavaScript-rendered content and evolving site behavior.
• Implement rigorous data quality protocols with validation checks, cross-source consistency validation, format standards compliance, and thorough verification before submission.
• Scale scraping operations efficiently for large datasets through batching or parallelization methods; monitor failures and ensure operational stability despite incremental site changes.

Candidate Requirements

• Minimum of five years' experience in data engineering, web scraping, automation, or software development roles.
• Educational background with a Bachelor’s or Master’s degree in Engineering, Applied Mathematics, Computer Science, or similar technical disciplines is advantageous.
• Strong technical foundation including proficiency in scripting, automation, and AI-supported workflows. Ability to solve complex challenges, utilize large language models (LLMs), and systematically procure, structure, and validate diverse data types.
• Extensive experience with Python-based scraping tools (BeautifulSoup, Selenium, or equivalents), handling dynamic content such as JavaScript, AJAX, infinite scrolling, and API integrations through proxies.
• Demonstrated expertise in extracting data from intricate structures including hierarchies, archived pages, and inconsistent HTML layouts.
• Proficiency in data cleansing, normalization, and validation to produce structured data files (CSV, JSON, Google Sheets).
• Experience managing anti-bot measures and dynamic website architectures at scale.
• Familiarity with cloud platforms (AWS or similar) and container technologies like Docker within actual project workflows.
• Hands-on involvement with LLM frameworks such as LangChain or OpenRouter for automating workflows.
• Detail-oriented mindset with dedication to ensuring data accuracy and quality.
• Ability to work autonomously with strong problem-solving skills.
• Providing a GitHub profile is advantageous.
• English language proficiency at an upper-intermediate level (B2) or higher is required.

Project Details and Compensation

The workload during active project phases is estimated to be approximately 10 to 20 hours per week, although this is subject to change and not guaranteed.

Compensation for this project can reach up to $37 per hour depending on expertise and productivity levels. Pay rates may vary for different projects throughout the platform based on their complexity and scope.