Freelance Python Data Scraping Engineer

About Mindrift and the Role

Mindrift is seeking experienced Python Data Scraping Engineers to contribute remotely on a part-time freelance basis to the Tendem project. This innovative project harnesses the synergy between AI systems and human expertise, where the role—referred to internally as an AI Pilot—supports automated Tendem Agents by applying advanced critical thinking, domain knowledge, and thorough quality control to achieve accurate, actionable data extraction results.

Key Responsibilities

• Manage comprehensive data extraction processes from complex websites, ensuring thorough coverage and accuracy while delivering structured datasets.
• Utilize internal tools such as Apify and OpenRouter in combination with custom-developed workflows to streamline data collection, verification, and task completion in alignment with project specifications.
• Adapt scraping methodologies to effectively extract data from dynamic, interactive web sources, including those powered by JavaScript and AJAX, adjusting strategies for evolving site behaviors.
• Maintain rigorous data quality by performing validation checks, ensuring consistency across sources, following formatting protocols, and implementing systematic verifications before delivery.
• Optimize scalability for large datasets via batching and parallelization techniques, monitor and troubleshoot failures, and ensure resilience against minor changes in web page structures.

Candidate Requirements

• Minimum 3 years of professional experience in data engineering, web scraping, automation, or software development.
• Educational background preferably includes a Bachelor's or Master's Degree in Engineering, Applied Mathematics, Computer Science, or a related technical field.
• Proficient in Python web scraping frameworks such as BeautifulSoup and Selenium, with substantial experience handling dynamic web content including JavaScript, AJAX, and infinite scroll, as well as API integrations via proxies.
• Demonstrated skill in extracting data from complex and inconsistent web structures, including hierarchical layouts and archived web pages.
• Strong expertise in data cleaning, normalization, validation, and producing well-structured outputs like CSV, JSON, or Google Sheets formats.
• Hands-on experience leveraging large language models and AI frameworks to improve automation and problem-solving capabilities.
• Exceptional attention to detail ensuring the accuracy of data delivered.
• Self-motivated with excellent problem-solving skills and ability to work independently.
• English language proficiency at an upper-intermediate (B2) level or higher is required.
• Providing a GitHub profile is advantageous but not mandatory.

Compensation and Benefits

• This freelance role offers compensation up to $37 per hour depending on expertise and contribution speed.
• Pay rates vary across different projects based on complexity and scope.
• Operate fully remotely on a flexible schedule requiring only a laptop and reliable internet connectivity.
• Engage in a cutting-edge hybrid environment that blends human capabilities with AI agents, fostering unique skill development in a fast-growing industry.
• Opportunity to earn performance-based bonuses rewarding high-quality and consistent work delivery.