About the Role:
We’re looking for a talented Web Data Automation Engineer to join our team and help us harness the power of web data. In this role, you’ll design and build automated systems to collect and monitor data from various websites, ensuring we stay ahead with up-to-date insights. You’ll create proactive tools and bots that track changes in web content over time, while also handling one-off data extraction tasks. Your expertise in web technologies and automation will be key to driving our data collection efforts efficiently and effectively.
Key Responsibilities:
Design and implement automated systems for continuous web data collection and monitoring.
Build proactive bots or agents to track and extract changes in web data, adapting to evolving site structures.
Perform one-off data scraping tasks to meet specific project needs.
Analyze websites using HTML, CSS, JavaScript, and the DOM to identify data acquisition opportunities.
Leverage browser developer tools to inspect network traffic and replicate API calls for efficient data retrieval.
Develop solutions to overcome challenges like captchas, IP restrictions, or dynamic content barriers.
Optimize automation tools for performance, scalability, and reliability.
Work with team members to understand data needs and deliver accurate, timely results.
Requirements:
Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience).
2-4 years of experience with web scraping, data extraction, or automation.
Strong coding skills in languages like Python, JavaScript, or similar.
Experience with web scraping tools (e.g., BeautifulSoup, Scrapy, Selenium, Puppeteer).
Experience with using webscraping API Services such as Zenrows or equivalent
Solid knowledge of web technologies, including HTML, CSS, JavaScript, and the DOM.
Ability to use network analysis tools to identify and utilize API calls.
Experience building automated bots or systems for data collection.
Problem-solving skills and a keen eye for detail.
Good English language communication skills for collaborating with the team.
Preferred Qualifications:
Experience with automation frameworks or scheduling tools (e.g., Apache Airflow, Cron) for ongoing tasks.
Knowledge of proxy services, IP rotation, or headless browsers to handle anti-scraping measures.
Familiarity with cloud platforms (e.g., AWS, Azure, GCP) for scalable solutions.
Understanding of ethical data collection practices and legal compliance.
Experience with dynamic, JavaScript-heavy websites.