Tech Stack

Description

You will build solutions for processing big and unstructured data sets, extract and ingest data from websites using compliance-approved web crawling tools, and own the creation of tools and workflows to improve crawl/scrape analysis, reports, and data management.

Requirements

• Solid Python and SQL knowledge
• Experience running automated programs at scale
• Working experience with cloud-based technologies
• Familiarity with Linux/UNIX, HTTP, HTML, JavaScript, and Networking
• Familiarity with crawling/extracting tools (e.g., Requests, BeautifulSoup, Scrapy, Pandas, Selenium, Spark)

Responsibilities

• Own creation process of web scraping tools, services, and workflows
• Test data and scrapes to ensure accuracy, quality, and compliance
• Identify and rectify issues with breaks and scale scrapes as needed