Java Developer with Web Crawler Experience

Role: Java Developer with Web Crawler Experience

Location: Austin TX(Hybrid)

Responsibilities:

1. Web Crawler Development: Design and implement efficient and scalable web crawlers in Java to collect data from various online sources.

2. Data Extraction: Develop and maintain systems for structured data extraction, handling various data formats (HTML, JSON, XML, etc.).

3. Data Storage and Processing: Design data storage and processing pipelines, ensuring extracted data is clean, structured, and easily accessible.

4. Performance Optimization: Optimize web crawling processes for speed, efficiency, and accuracy, while ensuring minimal impact on source websites.

5. Error Handling and Logging: Implement error-handling mechanisms and logging systems to detect and resolve issues during crawling operations.

6. Data Integrity and Compliance: Ensure data collection practices are ethical, legal, and compliant with relevant regulations (e.g.,...