Director of Software Engineering (Node.js & Web Scraping Expert)

Key Responsibilities- Software Engineering & Web Scraping Leadership:

• Architect, develop, and maintainscalable and distributed web scraping systems usingNode.js.
• Design and implementdata extraction pipelines to process large volumes of structured and unstructured data.
• Develop solutions tobypass anti-bot mechanisms, includingCAPTCHA handling, session management, fingerprinting, and IP rotation.
• Optimize scraping processes forperformance, reliability, and efficiency while managingproxy services(residential, datacenter, rotating).Overseedata storage and processing strategies, ensuring high availability and consistency.
• Collaborate withProduct, DevOps, and Data Science teams to integrate extracted data into analytics and business applications.
• Implementbest practices for microservices, API integrations, and real-time data streaming.

Key Responsibilities- Scalability, Security & DevOps:

• Lead the transition tocloud-native, containerized, and serverless architectures for web scraping.
• Ensure compliance withlegal and ethical standards (robots.txt, GDPR, CCPA, etc.).Optimizecloud resources (AWS, GCP, or Azure) to support high-throughput scraping.
• Managereal-time monitoring and alerting systems to detectscraping failures, IP bans, or performance bottlenecks.
• Work closely withDevOps teams to optimizeCI/CD pipelines, automated deployments, and system scalability.

Key Repsonsibilities- Engineering Team Management & Strategy:

• Lead, mentor, and grow ahigh-performance engineering team.
• Define and execute thetechnology roadmap, aligning with business objectives.
• Foster a culture ofcontinuous learning, collaboration, and innovation.
• Implementagile development methodologies (Scrum, Kanban) to optimize project execution.
• Ensurecode quality, security, and best practices across all engineering efforts.

Qualifications & Experience- Technical Expertise:

• 10+ years of experience insoftware engineering, with at least5+ years inweb scraping and large-scale data extraction.
• Strong hands-on expertise inNode.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
• Extensive experience inhandling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
• Deep knowledge ofproxy management (residential, datacenter, rotating, and VPNs).Experience withNoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
• Familiarity withdata processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).Strong experience withCI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).

Qualifications & Experience- Leadership & Soft Skills:

• Proven track record ofscaling engineering teams and leading complex projects.
• Strongproblem-solving and debugging skills, especially forscraping challenges and performance bottlenecks.
• Excellentcommunication and stakeholder management skills.
• Passion formentorship, team development, and continuous learning.

Preferred Qualifications:

• Experience withmachine learning for data extraction and NLP.
• Knowledge ofbrowser fingerprinting and bot detection mechanisms.
• Familiarity withenterprise-scale web crawling frameworks (Scrapy, Colly, Apify, etc.).
• Prior leadership experience indata-driven businesses or web scraping startups.