Python Developer

About the Role:

We are looking for an experiencedPython Developer with expertise inweb scraping,automated portal interactions, andcloud-native deployment usingAWS. The ideal candidate will have hands-on experience working withPlaywright for browser automation, managingmulti-factor authentication (MFA) flows, and deploying scalable scraping tasks viaAWS Lambda and related services.

You will assist in architecting and building robust, secure, and scalable scraping solutions that interact with complex web applications and secured portals.

Responsibilities:

• Assist with the design and implementation of advanced scraping solutions usingPython,Playwright, andAWS services.
• Automate interactions with JavaScript-heavy and authentication-secured websites, including handlingMFA, CAPTCHAs, and session/token-based login flows.
• Architect scraping pipelines usingserverless AWS components such asLambda,Step Functions,S3,CloudWatch, andSecrets Manager.
• Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging.
• Integrate and manage complex workflows across multiple portals, APIs, and data sources.
• Contribute to architectural decisions, tooling, and best practices.

Qualifications:

• 5+ years of experience in Python development, with a strong focus on automation and data extraction.
• Proven expertise inweb scraping using tools like Playwright, Selenium, Scrapy, BeautifulSoup, and requests.
• In-depth experience handlingmulti-step authentication flows, includingmulti-factor authentication (MFA), CAPTCHA solving, and session/cookie management.
• Proficient in deploying and managing scraping workloads inAWS, particularlyLambda,S3,IAM,CloudWatch, andSecrets Manager.
• Experience withasynchronous programming,headless browsers, andJavaScript-rendered content.
• Solid understanding of web protocols (HTTP, HTTPS, cookies, headers), and ability to reverse-engineer network calls and authentication mechanisms.
• Comfortable working withAPIs,JSON/XML, and data transformation.

Preferred Qualifications:

• Experience withCI/CD pipelines,Docker, and infrastructure as code (e.g., CloudFormation, Terraform).
• Familiarity withdata pipeline orchestration tools such as Apache Airflow or AWS Step Functions.
• Prior experience scraping fromsecure or enterprise-level portals (e.g., financial, healthcare, legal).
• Background indata engineering orETL workflows is a plus.
• Exposure toPython testing frameworks and writing unit/integration tests.

Education:

• Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field, or equivalent professional experience.
  
  
  

We believe the best ideas and innovation happen when we are together. Employees in this role will work in the office four days, with the flexibility to work remotely one day (Friday). 

Benefits

iCapital offers a comprehensive benefits package that includes a total compensation program consisting of competitive salary, annual performance bonus, and equity for all full-time employees; healthcare with 100% employer-paid health and dental insurance; and generous paid time off (PTO).

For additional information on iCapital Network, please visithttps://www.icapitalnetwork.com/about-us  Twitter: @icapitalnetwork | LinkedIn:https://www.linkedin.com/company/icapital-network-inc