About NewsCatcher

Turning the web into a structured database of real world events.

Tech description:

Large-scale web crawling and web scraping. Our tech stack: - Docker & K8s - Jenkins - Python (including asynchronous workloads) - RabbitMQ - Elasticsearch - MongoDB/Dynamo - PostgreSQL, MySQL - Gitlab - AWS, GCP, DO

Job description:

### **Functions:** * Design and implement new data pipelines tailored to a specific project or new product. * Continuously improve data pipelines to make them more generic and modular, reducing the effort required for subsequent integrations. * Write, optimise and refactor prompts and prompt-related pipelines. * Conduct thorough testing and validation of data pipelines to ensure accuracy and consistency of data. * Ensure pipelines are optimized for performance, scalability, and reusability to facilitate future projects. ### **Examples of day-to-day tasks** 1. We want to support people’s enrichment within the CatchAll Tool. Do research on what is available on the market that can help us enrich data, Double-check open databases, and verify whether there is already an existing solution available within the NewsCatcher’s code base. Plan and design the pipeline. Build a prototype and demo it to the whole team. Proceed to, test on Dev, write test cases around. Deploy on Prod, get feedback and improve. 2. One of the users complained about a job being stuck. Debug the pipeline, try to understand where the problem comes from: promts, code, database. Find the issue and prepare a fix. Test it within the whole pipeline on Dev. Break some pipeline, fix and test again. Deploy on Prod, make clients happy. 3. Prompt responsible for validating results is giving only 50% precision. Review the prompt, review the model. Do some prompt engineering. Realize that “with each new model I feel like LLMs become stupier”. Try another LLM provider, adapt the prompt. Test results, improve the accuracy and create a dataset to prove it. Get 70% precision, wait for compliments. ### **Experience:** * 3+ years of experience in B2B SaaS as Backend / Software / Data Engineer * Strong systems thinker with attention to performance and scalability * Comfortable working with both SQL and NoSQL databases * Experience shipping LLM-based functionality into production * Able to move from prototype → stable, maintainable architecture ### Must Have * Strong Python (including async workloads) * Docker & Kubernetes * RabbitMQ (or Kafka / PubSub / SNS / ActiveMQ) * MongoDB / DynamoDB / Redis * Hands-on experience with LLM frameworks ### We Also Use * Elasticsearch/OpenSearch * PostgreSQL / MySQL * GitLab * AWS / GCP / DigitalOcean * Jenkins ### Nice to Have * Experience building production AI agents * Frontend experience (useful for product UI) * Experience in API-first or DaaS companies ### **Compensation and Perks**: * Competitive salary and equity * Up to 24 days of vacation & 16 days of sick leave/holidays (all fully paid) * One meeting-free day per week * Co-working Budget * Training Budget * We provide all the necessary equipment to work comfortably and efficiently from home. * Yearly company retreats (2025 — Portugal, 2024 — Canary Islands, 2023 — French Alpes)

Skills:

Kubernetes, Python, Docker

NewsCatcher remote jobs

Software Engineer (Backend & LLM)