Staff Software Engineer (Agentic Search) - Crawler
Nebius — London, England, United Kingdom · In-Person · Staff+
Role description
Role Overview
Nebius is looking for a Staff Software Engineer to work on the content acquisition and crawling infrastructure of a novel search engine tailored for agentic AI consumption. The role involves building systems that discover, fetch, and continuously refresh content from the open web and other large-scale data sources.
What You Will Do
Design, implement, and operate web-scale crawling systems for acquiring content from the internet, build ingestion workflows for internal and external data sources, and develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
Why It Might Be a Fit
Strong candidates may have experience with web crawling, building streaming data pipelines and event-driven systems, and designing distributed schedulers, queues, and asynchronous processing systems.
Requirements
- 8+ years of experience building backend or distributed systems
- Strong Go or C++ expertise
- Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines)
- Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction
- Experience operating production systems and debugging failures in distributed environments
- Strong understanding of scalability, fault tolerance, and resource management
Benefits
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams