Staff Software Engineer (Agentic Search) - Crawler
Nebius — Amsterdam, North Holland, Netherlands · In-Person · Staff+
Role description
Role Overview
Nebius is building a full-stack AI cloud platform that supports developers and enterprises. The Staff Software Engineer will work on the content acquisition and crawling infrastructure of a novel search engine tailored for agentic AI consumption.
What You Will Do
Design, implement, and operate web-scale crawling systems for acquiring content from the internet, build ingestion workflows for internal and external data sources, and ensure reliable operation of crawling infrastructure under high-throughput conditions.
Why It Might Be a Fit
Strong Go or C++ expertise, experience with large-scale distributed systems, and understanding of web protocols, crawling, scraping, and content extraction are required. Experience with web crawling, building streaming data pipelines, and designing distributed schedulers is a plus.
Requirements
- 8+ years of experience building backend or distributed systems
- Strong Go or C++ expertise
- Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines)
- Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction
- Experience operating production systems and debugging failures in distributed environments
- Strong understanding of scalability, fault tolerance, and resource management
Benefits
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams