Staff Software Engineer (Agentic Search) - Crawler
Nebius — Zurich, Zurich, Switzerland · In-Person · Staff+
Role description
Role Overview
We are building an agent-native search platform designed specifically for AI systems. The Staff Software Engineer will focus on building systems that discover, fetch, and continuously refresh content from the open web and other large-scale data sources.
What You Will Do
Design, implement, and operate web-scale crawling systems for acquiring content from the internet, build ingestion workflows for internal and external data sources, and develop crawl scheduling, prioritisation, recrawl policies, and freshness strategies.
Why It Might Be a Fit
8+ years of experience building backend or distributed systems, strong Go or C++ expertise, and experience with large-scale distributed systems.
Requirements
- 8+ years of experience building backend or distributed systems
- Strong Go or C++ expertise
- Experience with large-scale distributed systems (10k+ RPS, billions of URLs, high-throughput pipelines)
- Understanding of web protocols (HTTP, DNS, TLS), crawling, scraping, and content extraction
- Experience operating production systems and debugging failures in distributed environments
- Strong understanding of scalability, fault tolerance, and resource management
Benefits
- Competitive compensation
- Career growth and learning opportunities
- Flexibility and ownership
- Collaborative and innovative culture
- Opportunity to work on impactful AI projects
- International environment and talented teams