Building a High-Concurrency OSINT Engine in Rust: How I Managed 35+ Async Streams Without Exhausting Sockets

Short summary

The author details building Reconx, an open-source Rust CLI for OSINT infrastructure mapping, focusing on solving socket exhaustion when running 35+ async collectors. The solution centers on a centralized shared reqwest::Client with Arc-based connection pooling, proxy routing, and built-in rate limiting with exponential backoff. The post includes concrete Rust code for the HTTP engine and discusses state management via asset diffing.

• •Centralized reqwest::Client in Arc eliminates ephemeral port exhaustion across 35+ async OSINT collectors
• •Built-in rate limiting with exponential backoff handles HTTP 429 and 5xx errors automatically
• •Shared proxy configuration gives all collectors instant HTTP/SOCKS5 support without per-module changes

Generated with AI, which can make mistakes.