Results
The result was a custom API gateway tailored for the company’s needs, with only 2ms of latency added versus a backend calling another backend directly. A new backend joins by handing Sentinel its OpenAPI spec, either a URL or an uploaded file but not both, and the service with all its routes is created from it. Every backend now routes through it, more than 24 of them.
Tracing came with it: structured logs shipped to OpenObserve, so we can see what other services are failing at, how many 500s we emit, our slowest endpoints, alerts...
But the part I am actually proud of is the proxy section : a body forwarded frame by frame, never collected in memory.
The situation
In my first year at my current company, they were building a lot of backends to organize different sections of the business (a logistics company in a heavy digitalization process, where all the software is built in house). Systems for HR, logistics and analytics.
Each new one made the same three things harder: permissions, knowing whether a service is up or down, and catching failures before a client does. We were reacting when something broke instead of detecting problems first.
There was already an attempt at a gateway: a proof of concept built in Python with Django that transitioned into production before anyone analyzed it. When I was asked to review it, I found significant latency, poor security, and GET requests only.
What I proposed
After giving my opinions on the current design, its limitations and where the company was heading, I proposed building a full API gateway in Rust, with the features for control and documentation that were missing: routes that register themselves instead of being typed in by hand, API key handling for auth, and visibility into every request.
With more than 24 backends in production, each one needing to talk to the others, there are 276 possible pairs, and every pair would otherwise need its own auth, tracing and retries. Through one gateway it is 24 integrations, written once.
What made it hard
I had built backends in Rust before, invoice processing and automations, but never anything at the network layer. I knew the most important thing, besides security, was performance and the lowest latency possible.
So I needed a way to keep memory flat even on a big request with a big body, and a way to send and receive bytes in real time without ever holding the whole body.
I knew Tokio would help enormously. But I had never had to think about moving bytes in real time before. That’s when I was introduced to streaming.
I built it on Salvo, a web framework on top of Tokio for async and low-level network libraries. I needed to learn about asynchronous streams, low-level polling, pinning, implemented Stream on custom struct...
Inside the gateway
- Response bodies are streamed frame by frame and never collected. The only thing kept in memory is a small fixed slice of an error response, captured for logging, and that slice is deliberately small
- Route matching scores candidates by specificity: a literal beats
{param}, which beats{**wildcard}, and a partial segment sits between the two - A fresh RS256 token on every request, carrying the caller and the destination endpoint, signed with an RSA private key so a service can verify with the public key that the request really came through Sentinel