HTTP client

Featurehttp-client Most apps call other services: a payment provider, a webhook, an exchange-rate API. navree-http-client gives you one shared HttpClient for that, in the spirit of Laravel's Http facade. It is built on reqwest with rustls, and it adds what you would otherwise write by hand: retries with backoff, a timeout per attempt, a response size limit, a guard against server-side request forgery, logs that never contain secrets, and a fake for tests.

Configuration

Add HttpClientModule to your app. It is global: every module can inject Arc<HttpClient> without importing it.

use navree::prelude::*;

pub fn app(sources: ConfigSources) -> AppBuilder {
    Navree::app()
        .module(ConfigModule::for_root(sources))
        .module(HttpClientModule::from_config()) // section `http_client`
}

Every key of the http_client section is optional:

[http_client]
timeout_secs = 30
connect_timeout_secs = 5
retries = 2
retry_base_ms = 200
max_response_bytes = 10485760
user_agent = "crm/1.0"

[http_client.ssrf]
block_private = true
allow_hosts = ["billing.internal", "*.svc.cluster.local", "10.20.0.0/16"]

From the environment, use the usual APP_ names: APP_HTTP_CLIENT__TIMEOUT_SECS=10, and APP_HTTP_CLIENT__SSRF__ALLOW_HOSTS=billing.internal,10.20.0.0/16 as a comma-separated list. Invalid options fail the boot, naming the section and the key.

To configure the client in code instead, use for_root. Build the options from their default:

use navree::http_client::{HttpClientOptions, SsrfPolicy};

Navree::app()
    .module(ConfigModule::for_root(sources))
    .module(HttpClientModule::for_root(HttpClientOptions {
        timeout_secs: 10,
        retries: 3,
        ssrf: SsrfPolicy {
            allow_hosts: vec!["billing.internal".into(), "10.20.0.0/16".into()],
            ..SsrfPolicy::default()
        },
        ..HttpClientOptions::default()
    }))

Making requests

Inject the client and start a request with get, post, put, patch, delete, or request(method, url) for any other method. Nothing is sent until you await send:

use std::collections::BTreeMap;

use navree::http_client::{HttpClient, HttpClientError};
use navree::prelude::*;
use serde::Deserialize;

#[derive(Debug, Deserialize)]
pub struct Rates {
    pub base: String,
    pub rates: BTreeMap<String, f64>,
}

#[injectable]
pub struct ExchangeRates {
    http: Arc<HttpClient>,
}

impl ExchangeRates {
    pub async fn latest(&self, base: &str) -> Result<Rates, HttpClientError> {
        self.http
            .get("https://rates.example.com/latest")
            .query(&[("base", base)])
            .send()
            .await?
            .error_for_status()?
            .json()
    }
}

#[module(providers = [ExchangeRates], exports = [ExchangeRates])]
pub struct RatesModule;

The builder methods are:

Builder methods never fail. The first problem (an invalid URL, header or body) is kept and returned by send as HttpClientError::Build. Only http and https URLs are accepted.

Here is a webhook call that authenticates, forwards the current request id and sends JSON:

use navree::contracts::RequestId;
use serde::Serialize;
use uuid::Uuid;

#[derive(Debug, Serialize)]
pub struct DealWon {
    pub deal_id: Uuid,
    pub amount_cents: i64,
}

impl ExchangeRates {
    pub async fn notify(
        &self,
        token: &str,
        request_id: &RequestId,
        event: &DealWon,
    ) -> Result<(), HttpClientError> {
        self.http
            .post("https://hooks.example.com/crm/deals")
            .bearer(token)
            .with_request_id(request_id)
            .header("idempotency-key", event.deal_id.to_string())
            .json(event)
            .send()
            .await?
            .error_for_status()?;
        Ok(())
    }
}

Responses

send reads the whole body before it returns, so a Response's accessors are synchronous and can be called any number of times:

let response = self.http.get("https://rates.example.com/status").send().await?;

response.status();             // StatusCode
response.is_success();         // true for 2xx
response.header("retry-after"); // Option<&str>, case-insensitive
response.headers();            // &HeaderMap
response.url();                // the final URL, after redirects
response.text();               // String
response.bytes();              // Bytes
let body: Rates = response.json()?;

A 4xx or 5xx status is not an error: you get the response and decide. Call error_for_status() to turn it into HttpClientError::Status (Laravel's throw()).

A body larger than max_response_bytes fails with HttpClientError::BodyTooLarge as soon as the limit is passed, so a misbehaving server can't fill your memory. There is no streaming response API: for large downloads, raise the limit for that client or use reqwest directly.

Retries and timeouts

Idempotent requests (GET, HEAD, OPTIONS, TRACE, PUT, DELETE) are retried when the connection fails or the server answers 5xx or 429. The delay starts at retry_base_ms and doubles each time, with random jitter, up to 30 seconds. POST and PATCH are never retried, because sending them twice could charge a card twice; retries(n) has no effect on them.

When the last attempt still gets a 5xx or 429, that response is returned as it is. Retry-After headers are not read; the backoff schedule decides.

Each attempt has its own timeout, from connecting to the last body byte. A timed-out attempt fails with HttpClientError::Timeout and is not retried, so a server that hangs costs you one timeout, not one per retry. Opening the connection has its own, shorter limit (connect_timeout_secs), and a connection that can't be opened is retried. Override the timeout and the retries per request:

let response = self
    .http
    .get("https://rates.example.com/status")
    .timeout(Duration::from_secs(2))
    .retries(0)
    .send()
    .await?;

Errors

Every failure is an HttpClientError. URLs in its messages have no query string or credentials, and header values and bodies never appear.

Propagated with ? from a handler, an HttpClientError becomes a logged 500. When the other service failing is not your server's fault, answer something more precise:

use navree::axum::http::StatusCode;

let rates = self.rates.latest(&base).await.map_err(|error| {
    AppError::new(StatusCode::BAD_GATEWAY, "rates_unavailable", "Bad Gateway").with_source(error)
})?;

The client sees {"statusCode": 502, "message": "Bad Gateway", "error": "Bad Gateway"}, and the cause is logged with the request.

The SSRF policy

If a URL comes from a user (a webhook address, an avatar URL to fetch), an attacker can point it at your internal network or at the cloud metadata service. With ssrf.block_private on (the default) the client refuses:

• loopback, private (RFC 1918), link-local, carrier-grade NAT, unique-local, multicast, documentation, benchmarking and other reserved ranges, in IPv4 and IPv6;
• IPv6 addresses that embed one of those IPv4 addresses (IPv4-mapped, IPv4-compatible, NAT64, 6to4);
• the cloud metadata endpoints 169.254.169.254, 169.254.170.2, 100.100.100.200, 192.0.0.192 and fd00:ec2::254, even with block_private off.

The check runs on literal IP URLs before anything is sent, on every address a host name resolves to (connections only go to the addresses that were checked, so DNS rebinding can't slip through), and on every redirect hop (at most 10). A refused request fails with HttpClientError::Blocked, naming the range and the address.

To call an internal service, list it in ssrf.allow_hosts:

• a host name, billing.internal (case-insensitive): every address it resolves to is accepted;
• a host glob, *.svc.cluster.local;
• an IP address, 127.0.0.1, or a CIDR network, 10.20.0.0/16.

Set block_private = false only when the client never sees user-supplied URLs. The client also ignores the system's proxy settings (HTTP_PROXY and friends), so requests go where the policy checked.

Logging

Each request is logged at debug level with its method, its URL without query string or credentials, the status, the number of attempts and the time taken; each retry is logged too. Header values and bodies are never logged, so tokens in headers or query strings stay out of your logs. To see them, set levels = { navree_http_client = "debug" } in the log section (or APP_LOG__LEVELS=navree_http_client=debug); see Logging.

Testing

HttpClient::fake() returns a client that never touches the network and the FakeServer that answers it. Stub responses with on, swap the fake into your app with override_value, and assert on what was sent:

use navree::axum::http::Method;
use navree::http_client::{FakeResponse, HttpClient};
use navree::prelude::*;
use navree::testing::TestApp;
use serde_json::json;

#[tokio::test]
async fn latest_rates_come_from_the_provider() {
    let (client, server) = HttpClient::fake();
    server.on(
        Method::GET,
        "https://rates.example.com/latest?*",
        FakeResponse::new(200).json(&json!({ "base": "EUR", "rates": { "USD": 1.08 } })),
    );
    let app = TestApp::builder()
        .module(RatesModule)
        .override_value(Arc::new(client))
        .build()
        .await;

    let rates = app.get::<ExchangeRates>().await.latest("EUR").await.unwrap();

    assert_eq!(rates.rates["USD"], 1.08);
    server.assert_sent(|request| request.url().query() == Some("base=EUR"));
    server.assert_sent_count(1);
    app.shutdown().await.unwrap();
}

• Stubs match a method and a glob over the full URL, query string included (* is any characters, ? one character). They are tried in the order you register them. A request no stub matches fails with HttpClientError::NoFakeResponse.
• on_sequence(method, glob, [responses]) answers with each response in turn, one per attempt (retries included), then repeats the last one.
• FakeResponse: new(status), then .json(&value), .text(..), .body(..) and .header(name, value); .delay(duration) to test timeouts; FakeResponse::connection_failed() for a refused connection.
• Assertions: assert_sent(predicate), assert_not_sent(predicate), assert_sent_count(n) (retries count) and assert_nothing_sent(). recorded() returns every RecordedRequest, with method(), url(), header(name), headers(), body(), text() and json().

Retries, timeouts and the size limit behave as they do on a real client; HttpClient::fake_with(options) takes options. The SSRF policy applies to literal IP URLs only, since a fake resolves no host names. Backoff delays and .delay(..) are real time, so pause Tokio's clock to keep tests fast (this needs Tokio's test-util feature in your dev-dependencies):

#[tokio::test(start_paused = true)]
async fn a_flaky_provider_is_retried() {
    let (client, server) = HttpClient::fake();
    server.on_sequence(
        Method::GET,
        "https://rates.example.com/status",
        [FakeResponse::new(503), FakeResponse::connection_failed(), FakeResponse::new(200)],
    );

    let response = client.get("https://rates.example.com/status").send().await.unwrap();

    assert_eq!(response.status(), 200);
    server.assert_sent_count(3);
}

See Fakes for the other fakes Navree ships.

Related