HTTP Crate with URL Support & a Simple HTTP Client

While the http crate generally has a great API I have been unsatisfied how it handles URLs. To create a HTTP request a full URL is needed with a scheme (http/https), authority (example.org) and a path (/search?q=rust) but http does enforce this and allows you to only state the path. This means both clients and servers are either unable to determine protocol and and authority information or have to do this manually.

There is already the well-established rust-url crate but it is not used by http. This has been criticized by multiple people as the http crate does not even provide conversion functions for better ergonomics. As Armin Ronacher (mitsuhiko) puts it:

With the url crate we have a pretty damn good (new) standard compliant crate that is very popular. Right now I use this everywhere and then when doing requests to hyper I need to convert them which is not very great ergonomics and performance wise.

I’ve forked the http crate and created my own http-with-url demo. Each request is created with a complete URL:

let url = Url::parse("https://www.google.com/search?q=rust");
let request = Request::new(url, ());

A client can deconstruct the URL into pieces: in HTTP/1.1 www.google.com is the Host header, /search?q=rust is part of the request line and https tells the client to establish a secure connection. A server already knows the protocol and can build from the Host header and the path a complete URL. Currently this task is left to the application authors by the http crate.

Some edge cases:

• HTTP defines “server-wide” requests with a path value of *. They are rarely if ever used but can’t be expressed by the standard url crate. Instead I allow adding an extension called WholeServer to requests. If it is present the path of the URL is ignored and instead a server-wide request is sent.
• Servers sometimes don’t know the host. The Host header field is only required in HTTP/1.1 but the ancient HTTP/1.0 assumed that each host/domain had its own IP address and therefore for each request sent to a given IP address the host was known by the server. As the host part can’t be omitted in an absolute URL I propose to instead use a placeholder value like the broadcast IP 255.255.255.255 instead.

HTTP demo client

To demonstrate how to use the http-with-url crate I have written a simple client.

extern crate boguin;
extern crate http_with_url as http;

fn main() {
    let mut client = boguin::Client::new();
    let url = http::Url::parse("https://httpbin.org/status/418").unwrap();
    let request = http::Request::new(url, ());
    let response: http::Response<String> = client.fetch(request).expect("request works");
    println!("{}", response.status());
    println!("{}", response.body());
}

It combines a few existing crates to parse HTTP (httparse), provide proven TLS encryption (native-tls) and of course the http crate itself. With these crates I found I quite easy to write the “glue” code to build a functioning HTTP client.

Also try out the command line client.