【1】Pingora Getting Started: Load Balancing and Health Checks

Preface

Ever since I heard that Pingora's performance crushes Nginx, handling over 1 trillion requests per day, I've been eager to try it out while learning some Rust along the way. So let's follow the official tutorial step by step.

Official tutorial - if you find any errors in the following content, please refer to the official tutorial: quick_start

Load Balancing Basics:

A load balancer is a device or software that sits between clients and backend servers. It receives client requests and distributes them across multiple backend servers.

Load balancing algorithms determine which backend server receives each request. Common algorithms include: Round Robin, Source IP Hashing, and Consistent Hashing.

Creating a Simple Load Balancing Proxy

Before starting, test connectivity to my blog domain with curl. The official tutorial uses ["1.1.1.1:443", "1.0.0.1:443"], but due to well-known reasons, connectivity can be intermittent. So this tutorial uses self-hosted services for testing.

curl https://testapi.runnable.run/hl/check
curl https://testapi2.runnable.run/hl/check

A successful response indicates everything is working

So testapi.runnable.run and testapi2.runnable.run serve as backend services in this tutorial, and we'll create a load balancer using Pingora.

Create a New Rust Project and Add Dependencies

async-trait="0.1"
pingora = { version = "0.3", features = [ "lb" ] }

Pingora is the star of this tutorial
async-trait: Provides a way to use the async keyword in trait methods. In Rust, async cannot be used directly in trait methods by default, and the async-trait library solves this through underlying mechanisms.

Code Section

The pingora-load-balancing crate provides common selection algorithms for the LoadBalancer struct, such as round-robin and hashing.

pub struct LB(Arc<LoadBalancer<RoundRobin>>);

To make the server a proxy, we need to implement the ProxyHttp trait.

In the upstream_peer() body, we use LoadBalancer's select() method to round-robin through upstream IPs. In this example, we connect to backends via HTTPS, so when building the peer object, we also need to specify use_tls and set SNI.

SNI ensures your service can handle multiple SSL certificates and domains on the same IP address

#[async_trait]
impl ProxyHttp for LB {

    /// In this simple example, we don't need context storage
    type CTX = ();
    fn new_ctx(&self) -> () {
        ()
    }

    async fn upstream_peer(&self, _session: &mut Session, _ctx: &mut ()) -> Result<Box<HttpPeer>> {
        let upstream = self.0
            .select(b"", 256) // Round robin doesn't depend on hash values
            .unwrap();

        println!("Upstream peer is: {upstream:?}");

        // Set SNI to runnable.run - SNI ensures your service can handle multiple SSL certificates and domains on the same IP
        let peer = Box::new(HttpPeer::new(upstream, true, "runnable.run".to_string()));
        Ok(peer)
    }

    async fn upstream_request_filter(
        &self,
        _session: &mut Session,
        upstream_request: &mut RequestHeader,
        _ctx: &mut Self::CTX,
    ) -> Result<()> {
        upstream_request.insert_header("Host", "runnable.run").unwrap();
        Ok(())
    }
}

For the runnable.run backend to accept our requests, the Host header must be present. We can add this header via the upstream_request_filter() callback, which can modify request headers after connecting to the backend but before sending them.

async fn upstream_request_filter(
        &self,
        _session: &mut Session,
        upstream_request: &mut RequestHeader,
        _ctx: &mut Self::CTX,
    ) -> Result<()> {
        upstream_request.insert_header("Host", "runnable.run").unwrap();
        Ok(())
    }

Creating the pingora-proxy Service

Next, let's create a proxy service based on the load balancer implementation above.

A Pingora service listens on one or more endpoints (TCP or Unix socket). pingora-proxy is an application that proxies HTTP requests to given backends according to the configuration.

In the example below, we create an LB instance with two backends ["testapi.runnable.run:443", "testapi2.runnable.run:443"]. We place the LB instance in a proxy service via http_proxy_service(), then tell the server to host that proxy service.

fn main() {
    let mut my_server = Server::new(None).unwrap();
    my_server.bootstrap();

    let upstreams = LoadBalancer::try_from_iter(["testapi.runnable.run:443", "testapi2.runnable.run:443"]).unwrap();

    let mut lb = http_proxy_service(&my_server.configuration, LB(Arc::new(upstreams)));
    lb.add_tcp("0.0.0.0:6188");

    my_server.add_service(lb);

    my_server.run_forever();
}

Complete Code:

use async_trait::async_trait;
use pingora::prelude::*;
use std::sync::Arc;

pub struct LB(Arc<LoadBalancer<RoundRobin>>);

#[async_trait]
impl ProxyHttp for LB {

    /// In this simple example, we don't need context storage
    type CTX = ();
    fn new_ctx(&self) -> () {
        ()
    }

    async fn upstream_peer(&self, _session: &mut Session, _ctx: &mut ()) -> Result<Box<HttpPeer>> {
        let upstream = self.0
            .select(b"", 256) // Round robin doesn't depend on hash values
            .unwrap();

        println!("Upstream peer is: {upstream:?}");

        // Set SNI to runnable.run - SNI ensures your service can handle multiple SSL certificates and domains on the same IP
        let peer = Box::new(HttpPeer::new(upstream, true, "runnable.run".to_string()));
        Ok(peer)
    }

    async fn upstream_request_filter(
        &self,
        _session: &mut Session,
        upstream_request: &mut RequestHeader,
        _ctx: &mut Self::CTX,
    ) -> Result<()> {
        upstream_request.insert_header("Host", "runnable.run").unwrap();
        Ok(())
    }
}



fn main() {
    let mut my_server = Server::new(None).unwrap();
    my_server.bootstrap();

    let upstreams = LoadBalancer::try_from_iter(["testapi.runnable.run:443", "testapi2.runnable.run:443"]).unwrap();

    let mut lb = http_proxy_service(&my_server.configuration, LB(Arc::new(upstreams)));
    lb.add_tcp("0.0.0.0:6188");

    my_server.add_service(lb);

    my_server.run_forever();
}

Running

cargo run

Test command

curl 127.0.0.1:6188 -svo /dev/null

This test case curl 127.0.0.1:6188 -svo /dev/null verifies whether the proxy service is working properly. Here's a breakdown:

curl: A command-line tool for making HTTP requests.
127.0.0.1:6188: The target address - the local proxy server (127.0.0.1 is localhost, 6188 is the port your proxy listens on).
-s: "Silent" mode - hides progress bar and error messages unless there's a serious error.
-v: "Verbose" mode - shows detailed request and response headers, including connection details, requests sent, responses received, redirects, etc.
-o /dev/null: Redirects output to /dev/null, discarding all response content. This focuses on request/response headers rather than actual page content.
-svo combined: Execute silently (no extra progress bar or error output), show detailed request/response headers in verbose mode, and discard response body.

This command lets you verify your proxy is handling requests correctly. The verbose output shows request headers sent to upstream, verifying if the Host header is correctly set to "one.one.one.one" and which upstream node (1.1.1.1 or 1.0.0.1) the proxy connected to.

Screenshot of successful request

Peer Health Checks

To make our load balancer more reliable, we want to add health checks for upstream peers. This way, if a peer fails, we can quickly stop routing traffic to it.

First, let's see how our load balancer behaves when one peer is down. We'll update the peer list to include a guaranteed-to-fail peer:

fn main() {
    // ...
    let mut upstreams =
        LoadBalancer::try_from_iter(["testapi.runnable.run:443", "testapi2.runnable.run:443", "127.0.0.1:343"]).unwrap();
    // ...
}

Now, if we run the load balancer again with cargo run and use

curl 127.0.0.1:6188 -svo /dev/null

We can see that every 3rd request fails with 502: Bad Gateway. This is because our peer selection strictly follows the RoundRobin pattern without considering peer health. We can fix this by adding a basic health check service.

fn main() {
    let mut my_server = Server::new(None).unwrap();
    my_server.bootstrap();

    // Note that upstreams now needs to be declared as `mut`
    let mut upstreams =
        LoadBalancer::try_from_iter(["testapi.runnable.run:443", "testapi2.runnable.run:443", "127.0.0.1:343"]).unwrap();

    let hc = TcpHealthCheck::new();
    upstreams.set_health_check(hc);
    upstreams.health_check_frequency = Some(std::time::Duration::from_secs(1));

    let background = background_service("health check", upstreams);
    let upstreams = background.task();

    // `upstreams` no longer needs to be wrapped in Arc
    let mut lb = http_proxy_service(&my_server.configuration, LB(upstreams));
    lb.add_tcp("0.0.0.0:6188");

    my_server.add_service(background);

    my_server.add_service(lb);
    my_server.run_forever();
}

As you can see, no requests go to 127.0.0.1, proving our health check is working

Preface​

Load Balancing Basics:​

Creating a Simple Load Balancing Proxy​

Create a New Rust Project and Add Dependencies​

Code Section​

Creating the pingora-proxy Service​

Complete Code:​

Running​

Peer Health Checks​