MISC-15: Theoretical: Productisation of a CDN as a service

The very first issue that would need to be handled is distribution of keys across the edge.

If (as a customer) I hold the private key for foo.onion, handing that key over to a third party so that it can be distributed across the edge would be a huge risk.

So, an alternative solution would be to use Tor's "subdomain" behaviour.

The CDN's edge would have a fixed private key (for example, resulting in bar.onion). As the operator of foo.onion, I'd have my static content (e.g. images) referenced as being hosted on foo.bar.onion. The CDN would know from the subdomain which origin to proxy onto, and an appropriate cache-key would be used to keep my cached content distinct from other users/customers.

The subdomain itself wouldn't actually need to specify the origin to proxy to, as that could lead to abuse of the CDN's proxy capability (for example, by trying to send requests to www.google.com.bar.onion). It could equally well be of the format user123456.bar.onion or even abcdefg.bar.onion

Updating configuration when a new user was added could also be problematic with a large edge. One solution would be to have the edge proxy every cache_miss onto the midtier, regardless of the Host header received.

Configuration specific to customers would then be added at the midtier. So the mid-tier would then, in effect, also become an authentication layer.

So, if a request was received at the edge for www.google.com.bar.onion, the edge would pass the request upstream. Unless the mid-tier caches have a server block specifically for www.google.com.bar.onion the request would result in a 403.

Which might give config like the following

Edge

server {
    listen       127.0.0.1:80;
    server_name  *.foo.onion;
    root /usr/share/nginx/onions/empty;

    resolver 127.0.0.1;

    # Proxy to the back-end
    location / {
        set $cachename "Edge1";
        proxy_set_header X-DOWNSTREAM  $cachename;

        # Make sure the host header is correct
        proxy_set_header Host $host;

        # Send the request
        proxy_pass   http://cix7cricsvweeu6k.onion:8091;

        # Enable Keep-Aluve
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Allow revalidations
        proxy_cache_revalidate on;

        # Allow request pipe-lining
        proxy_cache_lock on;
        proxy_cache streamingcache;
        proxy_cache_key "$scheme$host$request_uri";
        add_header X-Cache-Status "$cachename-$upstream_cache_status";

    }

}

Midtier

server {
    listen       127.0.0.1:80;
    server_name  user1234.foo.onion;
    root /usr/share/nginx/onions/empty;

    resolver 127.0.0.1;

    # Proxy to the back-end
    location / {
        set $cachename "midtier1";
        proxy_set_header X-DOWNSTREAM  $cachename;

        # Make sure the host header is correct
        proxy_set_header Host "cix7cricsvweeu6k.onion";

        # Send the request
        proxy_pass   http://origin1234.onion:8091;

        # Enable Keep-Aluve
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Allow revalidations
        proxy_cache_revalidate on;

        # Allow request pipe-lining
        proxy_cache_lock on;
        proxy_cache streamingcache;
        proxy_cache_key "$scheme$host$request_uri";
        add_header X-Cache-Status "$cachename-$upstream_cache_status";

    }

}

server {
    listen      127.0.0.1:80 default_server;
    server_name invalid.foo.onion;

    error_page 403 /custom_403.html;
    location = /custom_403.html {
	    root /usr/share/nginx/html;
	    internal;
    }

    location / {
	    deny all;
    }
}

So in that example, example.foo.onion would be passed to the midtier, but would be denied. Only requests with a host header of user1234.foo.onion would be acceptable.

The problem with that setup, though, is that it would become quite easy to effectively DDoS the mid-tier by spreading "invalid" requests across the edge. So there'd need to be some consideration given to infrastructure to mitigate this risk (probably a requirement for a larger midtier).

An alternative route might be to use the NGinx LUA module to build an authentication system, or to use something like Ansible to roll configuration out to the edge.

Marking this as related to MISC-18 as some of what has been considered here will probably tie into that issue.

Marking this as resolved as I haven't touched it in ages.

For completeness though - ultimately, I went with HTTP redirect based load balancing, using the HTTP router component of my RequestRouter Geo routing solution.

Each node in the "CDN" has a dedicated onion address, and the router selects one of the nodes and generates a redirect to it's onion address. Works fine for video delivery and/or static content, though you obviously wouldn't want to use it for an entire onion site (as the address in the address bar would change, and people would bookmark the wrong URL etc).

For user-facing URLs, the existing balancing mechanism (using the descriptor race) still seems to be the best solution. Possibly paired with having a traditional load balancer behind each so that you can have pools of origins in multiple locations