JILS-42: Last-Modified isn't updated if a Project Description changes

Looking at the NGinx configuration options, it seems to imply that If-None-Match is supported.

My testbox (nginx/1.8.0) definitely isn't using it, and the NGinx caching guide (https://www.nginx.com/blog/nginx-caching-guide/) says If-Modified-Since will be used.

So it's possible If-None-Match support is more recent. Alternatively - Etags aren't currently enabled on that proxy, so perhaps they need to be turned on for it to consider etags when going upstream?

Got sidetracked...


Simply enabling etags isn't sufficient, as by default NGinx is using HTTP/1.0 to go upstream

GET /browse/EXAMPLE-1.html HTTP/1.0

E-tags don't exist in 1.0, so need to tell NGinx to use 1.1 when going to the origin

proxy_http_version 1.1;

But even then they don't seem to be used. I'm probabyl missing a config option somewhere, will have a look at the code-base to try and see what (and when it was introduced)

Looks like If-None-Match support was introduced in 1.7.3 (see NGinx ticket 101), so should be available on on my install.

Looking at the commits against that issue (primarily https://trac.nginx.org/nginx/changeset/c95d7882dfc9ff90ee161328747114b6b54667a5/nginx ) it looks like the only config variable taken into account is proxy_cache_revalidate, which is already set.

The check's pretty simple:

static ngx_int_t
ngx_http_upstream_cache_etag(ngx_http_request_t *r,
    ngx_http_variable_value_t *v, uintptr_t data)
{
    if (r->upstream == NULL
        || !r->upstream->conf->cache_revalidate
        || r->upstream->cache_status != NGX_HTTP_CACHE_EXPIRED
        || r->cache->etag.len == 0)
    {
        v->not_found = 1;
        return NGX_OK;
    }
 
    v->valid = 1;
    v->no_cacheable = 0;
    v->not_found = 0;
    v->len = r->cache->etag.len;
    v->data = r->cache->etag.data;

    return NGX_OK;
}

That moment where you realise you're being a twatspanner on the public internet. I'm returning an E-Tag header not an ETag header, so of course NGinx isn't trying to use it.

Despite the fix, NGinx is still going upstream with If-Modified-Since instead of If-None-Match. I'm sure I'm probably missing something here though.

I've just double-checked the cached item on disk and the headers are definitely present and correct

# egrep -a "etag|Last-Mod" /usr/share/nginx/cache/static/d/5b/65d69cf4d601773f304231cdb736a5bd
Last-Modified: Thu, 12 Mar 2015 06:24:14 GMT
etag: is-12709-deda25ecae31b80de016e60bed193286bf93fd54

Just to rule out the obvious I switched the ordering of the headers, but no dice. Not that ordering should matter anyway

Updating to Nginx 1.8.1 hasn't helped either.

Doesn't seem to be anything wrong with the header, Chrome is happy to use If-None-Match when revalidating a cached copy of the page. So there's either something wrong with my NGinx config, or NGinx itself.

Looking at the most recent source - https://trac.nginx.org/nginx/browser/nginx/src/http/ngx_http_upstream.c#L5372 - it's not taking any new configuration variables into account when deciding whether an ETag can be used for revalidation.

After running a few tests, it seems to relate to the length of the ETag. I swapped the SHA1 for a MD5 and NGinx now presents an If-None-Match header when attempting to revalidate. Shouldn't matter as ETags are of type Entity-Tag, which according to the spec can be any length (though I had forgotten to quote them).

As the page type, and issue ID are in the clear, only need to worry about collisions within a page, so I think it should be safe to swap everything over to MD5. Will work through and do that now (and insert the errant quotes)

It definitely seems to have to do with the length of the ETag. I originally updated the ETag generation for the issue page to

$etag='"is-'.$issue->ID."-".md5($changes->maxcreate.$comments->maxcreate).'"';

and that still wasn't sufficiently short for NGinx to present an If-None-Match.

Shortening to

$etag='"is-'.$issue->ID."-".md5($changes->maxcreate.$comments->maxcreate).'"';

worked but would mean NGinx would stop using ETags for revalidation if an issues internal ID were higher than 999999. Whilst that's a fairly large JIRA install, it's not necessarily huge, so I've updated the etag generation to

$etag='"i'.$issue->ID."-".md5($changes->maxcreate.$comments->maxcreate).'"';

So, it should work for up to 99,999,999 issues. If someone's got more than that number of issues, it's going to fall back to IMS, but there's not much can be done about that other than looking at how to adjust NGinx to remove the limitation (seems to be 40 bytes/320 bits)

I've made a similar change to all the other etags (will commit in a moment) to get those working as well, and revalidation does now seem to be using If-None-Match

Obviously it's still worth trying to think of a way to identify a Last-Modified date for Project/Version/Component pages, but NGinx will now correctly use If-None-Match, so that reduces the impact a little.

So, to summarise

Although it'll accept and serve longer ETags returned from the origin, for revalidation purposes, NGinx appears to limit the length of ETags to 40 bytes. Any longer and it won't include an If-None-Match header when sending a conditional request upstream (instead falling back on If-Modified-Since).

In order for NGinx to use If-None-Match, upstream connections need to be HTTP/1.1 (because etags didn't exist in 1.0), so the following must be set (whether globally - in the http block, on a per host basis - in the server block, or within the location being proxied - in the location block)

proxy_http_version 1.1;

There's no need to explicitly enable ETags with the etag directive (though you might want to enable it for other reasons).