REST caching pattern: A not_before parameter

Just an idea at this stage. Could be adopted by any RSS/Atom/status type platform.

Background

I've previously mentioned how we're caching content using what I'm calling the "1+N caching pattern". On each sync:

* There's one top-level call to fetch the very latest, never cached, content. In Player FM's case, that's basically a user JSON, with URLs to each topic they're interested in. (This is actually very efficient when combined with HTTP freshness and an intermediary caching layer, ie CloudFlare. Most times, the request makes it only as far as CloudFlare, not my server, and the response is a 304 with empty body. It wil only be a 200 with body if any of the topics have updated since last time.)

* The subsequent responses - calls for each topic - are cached forever. That's because their URLs include a timestamp parameter, and that's the URL that's included in the top-level call mentioned above. So this is very efficient as these calls are all served by the intermediary cache (CloudFlare) except for the very first client to request it. (Again, HTTP freshness means each client will most of the time be receiving empty bodies. They only receive a full response if the topic has been updated.)

Opportunity and Related Patterns

So this is all pretty efficient, but there's some redundancy here. When a single change happens to a topic, the whole topic body has to be re-downloaded again. Sure, that only happens once for any topic change, but it's still redundancy.

So I was thinking, how could the response just include the diff.

Something like Cloudflare's Railgun solution (https://www.cloudflare.com/railgun), but server-to-client instead of server-to-proxy.

This is also something like the HTTP freshness principle, which lets the client tell the server what they already have and lets the server decide if that needs to be updated. But HTTP freshness is binary - you either get the update or you don't. This is a sweet-spot middle ground.

It's can also be seen as an inverse HTTP PATCH. The HTTP patch protocol (http://www.mnot.net/blog/2012/09/05/patch) lets the client upload a change; here, we're effectively send a patch in the opposite direction.

Solution

The client sends a not_before parameter. The server treats it as a filter and sends only things that have changed.

AND one other important thing. not_before must be a multiple of 600,000, i.e. 10 minutes (the exact duration is of course a decision the server developer can make). If it's not, the server rejects it with a 412 Precondition Failed error. And it must also be a time in the past. So clients should 

This constraint is important for caching. It means all clients looking for "baseball" will ask for the "baseball since 3:40pm" resource instead of one asking for the "baseball since 3:41pm", another coming along a few seconds later and asking for "baseball since 3:42pm", etc. In other words, the cached resource is shared between all clients and with an intermediary layer, it only reaches the server every 10 minutes.

Notes

Why is the parameter called not_before? What's wrong with after or since?

Yes, normally I avoid negative variable names, but in this case it's justified. after and since imply the server remembers everything in the past and will send those items immediately following that date. Whereas we still want the very latest content; we're just adding a filter to say we don't need that older content. This is the same thing if responses are unlimited, but in practice, they should be limited, so "the very latest" trumps "the ones that happened immediately after this date".

e.g. If we're in June and we ask for "baseball since January 1", we should get the most recent items, not necessarily those that happened just after Jan 1, which "after" or "since" might imply.

Why are you always trying to complicate things. KISS.

This parameter could, and should, be entirely optional. The client can still ask for "baseball", but if they want to turbo-charge performance, they can and should add the extra parameter.
Shared publiclyView activity