########################################################################################## FKAMP-2: Anti-Amp Script doesn't work on Google's AMP cache ########################################################################################## Issue Type: Bug ----------------------------------------------------------------------------------------- Issue Information ==================== Priority: Major Status: Closed Resolution: Fixed (2019-06-09 12:38:55) Project: Anti-AMP Scripts (FKAMP) Reported By: btasker Assigned To: btasker Affected Versions: - V1.1 - v1.2 - v1.3 Targeted for fix in version: - v1.4 - v1.4.1a - v1.4.1 Time Estimate: 0 minutes Time Logged: 0 minutes ----------------------------------------------------------------------------------------- Issue Description ================== Reported in Github #2 (https://github.com/bentasker/RemoveAMP/issues/2) The script doesn't rewrite/redirect on - https://www.google.com/amp/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/ - https://www.google.com/amp/s/www.theverge.com/platform/amp/2019/4/16/18402628/google-amp-url-problem-signed-exchange-original-chrome-cloudflare - https://www.google.com/amp/s/www.wired.com/2016/02/googles-amp-speeding-web-changing-works/winamp/ (Note that a mobile UA must be used, otherwise Google will just serve a redirect). Console shows that the script thinks it's not AMP (though that could be referring to some subresource) ----------------------------------------------------------------------------------------- Attachments ============ - Screenshot_20190510_230246.png - 57573041-742d9380-73f0-11e9-8346-1c85a23ee1ea.png - dodgy_amp.har ----------------------------------------------------------------------------------------- Issue Relations ================ - relates to FKAMP-1: Create functionality to block AMP pages - Github #2 (https://github.com/bentasker/RemoveAMP/issues/2) ----------------------------------------------------------------------------------------- Activity ========== ----------------------------------------------------------------------------------------- 2019-05-10 23:03:55 btasker ----------------------------------------------------------------------------------------- I can't really paste the full amount into JIRA because it's so messy, but here's what Google are actually serving when you hit that location (assuming they believe you're a mobile) -- BEGIN SNIPPET -- -- END SNIPPET -- The opening is followed by some CSS and then a whole load of obfuscated javascript (see screenshot). The important things of note though are - All that's one one line (yeuch) - Google aren't actually declaring the document as being AMP (which is why the detector isn't firing) - note the html tag ----------------------------------------------------------------------------------------- 2019-05-10 23:03:55 ----------------------------------------------------------------------------------------- btasker added 'Screenshot_20190510_230246.png' to Attachments ----------------------------------------------------------------------------------------- 2019-05-10 23:07:41 ----------------------------------------------------------------------------------------- btasker changed Project from 'STAGING' to 'Miscellaneous' ----------------------------------------------------------------------------------------- 2019-05-10 23:07:41 ----------------------------------------------------------------------------------------- btasker changed Key from 'STGNG-10' to 'MISC-29' ----------------------------------------------------------------------------------------- 2019-05-10 23:16:46 git ----------------------------------------------------------------------------------------- -- BEGIN QUOTE -- Repo: RemoveAMP Commit: 074b168c44777fb3ea7343afdab8caa31badfb99 Author: B Tasker > Date: Fri May 10 23:07:46 2019 +0100 Commit Message: MISC-29 Search for Canonical and trigger redirect on Google AMP cache pages (see Github #2) We need to do this because Google's cache doesn't properly declare that the resource is AMP. Modified (-)(+) ------- anti-amp.js -- END QUOTE -- *Webhook User-Agent* -- BEGIN SNIPPET -- GitHub-Hookshot/ae20175 -- END SNIPPET -- https://github.com/bentasker/RemoveAMP/commit/074b168c44777fb3ea7343afdab8caa31badfb99 ----------------------------------------------------------------------------------------- 2019-05-10 23:25:58 btasker ----------------------------------------------------------------------------------------- -- BEGIN QUOTE -- Repo: RemoveAMP Commit: 354f2c3a36f46a2006744be73ec69c3de206c03f Author: B Tasker > Date: Fri May 10 23:22:21 2019 +0100 Commit Message: Update Greasemonkey hook to reference v1.4 Means we should auto-roll out the changes made for MISC-27 Modified (-)(+) ------- greasemonkey_hook.user.js -- END QUOTE -- *Webhook User-Agent* -- BEGIN SNIPPET -- GitHub-Hookshot/ae20175 -- END SNIPPET -- https://github.com/bentasker/RemoveAMP/commit/354f2c3a36f46a2006744be73ec69c3de206c03f ----------------------------------------------------------------------------------------- 2019-05-11 09:52:44 btasker ----------------------------------------------------------------------------------------- You do get a couple of window repaints when going to the Google hosted cache though. Looking at logs it's because there are a couple of hops to make: - Start: https://www.google.com/amp/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/ - Go to listed canonical (still AMP): https://www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/ - Go to listed Canonical (proper HTML): https://www.theregister.co.uk/2017/05/19/open_source_insider_google_amp_bad_bad_bad/ ----------------------------------------------------------------------------------------- 2019-05-12 09:08:24 btasker ----------------------------------------------------------------------------------------- The user is reporting that they have to manually reload the page before they're redirected away from Google's AMP cache. This only happens when they visit via Google Search though. I've had no luck (user-agent changes, incognito + UA change etc) in getting Google to actually serve me AMP results so haven't been able to repro. ----------------------------------------------------------------------------------------- 2019-05-12 09:10:18 btasker ----------------------------------------------------------------------------------------- User's provided a console screenshot and also provided the outer-HTML they see (the iframe itself serves up the same content as when you go direct). The URL is the same as those mentioned earlier, but when the user goes there direct the redirect works. ----------------------------------------------------------------------------------------- 2019-05-12 09:10:18 ----------------------------------------------------------------------------------------- btasker added '57573041-742d9380-73f0-11e9-8346-1c85a23ee1ea.png' to Attachments ----------------------------------------------------------------------------------------- 2019-05-12 09:35:21 btasker ----------------------------------------------------------------------------------------- Now this is interesting, if not particularly helpful in answering the outstanding issue. I've now been able to get Google to serve me AMP results. But... the must be doing some kind of compatability or experience tracking on their side, because if you do the following: - Open Private Window in FF - Select UA that wouldn't get AMP results - Search "theregister AMP bad" - No AMPs - Change to UA that should, re-search You still get no AMP results. This is important because it seems that the default Iphone user-agent in the User-Agent switcher extensions, for whatever reason, doesn't get given AMP results. So when I later switched to Mobile/Chrome in there, I still didn't get the results even though I should have. Closing the private window, and then opening anew and starting off with the correct UA gets me AMP results. Unfortunately, the behaviour I'm getting still differs to the user. I may need to lay hands a Mac to try and repro with Safari. I don't see any obvious smoking gun in the console output - the call out to www-the-register-co-uk.cdn.ampproject.org is interesting (though you can't see the full querystring, so I can't get anything beyond a 404 back atm, but it's going to https://www-the-register-co-uk.cdn.ampproject.org/v/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/ which does make me wonder if that's the path that's actually being served from). It's a blind shot in the dark, but I'm tempted to get the user to run a copy of the script which adds that domain to the checker, just to see what the behaviour ends up being. Either way, we already know the initial change won't be sufficient on it's own, as both Cloudflare and Bing run amp caches too. The problem there, though, is they're rolling out "Real URL" support, so we won't be able to rely on the source hostname in future as they'll serve AMP from theregister.co.uk rather than (say) amp.cloudflare.com/c/theregister.co.uk/path. Though, to be fair, based on a very quick check, Cloudflare do seem to be honouring the spec and properly declaring AMP -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl -s https://amp-usatoday-com.amp.cloudflare.com/c/amp.usatoday.com/story/82055560/ -o /tmp/cf; head /tmp/cf | grep -o -P ']*' -- END SNIPPET -- But, ultimately, Google's page just results in a page which has an iframe in it referencing the actual amp cache. So, if we go to https://www.google.com/amp/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/ the iframe loads content from https://www-theregister-co-uk.cdn.ampproject.org/v/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/?usqp=mq331AQECAEoAQ%3D%3D&_js_v=0.1#origin=https://www.google.com&cid=1&prerenderSize=1&visibilityState=visible&paddingTop=32&history=1&p2r=0&horizontalScrolling=0&csi=1&storage=1&viewerUrl=https://www.google.com/amp/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/&cap=navigateTo,fragment,handshakepoll,cid,replaceUrl,fullReplaceHistory If you try and access that direct in a browser, you'll get a 404, need to make sure that _every_ request header is correct -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl 'https://www-theregister-co-uk.cdn.ampproject.org/v/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/?usqp=mq331AQECAEoAQ%3D%3D&_js_v=0.1#origin=https://www.google.com&cid=1&prerenderSize=1&visibilityState=visible&paddingTop=32&history=1&p2r=0&horizontalScrolling=0&csi=1&storage=1&viewerUrl=https://www.google.com/amp/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/&cap=navigateTo,fragment,handshakepoll,cid,replaceUrl,fullReplaceHistory' -H 'User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Referer: https://www.google.com/' -H 'DNT: 1' -H 'Connection: keep-alive' -H 'Upgrade-Insecure-Requests: 1' -H 'Pragma: no-cache' -H 'Cache-Control: no-cache' -o /tmp/amp -- END SNIPPET -- At which point, you get our source document -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ head /tmp/amp | wc -c 13669 -- END SNIPPET -- Neither of those pages properly declares themselves as being AMP. Now, if we try and detect the AMP hostname, that'll trigger from within the iframe, and so won't be able to redirect the browser. The bit I still can't fully answer though, is why the user's Safari isn't triggering as the wrapper domain is (or, at least, should be) correct for the check that was added earlier in this issue. Need to look at that screenshot again, there must be something I'm missing. ----------------------------------------------------------------------------------------- 2019-05-12 09:57:54 btasker ----------------------------------------------------------------------------------------- Actually, thinking about it, the user was originally trying to adjust so they could use the Anti-amp script without the require directive - https://github.com/bentasker/RemoveAMP/issues/1 (seems it's not supported on iOS) That directive was added because otherwise the anti-AMP protection wouldn't work on sites with moderately strict Content Security Policies. See here - https://projects.bentasker.co.uk/jira_projects/browse/MISC-25.html#comment2186467 There is a complaint about CSP in the user's screenshot, and the bit of the path that's visible would suggest that it's on Google's wrapper page. The line number's not particularly helpful there, of course as pretty much everything is on one line within the document. But, it seems reasonable to assume that Google wouldn't leave it's own cache fucked by it's CSP, so it's probably the result of injected code. I should be able to test this to reproduce by creating a new user script which injects the code *without* using require and then seeing whether I get a similar line in the console. Wouldn't explain why it then works following a refresh, but we can come to that later. ----------------------------------------------------------------------------------------- 2019-05-12 10:08:49 btasker ----------------------------------------------------------------------------------------- Ok, tested by disabling the anti-amp hook and creating a new userscript with this content - https://projectsstatic.bentasker.co.uk/MISC/MISC29/csp_test.user.js.txt The result isn't actually quite as expected. What I got was -- BEGIN SNIPPET -- Navigated to https://www.google.com/amp/s/www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/ eww vile... trying to redirect open_source_insider_google_amp_bad_bad_bad:5:9 Redirecting you to https://www.theregister.co.uk/AMP/2017/05/19/open_source_insider_google_amp_bad_bad_bad/ open_source_insider_google_amp_bad_bad_bad:13:17 Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). open_source_insider_google_amp_bad_bad_bad:1:1 Loading failed for the

Redirect Notice

Â The page you were on is trying to send you to an invalid URL.

Â If you do not want to visit that page, you can return to the previous page.

-- END SNIPPET -- Makes sense really otherwise they'd have people bouncing victims of their redirect page onto other locations. OK, will leave as is for now ----------------------------------------------------------------------------------------- 2019-05-15 16:04:24 btasker ----------------------------------------------------------------------------------------- Attaching the HAR for posterities sake ----------------------------------------------------------------------------------------- 2019-05-15 16:04:24 ----------------------------------------------------------------------------------------- btasker added 'dodgy_amp.har' to Attachments ----------------------------------------------------------------------------------------- 2019-05-15 16:22:46 git ----------------------------------------------------------------------------------------- -- BEGIN QUOTE -- Repo: RemoveAMP Commit: 217d5e8ff8150eb1988d2dbb4837f0f65a4df696 Author: Ben Tasker > Date: Wed May 15 16:20:14 2019 +0100 Commit Message: Revert "MISC-29 Add slightly snarky and very temporary iframe detection to try and work around AMP detection issues on Safari when hitting Google's cache (see #2)" This reverts commit 51400177ebdc778364be15fc06f2cc3b6c3629e3. Further testing has identified the cause of the issues in Safari/Google Search, and this detection won't help with that. Might be useful in the long run, but as it was thrown together quite quickly I don't feel it's been adequately tested so should be removed for now Modified (-)(+) ------- anti-amp.js -- END QUOTE -- *Webhook User-Agent* -- BEGIN SNIPPET -- GitHub-Hookshot/ae20175 -- END SNIPPET -- https://github.com/bentasker/RemoveAMP/commit/217d5e8ff8150eb1988d2dbb4837f0f65a4df696 ----------------------------------------------------------------------------------------- 2019-05-15 16:27:36 btasker ----------------------------------------------------------------------------------------- OK, user has confirmed this has fixed the issue for them. I've rolled this as v1.4.1 - https://github.com/bentasker/RemoveAMP/releases/tag/v1.4.1 Marking as fixed ----------------------------------------------------------------------------------------- 2019-05-15 16:27:41 ----------------------------------------------------------------------------------------- btasker changed status from 'Open' to 'Resolved' ----------------------------------------------------------------------------------------- 2019-05-15 16:27:41 ----------------------------------------------------------------------------------------- btasker added 'Fixed' to resolution ----------------------------------------------------------------------------------------- 2019-05-15 16:27:45 ----------------------------------------------------------------------------------------- btasker changed status from 'Resolved' to 'Closed' ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker changed Project from 'Miscellaneous' to 'Anti-AMP Scripts' ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker changed Key from 'MISC-29' to 'FKAMP-2' ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker added 'V1.1' to Version ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker added 'v1.2' to Version ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker added 'v1.3' to Version ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker removed 'Google AMP' from Version ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker added 'v1.4' to Fix Version ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker added 'v1.4.1a' to Fix Version ----------------------------------------------------------------------------------------- 2019-06-09 12:36:43 ----------------------------------------------------------------------------------------- btasker removed 'Google AMP' from Fix Version ----------------------------------------------------------------------------------------- 2019-06-09 12:38:36 btasker ----------------------------------------------------------------------------------------- I've created a new JIRA project to track development of these scripts, so - MISC-25 becomes FKAMP-1 - MISC-29 becomes FKAMP-2 - MISC-31 becomes FKAMP-3 ----------------------------------------------------------------------------------------- 2019-06-09 12:38:44 ----------------------------------------------------------------------------------------- btasker removed 'Fixed' from resolution ----------------------------------------------------------------------------------------- 2019-06-09 12:38:44 ----------------------------------------------------------------------------------------- btasker changed status from 'Closed' to 'Reopened' ----------------------------------------------------------------------------------------- 2019-06-09 12:38:51 ----------------------------------------------------------------------------------------- btasker added 'v1.4.1' to Fix Version ----------------------------------------------------------------------------------------- 2019-06-09 12:38:55 ----------------------------------------------------------------------------------------- btasker changed status from 'Reopened' to 'Resolved' ----------------------------------------------------------------------------------------- 2019-06-09 12:38:55 ----------------------------------------------------------------------------------------- btasker added 'Fixed' to resolution ----------------------------------------------------------------------------------------- 2019-06-09 12:39:00 ----------------------------------------------------------------------------------------- btasker changed status from 'Resolved' to 'Closed'