FKAMP-4: Redirect Away from Google News

Issue Information

Issue Type: Bug
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: Anti-AMP Scripts (FKAMP)
Resolution: Done (2019-06-11 17:40:32)
Affects Version: v1.4.1,
Target version: v1.4.21,
Environment: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0

Created: 2019-06-09 12:49:47
Time Spent Working


Description
Original description:
Much like the issues described in FKAMP-2, we still get AMP pages served (from Google's cache) when visiting news stories linked to from new.google.co.uk. Using "Request Desktop Site" resolves that to some extent, but Google seem to have opted not to use responsive design on Google News and instead rely on user-agent sniffing. So you wind up with a page that's formatted for a widescreen monitor.

Means, half the time, important images are missing from the news stories and they're rendered otherwise pointless.

Need to look at whether the existing Google search centric script is directly applicable to Google News, or, failing that, at implementing a new script to get rid of AMP.


Google News is horrendously AMPy, but just like in their search results isn't declared as AMP in the markup. Unlike in their search results, they're also not using the canonical correctly and simply declare themselves as being canonical (so you can't even auto-follow to a proper page).

This issue implements a new user script redirect_to_non_amp_sites.user.js to handle similarly hostile pages by redirecting known bad domains to another domain that provides similar functionality without the AMP baggage. In this case, that's redirecting Google News to Bing news.


Issue Links

relates to FKAMP-5: See if we can find a way to handle AMP on Google News
Toggle State Changes

Activity


2019-06-09 13:02:10
Unfortunately it doesn't look like the FKAMP-2 is directly applicable here.

On news.google.co.uk the outbound link is the same whether you're using an AMP targetted device or not:

AMPy U-A 
<a class="VDXfz" href="./articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen" jsaction="click:ksu5Sd" data-n-u="./articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen" tabindex="-1" aria-hidden="true"></a>


Default UA 
<a class="VDXfz" href="./articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen" tabindex="-1" target="_blank" aria-hidden="true"></a>


Although that page will show an AMP icon if AMP is considered acceptable for your device, the redirection to AMP content happens further down the line - If we look at the result of hitting those URLs both with and without a mobile UA the result is much the same: 

ben@milleniumfalcon:~$ curl -v -H "User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0" "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen"
*   Trying 216.58.201.163...
* Connected to news.google.co.uk (216.58.201.163) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.google.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: EC
* 	 certificate version: #3
* 	 subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com
* 	 start date: Tue, 21 May 2019 20:43:22 GMT
* 	 expire date: Tue, 13 Aug 2019 20:31:00 GMT
* 	 issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen HTTP/1.1
> Host: news.google.co.uk
> Accept: */*
> User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0
> 
< HTTP/1.1 302 Found
< Content-Type: application/binary
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Date: Sun, 09 Jun 2019 11:58:09 GMT
< Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&amp;ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Content-Security-Policy: script-src 'nonce-NbfEafbenlZpZwvUdQfKTA' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self'
< Content-Security-Policy: script-src 'nonce-NbfEafbenlZpZwvUdQfKTA' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport
< Server: ESF
< Content-Length: 0
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: NID=185=h7EgTSqYfXVI55DuTNbCIvJa_fr8KvKunItrVe8_hu1iMIe8o6f-r7xUZfUfPNS4vsn9PdKaeAwDKZpjAj1nbWA2daylnjDPjo4KU4-nlUn6M1mScdNXo_B6x59JMf9sr2hwgctOqmcLdmdJW6uv76Bm_ls3J7pRjsQ-lKOhmQs;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 11:58:09 GMT;HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< 
* Connection #0 to host news.google.co.uk left intact

# And without

ben@milleniumfalcon:~$ curl -v "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen"
*   Trying 216.58.201.163...
* Connected to news.google.co.uk (216.58.201.163) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.google.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: EC
* 	 certificate version: #3
* 	 subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com
* 	 start date: Tue, 21 May 2019 20:43:22 GMT
* 	 expire date: Tue, 13 Aug 2019 20:31:00 GMT
* 	 issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen HTTP/1.1
> Host: news.google.co.uk
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Content-Type: application/binary
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Date: Sun, 09 Jun 2019 11:58:31 GMT
< Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&amp;ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Content-Security-Policy: script-src 'nonce-g86r229GvsnrEvRuPelBrg' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self'
< Server: ESF
< Content-Length: 0
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: NID=185=FBtal_PV0_1tN_ORYC4rEjeJ_1Q0U-b08GAmsIjW8o8FLRqEkrEEtQhmA7douMdFj06_OaSdYe2bC2HUUZf7BQIBuezY12fJVrq8NEoWuHGzQ6yfio_g0wf9gUDOqIe5_DtS-o8oOVLamcR661LzZdYw71WbxvW4JsFuKSczNNQ;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 11:58:31 GMT;HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< 
* Connection #0 to host news.google.co.uk left intact


If we follow the redirects, then one results in AMP content being served from new.google.co.uk: 
ben@milleniumfalcon:~$ curl -v -H "User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0" "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen" -L -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 216.58.201.163...
* Connected to news.google.co.uk (216.58.201.163) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.google.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: EC
* 	 certificate version: #3
* 	 subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com
* 	 start date: Tue, 21 May 2019 20:43:22 GMT
* 	 expire date: Tue, 13 Aug 2019 20:31:00 GMT
* 	 issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen HTTP/1.1
> Host: news.google.co.uk
> Accept: */*
> User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0
> 
< HTTP/1.1 302 Found
< Content-Type: application/binary
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Date: Sun, 09 Jun 2019 12:00:20 GMT
< Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&amp;ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Content-Security-Policy: script-src 'nonce-aRhxgVfjXSIt+iuMTgm+sA' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self'
< Content-Security-Policy: script-src 'nonce-aRhxgVfjXSIt+iuMTgm+sA' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport
< Server: ESF
< Content-Length: 0
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: NID=185=2_doVDioENw_-bLjk5rMNX8_3p7ahtQb8b_JS6VlgPf6XVXzZVVoa2WtlAI_nWNNVyC0HrijmmdxQcMByS4Nqw59lPtPt39FbEgHwsrmKvj73m5YhMDV0dHTyywbhqLb6BbrUAQYpZmUyWuxbGaI7iqJeFUSB6jlsLY-7kbc3GA;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 12:00:20 GMT;HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host news.google.co.uk left intact
* Issue another request to this URL: 'https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&amp;ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en'
*   Trying 216.58.213.14...
* Connected to news.google.com (216.58.213.14) port 443 (#1)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.news.google.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.news.google.com
* 	 start date: Tue, 21 May 2019 20:43:51 GMT
* 	 expire date: Tue, 13 Aug 2019 20:31:00 GMT
* 	 issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&amp;ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en HTTP/1.1
> Host: news.google.com
> Accept: */*
> User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0
> 
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< x-ua-compatible: IE=edge
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Date: Sun, 09 Jun 2019 12:00:21 GMT
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Strict-Transport-Security: max-age=31536000
< Content-Security-Policy: script-src 'nonce-j9hhCrnH2e5Qu+ZSfzzjBw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self'
< Content-Security-Policy: script-src 'nonce-j9hhCrnH2e5Qu+ZSfzzjBw' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport
< Server: ESF
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: NID=185=klWjj78MtMd5XM_m1vyGmweF5f20rVbjg6WKLOdAAlA1I8dXEI8LfxWIQz99OVvve6w0WOtPWylikpReid8ElB1YIxKnay2svTBu8ThDrVT42rxibbMY46oSp1k8SpudnHhyFp3zQLzdQyNyarkITT6LqR--TPdFyaNhOPMfaL0;Domain=.google.com;Path=/;Expires=Mon, 09-Dec-2019 12:00:21 GMT;HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< 
{ [1433 bytes data]
100  380k    0  380k    0     0   481k      0 --:--:-- --:--:-- --:--:--  481k


The other results in us ultimately hitting the actual news site 
ben@milleniumfalcon:~$ curl -v "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen" -L -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 216.58.201.163...
* Connected to news.google.co.uk (216.58.201.163) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.google.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: EC
* 	 certificate version: #3
* 	 subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com
* 	 start date: Tue, 21 May 2019 20:43:22 GMT
* 	 expire date: Tue, 13 Aug 2019 20:31:00 GMT
* 	 issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&amp;gl=GB&amp;ceid=GB%3Aen HTTP/1.1
> Host: news.google.co.uk
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 302 Found
< Content-Type: application/binary
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Date: Sun, 09 Jun 2019 12:01:30 GMT
< Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&amp;ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Content-Security-Policy: script-src 'nonce-F/V/daD8mV2ECVVBMREhzw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self'
< Server: ESF
< Content-Length: 0
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: NID=185=4b9SjV97th3JUT-Q3wyi0L69Pq0H4rliKXXlvF1aFv_ZG1f24foi_SnM3VWdYxYpYTQ8qlcBgMBD_MDv0fSVyuZQwEC1nLD32lKNi3lYYCqBqrOWSVeDMRg43uVWnM4dMgWOd7yY5tpD-oafBdASRbm9WtjeuYBstVqLZpJUhs4;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 12:01:30 GMT;HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host news.google.co.uk left intact
* Issue another request to this URL: 'https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&amp;ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en'
*   Trying 216.58.213.14...
* Connected to news.google.com (216.58.213.14) port 443 (#1)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.news.google.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.news.google.com
* 	 start date: Tue, 21 May 2019 20:43:51 GMT
* 	 expire date: Tue, 13 Aug 2019 20:31:00 GMT
* 	 issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&amp;ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en HTTP/1.1
> Host: news.google.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Type: application/binary
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Date: Sun, 09 Jun 2019 12:01:31 GMT
< Location: https://www.dailyrecord.co.uk/news/scottish-news/scots-mum-killed-lightning-bolt-16487470
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Strict-Transport-Security: max-age=31536000
< Content-Security-Policy: script-src 'nonce-VDmUwigB885r2a6kE5x82g' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self'
< Server: ESF
< Content-Length: 0
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: NID=185=hMKgo__6JLMGlwNSHrZSo6R2GcRPpK9MhuC8Eg1NqMa0BSiPsvjSqbGPi3v6I1iWv3vlBLkatiK2U8owsjeIv9Tiq8lYuYf45MBjBFVogXEmZCzTzNNbW2PUpvSS3rXQ50J4B9Hzz_Um-OhHoPO8_ePlH1kZLek9BXxXS7PyTlU;Domain=.google.com;Path=/;Expires=Mon, 09-Dec-2019 12:01:31 GMT;HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #1 to host news.google.com left intact
* Issue another request to this URL: 'https://www.dailyrecord.co.uk/news/scottish-news/scots-mum-killed-lightning-bolt-16487470'
*   Trying 104.82.77.144...
* Connected to www.dailyrecord.co.uk (104.82.77.144) port 443 (#2)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: www.mirror.co.uk (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: C=GB,L=London,O=Trinity Mirror LTD,CN=www.mirror.co.uk
* 	 start date: Wed, 01 May 2019 00:00:00 GMT
* 	 expire date: Thu, 30 Jul 2020 12:00:00 GMT
* 	 issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /news/scottish-news/scots-mum-killed-lightning-bolt-16487470 HTTP/1.1
> Host: www.dailyrecord.co.uk
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Access-Control-Allow-Origin: *
< Content-Type: text/html;charset=UTF-8
< Server: nginx
< X-Cacheable: YES
< X-ProcessESI: yes
< X-RemovedCookies: YES
< X-Served-By: nat-cache2a.tm-aws.com
< X-Varnish: 2130134286
< Cache-Control: max-age=421
< Expires: Sun, 09 Jun 2019 12:08:32 GMT
< Date: Sun, 09 Jun 2019 12:01:31 GMT
< Transfer-Encoding:  chunked
< Connection: keep-alive
< Connection: Transfer-Encoding
< Set-Cookie: GS_GROUP=1; expires=Tue, 09-Jul-2019 12:01:31 GMT
< Set-Cookie: GS_REVENUE_LOC=1; expires=Tue, 09-Jul-2019 12:01:31 GMT
< Content-Security-Policy-Report-Only: default-src ms-appx-web: data: blob: webviewprogressproxy: ws: wss: https: 'unsafe-inline' 'unsafe-eval'; font-src data: https: blob:; img-src http: https: data: blob: android-webview-video-poster: android-webview:; script-src blob: https: data: 'unsafe-inline' 'unsafe-eval'; media-src data: https: blob:; report-uri https://felix.data.tm-awx.com/cspReport
< 
{ [15442 bytes data]
100  222k    0  222k    0     0   209k      0 --:--:--  0:00:01 --:--:-- 1130k
* Connection #2 to host www.dailyrecord.co.uk left intact

2019-06-09 13:05:39
As we saw in FKAMP-2, Google are not declaring their AMP as being AMP: 
<!doctype html><html lang="en" dir="ltr"><head><base href="https://news.google.com/"><meta name="referrer" content="origin">


Worse, they are not actually providing an accurate canonical link 
<link rel="canonical" href="https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU">


Following that link just sends us round the merry-go-round again as Google still redirect us back to the AMP content: 
ben@milleniumfalcon:~$ curl -v -H "User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0" "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU" -L -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 216.58.208.99...
* Connected to news.google.co.uk (216.58.208.99) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.google.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: EC
* 	 certificate version: #3
* 	 subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com
* 	 start date: Tue, 21 May 2019 20:43:22 GMT
* 	 expire date: Tue, 13 Aug 2019 20:31:00 GMT
* 	 issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU HTTP/1.1
> Host: news.google.co.uk
> Accept: */*
> User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0
> 
< HTTP/1.1 302 Found
< Content-Type: application/binary
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Date: Sun, 09 Jun 2019 12:04:50 GMT
< Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB:en
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Content-Security-Policy: script-src 'nonce-gi1AbTumDhUcLYT2rvlxyw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self'
< Content-Security-Policy: script-src 'nonce-gi1AbTumDhUcLYT2rvlxyw' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport
< Server: ESF
< Content-Length: 0
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: NID=185=jzRipStd1I9qUz43xuHXrCMO0hB3Wwb64Yb5tVR5z4QYvoLa--V9tmMTAQMNpIF0A40io4joG5_pjEhaVQeCH97VckBhbOY-1wMLYGglrNz11xutEU51dvTgO9_Y1zTP-NKu5qpK_beeFE0MLqioH_HgUBFXgYU56JBRofTUKwo;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 12:04:50 GMT;HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host news.google.co.uk left intact
* Issue another request to this URL: 'https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB:en'
*   Trying 216.58.213.14...
* Connected to news.google.com (216.58.213.14) port 443 (#1)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.news.google.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.news.google.com
* 	 start date: Tue, 21 May 2019 20:43:51 GMT
* 	 expire date: Tue, 13 Aug 2019 20:31:00 GMT
* 	 issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB:en HTTP/1.1
> Host: news.google.com
> Accept: */*
> User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0
> 
< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< x-ua-compatible: IE=edge
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: Mon, 01 Jan 1990 00:00:00 GMT
< Date: Sun, 09 Jun 2019 12:04:50 GMT
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Strict-Transport-Security: max-age=31536000
< Content-Security-Policy: script-src 'nonce-15oeMA3A6tRGY5mQPJSejg' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self'
< Content-Security-Policy: script-src 'nonce-15oeMA3A6tRGY5mQPJSejg' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport
< Server: ESF
< X-XSS-Protection: 0
< X-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Set-Cookie: NID=185=qtK1bewKdiUWvf7iJ9VV27D_FX4HFRzLlL3SIHGxgXYv0p9hFl1Hc7AukPl3KyOO4UOxhSz7C_24AJ9E6RuxB6RpCZo4AWT-bHMHMv4hJgJoRP10WObbVCOpavdQ8TStnmG6FSX5-1agF1D4Dzv1j6COWITYSiCpIT_45XY42JA;Domain=.google.com;Path=/;Expires=Mon, 09-Dec-2019 12:04:50 GMT;HttpOnly
< Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< 
{ [1433 bytes data]
100  380k    0  380k    0     0   643k      0 --:--:-- --:--:-- --:--:--  643k
* Connection #1 to host news.google.com left intact


2019-06-09 13:13:43
However, it looks like they are making AMP specific changes on news.google.com

If you load the page with a non-AMP UA, then switch the UA to be a mobile one and click the link, you end up being served a proper page rather than the AMP one.

Looking at it with Developer Tools, just like in FKAMP-2, what Google are actually doing is rewriting the page with Javascript and then updating the address bar.

2019-06-09 13:33:35
Things were far less obfuscated in FKAMP-2 though, Google really seem to have gone out of their way on the news site to try and make it an impenetrable wall of bollocks. I don't say this nearly enough, fuck Google and what they've become.

So, actually looking at the source for news.google.co.uk as requested with that user-agent, they're already feeding you their half-breed HTML, which is probably why it looks a lot like it's been obfuscated by someone on crack.

To be honest, at this point I'm a little inclined to do the unthinkable and just have the script redirect Google News to Bing News (which doesn't currently have the same issue). Never thought I'd find myself actually preferring anything Bing to Google's offering....

2019-06-09 14:05:53

Repo: RemoveAMP
Commit: 5a127c52a58797f40bf8183725ceedec5204b372
Author: B Tasker <github@<Domain Hidden>>

Date: Sun Jun 09 13:54:00 2019 +0100
Commit Message: FKAMP-3 Add script to redirect certain sites/domains to non-AMP enforcing alternatives

This is primarily for things like Google News where it's non-trivial to un-AMP them.

This current version redirects Google News to Bing News (which isn't something I ever thought I'd find myself willingly doing)

The intention is that this will only ever be used for domains that use AMP but don't declare it, or don't use canonical properly. Google are currently the only ones I've seen doing that so far



Added (+)
-------
redirect_to_non_amp_sites.user.js




Webhook User-Agent 

GitHub-Hookshot/d408d22


View Commit

2019-06-09 14:10:46

Repo: RemoveAMP
Commit: dd908449ac7df045bf9fe02f6600fd38ecb41e3a
Author: B Tasker <github@<Domain Hidden>>

Date: Sun Jun 09 14:10:17 2019 +0100
Commit Message: FKAMP-4 Update README



Modified (-)(+)
-------
README.md




Webhook User-Agent 

GitHub-Hookshot/d408d22


View Commit


2019-06-11 17:40:32
btasker changed status from 'Open' to 'Resolved'

2019-06-11 17:40:32
btasker added 'Done' to resolution

2019-06-11 17:40:37
btasker changed status from 'Resolved' to 'Closed'