########################################################################################## FKAMP-4: Redirect Away from Google News ########################################################################################## Issue Type: Bug ----------------------------------------------------------------------------------------- Issue Information ==================== Priority: Major Status: Closed Resolution: Done (2019-06-11 17:40:32) Project: Anti-AMP Scripts (FKAMP) Reported By: btasker Assigned To: btasker Environment:Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0 Affected Versions: - v1.4.1 Targeted for fix in version: - v1.4.21 Time Estimate: 0 minutes Time Logged: 0 minutes ----------------------------------------------------------------------------------------- Issue Description ================== Original description: -- BEGIN QUOTE -- Much like the issues described in FKAMP-2, we still get AMP pages served (from Google's cache) when visiting news stories linked to from new.google.co.uk. Using "Request Desktop Site" resolves that to some extent, but Google seem to have opted not to use responsive design on Google News and instead rely on user-agent sniffing. So you wind up with a page that's formatted for a widescreen monitor. Means, half the time, important images are missing from the news stories and they're rendered otherwise pointless. Need to look at whether the existing Google search centric script is directly applicable to Google News, or, failing that, at implementing a new script to get rid of AMP. -- END QUOTE -- Google News is horrendously AMPy, but just like in their search results isn't declared as AMP in the markup. Unlike in their search results, they're also not using the canonical correctly and simply declare themselves as being canonical (so you can't even auto-follow to a proper page). This issue implements a new user script redirect_to_non_amp_sites.user.js to handle similarly hostile pages by redirecting known bad domains to another domain that provides similar functionality without the AMP baggage. In this case, that's redirecting Google News to Bing news. ----------------------------------------------------------------------------------------- Issue Relations ================ - relates to FKAMP-5: See if we can find a way to handle AMP on Google News ----------------------------------------------------------------------------------------- Activity ========== ----------------------------------------------------------------------------------------- 2019-06-09 13:02:10 btasker ----------------------------------------------------------------------------------------- Unfortunately it doesn't look like the FKAMP-2 is directly applicable here. On news.google.co.uk the outbound link is the same whether you're using an AMP targetted device or not: AMPy U-A -- BEGIN SNIPPET -- -- END SNIPPET -- Default UA -- BEGIN SNIPPET -- -- END SNIPPET -- Although that page will show an AMP icon if AMP is considered acceptable for your device, the redirection to AMP content happens further down the line - If we look at the result of hitting those URLs both with and without a mobile UA the result is much the same: -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl -v -H "User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0" "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB%3Aen" * Trying 216.58.201.163... * Connected to news.google.co.uk (216.58.201.163) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: EC * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com * start date: Tue, 21 May 2019 20:43:22 GMT * expire date: Tue, 13 Aug 2019 20:31:00 GMT * issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3 * compression: NULL * ALPN, server accepted to use http/1.1 > GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB%3Aen HTTP/1.1 > Host: news.google.co.uk > Accept: */* > User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0 > < HTTP/1.1 302 Found < Content-Type: application/binary < Cache-Control: no-cache, no-store, max-age=0, must-revalidate < Pragma: no-cache < Expires: Mon, 01 Jan 1990 00:00:00 GMT < Date: Sun, 09 Jun 2019 11:58:09 GMT < Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Content-Security-Policy: script-src 'nonce-NbfEafbenlZpZwvUdQfKTA' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self' < Content-Security-Policy: script-src 'nonce-NbfEafbenlZpZwvUdQfKTA' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport < Server: ESF < Content-Length: 0 < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < Set-Cookie: NID=185=h7EgTSqYfXVI55DuTNbCIvJa_fr8KvKunItrVe8_hu1iMIe8o6f-r7xUZfUfPNS4vsn9PdKaeAwDKZpjAj1nbWA2daylnjDPjo4KU4-nlUn6M1mScdNXo_B6x59JMf9sr2hwgctOqmcLdmdJW6uv76Bm_ls3J7pRjsQ-lKOhmQs;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 11:58:09 GMT;HttpOnly < Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39" < * Connection #0 to host news.google.co.uk left intact # And without ben@milleniumfalcon:~$ curl -v "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB%3Aen" * Trying 216.58.201.163... * Connected to news.google.co.uk (216.58.201.163) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: EC * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com * start date: Tue, 21 May 2019 20:43:22 GMT * expire date: Tue, 13 Aug 2019 20:31:00 GMT * issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3 * compression: NULL * ALPN, server accepted to use http/1.1 > GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB%3Aen HTTP/1.1 > Host: news.google.co.uk > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 302 Found < Content-Type: application/binary < Cache-Control: no-cache, no-store, max-age=0, must-revalidate < Pragma: no-cache < Expires: Mon, 01 Jan 1990 00:00:00 GMT < Date: Sun, 09 Jun 2019 11:58:31 GMT < Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Content-Security-Policy: script-src 'nonce-g86r229GvsnrEvRuPelBrg' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self' < Server: ESF < Content-Length: 0 < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < Set-Cookie: NID=185=FBtal_PV0_1tN_ORYC4rEjeJ_1Q0U-b08GAmsIjW8o8FLRqEkrEEtQhmA7douMdFj06_OaSdYe2bC2HUUZf7BQIBuezY12fJVrq8NEoWuHGzQ6yfio_g0wf9gUDOqIe5_DtS-o8oOVLamcR661LzZdYw71WbxvW4JsFuKSczNNQ;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 11:58:31 GMT;HttpOnly < Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39" < * Connection #0 to host news.google.co.uk left intact -- END SNIPPET -- If we follow the redirects, then one results in AMP content being served from new.google.co.uk: -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl -v -H "User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0" "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB%3Aen" -L -o /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 216.58.201.163... * Connected to news.google.co.uk (216.58.201.163) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: EC * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com * start date: Tue, 21 May 2019 20:43:22 GMT * expire date: Tue, 13 Aug 2019 20:31:00 GMT * issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3 * compression: NULL * ALPN, server accepted to use http/1.1 > GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB%3Aen HTTP/1.1 > Host: news.google.co.uk > Accept: */* > User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0 > < HTTP/1.1 302 Found < Content-Type: application/binary < Cache-Control: no-cache, no-store, max-age=0, must-revalidate < Pragma: no-cache < Expires: Mon, 01 Jan 1990 00:00:00 GMT < Date: Sun, 09 Jun 2019 12:00:20 GMT < Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Content-Security-Policy: script-src 'nonce-aRhxgVfjXSIt+iuMTgm+sA' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self' < Content-Security-Policy: script-src 'nonce-aRhxgVfjXSIt+iuMTgm+sA' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport < Server: ESF < Content-Length: 0 < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < Set-Cookie: NID=185=2_doVDioENw_-bLjk5rMNX8_3p7ahtQb8b_JS6VlgPf6XVXzZVVoa2WtlAI_nWNNVyC0HrijmmdxQcMByS4Nqw59lPtPt39FbEgHwsrmKvj73m5YhMDV0dHTyywbhqLb6BbrUAQYpZmUyWuxbGaI7iqJeFUSB6jlsLY-7kbc3GA;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 12:00:20 GMT;HttpOnly < Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39" < 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Connection #0 to host news.google.co.uk left intact * Issue another request to this URL: 'https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en' * Trying 216.58.213.14... * Connected to news.google.com (216.58.213.14) port 443 (#1) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.news.google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.news.google.com * start date: Tue, 21 May 2019 20:43:51 GMT * expire date: Tue, 13 Aug 2019 20:31:00 GMT * issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3 * compression: NULL * ALPN, server accepted to use http/1.1 > GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en HTTP/1.1 > Host: news.google.com > Accept: */* > User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0 > < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 < x-ua-compatible: IE=edge < Cache-Control: no-cache, no-store, max-age=0, must-revalidate < Pragma: no-cache < Expires: Mon, 01 Jan 1990 00:00:00 GMT < Date: Sun, 09 Jun 2019 12:00:21 GMT < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Strict-Transport-Security: max-age=31536000 < Content-Security-Policy: script-src 'nonce-j9hhCrnH2e5Qu+ZSfzzjBw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self' < Content-Security-Policy: script-src 'nonce-j9hhCrnH2e5Qu+ZSfzzjBw' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport < Server: ESF < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < Set-Cookie: NID=185=klWjj78MtMd5XM_m1vyGmweF5f20rVbjg6WKLOdAAlA1I8dXEI8LfxWIQz99OVvve6w0WOtPWylikpReid8ElB1YIxKnay2svTBu8ThDrVT42rxibbMY46oSp1k8SpudnHhyFp3zQLzdQyNyarkITT6LqR--TPdFyaNhOPMfaL0;Domain=.google.com;Path=/;Expires=Mon, 09-Dec-2019 12:00:21 GMT;HttpOnly < Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39" < Accept-Ranges: none < Vary: Accept-Encoding < Transfer-Encoding: chunked < { [1433 bytes data] 100 380k 0 380k 0 0 481k 0 --:--:-- --:--:-- --:--:-- 481k -- END SNIPPET -- The other results in us ultimately hitting the actual news site -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl -v "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB%3Aen" -L -o /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 216.58.201.163... * Connected to news.google.co.uk (216.58.201.163) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: EC * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com * start date: Tue, 21 May 2019 20:43:22 GMT * expire date: Tue, 13 Aug 2019 20:31:00 GMT * issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3 * compression: NULL * ALPN, server accepted to use http/1.1 > GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB%3Aen HTTP/1.1 > Host: news.google.co.uk > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 302 Found < Content-Type: application/binary < Cache-Control: no-cache, no-store, max-age=0, must-revalidate < Pragma: no-cache < Expires: Mon, 01 Jan 1990 00:00:00 GMT < Date: Sun, 09 Jun 2019 12:01:30 GMT < Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Content-Security-Policy: script-src 'nonce-F/V/daD8mV2ECVVBMREhzw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self' < Server: ESF < Content-Length: 0 < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < Set-Cookie: NID=185=4b9SjV97th3JUT-Q3wyi0L69Pq0H4rliKXXlvF1aFv_ZG1f24foi_SnM3VWdYxYpYTQ8qlcBgMBD_MDv0fSVyuZQwEC1nLD32lKNi3lYYCqBqrOWSVeDMRg43uVWnM4dMgWOd7yY5tpD-oafBdASRbm9WtjeuYBstVqLZpJUhs4;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 12:01:30 GMT;HttpOnly < Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39" < 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Connection #0 to host news.google.co.uk left intact * Issue another request to this URL: 'https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en' * Trying 216.58.213.14... * Connected to news.google.com (216.58.213.14) port 443 (#1) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.news.google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.news.google.com * start date: Tue, 21 May 2019 20:43:51 GMT * expire date: Tue, 13 Aug 2019 20:31:00 GMT * issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3 * compression: NULL * ALPN, server accepted to use http/1.1 > GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?amp;gl=GB&ceid=GB:en&hl=en-GB&gl=GB&ceid=GB:en HTTP/1.1 > Host: news.google.com > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 301 Moved Permanently < Content-Type: application/binary < Cache-Control: no-cache, no-store, max-age=0, must-revalidate < Pragma: no-cache < Expires: Mon, 01 Jan 1990 00:00:00 GMT < Date: Sun, 09 Jun 2019 12:01:31 GMT < Location: https://www.dailyrecord.co.uk/news/scottish-news/scots-mum-killed-lightning-bolt-16487470 < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Strict-Transport-Security: max-age=31536000 < Content-Security-Policy: script-src 'nonce-VDmUwigB885r2a6kE5x82g' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self' < Server: ESF < Content-Length: 0 < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < Set-Cookie: NID=185=hMKgo__6JLMGlwNSHrZSo6R2GcRPpK9MhuC8Eg1NqMa0BSiPsvjSqbGPi3v6I1iWv3vlBLkatiK2U8owsjeIv9Tiq8lYuYf45MBjBFVogXEmZCzTzNNbW2PUpvSS3rXQ50J4B9Hzz_Um-OhHoPO8_ePlH1kZLek9BXxXS7PyTlU;Domain=.google.com;Path=/;Expires=Mon, 09-Dec-2019 12:01:31 GMT;HttpOnly < Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39" < 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Connection #1 to host news.google.com left intact * Issue another request to this URL: 'https://www.dailyrecord.co.uk/news/scottish-news/scots-mum-killed-lightning-bolt-16487470' * Trying 104.82.77.144... * Connected to www.dailyrecord.co.uk (104.82.77.144) port 443 (#2) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384 * server certificate verification OK * server certificate status verification SKIPPED * common name: www.mirror.co.uk (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: C=GB,L=London,O=Trinity Mirror LTD,CN=www.mirror.co.uk * start date: Wed, 01 May 2019 00:00:00 GMT * expire date: Thu, 30 Jul 2020 12:00:00 GMT * issuer: C=US,O=DigiCert Inc,CN=DigiCert SHA2 Secure Server CA * compression: NULL * ALPN, server accepted to use http/1.1 > GET /news/scottish-news/scots-mum-killed-lightning-bolt-16487470 HTTP/1.1 > Host: www.dailyrecord.co.uk > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 OK < Access-Control-Allow-Origin: * < Content-Type: text/html;charset=UTF-8 < Server: nginx < X-Cacheable: YES < X-ProcessESI: yes < X-RemovedCookies: YES < X-Served-By: nat-cache2a.tm-aws.com < X-Varnish: 2130134286 < Cache-Control: max-age=421 < Expires: Sun, 09 Jun 2019 12:08:32 GMT < Date: Sun, 09 Jun 2019 12:01:31 GMT < Transfer-Encoding: chunked < Connection: keep-alive < Connection: Transfer-Encoding < Set-Cookie: GS_GROUP=1; expires=Tue, 09-Jul-2019 12:01:31 GMT < Set-Cookie: GS_REVENUE_LOC=1; expires=Tue, 09-Jul-2019 12:01:31 GMT < Content-Security-Policy-Report-Only: default-src ms-appx-web: data: blob: webviewprogressproxy: ws: wss: https: 'unsafe-inline' 'unsafe-eval'; font-src data: https: blob:; img-src http: https: data: blob: android-webview-video-poster: android-webview:; script-src blob: https: data: 'unsafe-inline' 'unsafe-eval'; media-src data: https: blob:; report-uri https://felix.data.tm-awx.com/cspReport < { [15442 bytes data] 100 222k 0 222k 0 0 209k 0 --:--:-- 0:00:01 --:--:-- 1130k * Connection #2 to host www.dailyrecord.co.uk left intact -- END SNIPPET -- ----------------------------------------------------------------------------------------- 2019-06-09 13:05:39 btasker ----------------------------------------------------------------------------------------- As we saw in FKAMP-2, Google are not declaring their AMP as being AMP: -- BEGIN SNIPPET -- -- END SNIPPET -- Worse, they are not actually providing an accurate canonical link -- BEGIN SNIPPET -- -- END SNIPPET -- Following that link just sends us round the merry-go-round again as Google still redirect us back to the AMP content: -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl -v -H "User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0" "https://news.google.co.uk/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU" -L -o /dev/null % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 216.58.208.99... * Connected to news.google.co.uk (216.58.208.99) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: EC * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.google.com * start date: Tue, 21 May 2019 20:43:22 GMT * expire date: Tue, 13 Aug 2019 20:31:00 GMT * issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3 * compression: NULL * ALPN, server accepted to use http/1.1 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0> GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU HTTP/1.1 > Host: news.google.co.uk > Accept: */* > User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0 > < HTTP/1.1 302 Found < Content-Type: application/binary < Cache-Control: no-cache, no-store, max-age=0, must-revalidate < Pragma: no-cache < Expires: Mon, 01 Jan 1990 00:00:00 GMT < Date: Sun, 09 Jun 2019 12:04:50 GMT < Location: https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB:en < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Content-Security-Policy: script-src 'nonce-gi1AbTumDhUcLYT2rvlxyw' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self' < Content-Security-Policy: script-src 'nonce-gi1AbTumDhUcLYT2rvlxyw' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport < Server: ESF < Content-Length: 0 < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < Set-Cookie: NID=185=jzRipStd1I9qUz43xuHXrCMO0hB3Wwb64Yb5tVR5z4QYvoLa--V9tmMTAQMNpIF0A40io4joG5_pjEhaVQeCH97VckBhbOY-1wMLYGglrNz11xutEU51dvTgO9_Y1zTP-NKu5qpK_beeFE0MLqioH_HgUBFXgYU56JBRofTUKwo;Domain=.google.co.uk;Path=/;Expires=Mon, 09-Dec-2019 12:04:50 GMT;HttpOnly < Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39" < 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Connection #0 to host news.google.co.uk left intact * Issue another request to this URL: 'https://news.google.com/articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB:en' * Trying 216.58.213.14... * Connected to news.google.com (216.58.213.14) port 443 (#1) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.news.google.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: C=US,ST=California,L=Mountain View,O=Google LLC,CN=*.news.google.com * start date: Tue, 21 May 2019 20:43:51 GMT * expire date: Tue, 13 Aug 2019 20:31:00 GMT * issuer: C=US,O=Google Trust Services,CN=Google Internet Authority G3 * compression: NULL * ALPN, server accepted to use http/1.1 > GET /articles/CAIiEFWHlmXqDRAqOCoJm_OEznEqGQgEKhAIACoHCAowrueiCTDmn7gCMIbTtwU?hl=en-GB&gl=GB&ceid=GB:en HTTP/1.1 > Host: news.google.com > Accept: */* > User-Agent: Mozilla/5.0 (Android 8.0.0; Mobile; rv:67.0) Gecko/67.0 Firefox/67.0 > < HTTP/1.1 200 OK < Content-Type: text/html; charset=utf-8 < x-ua-compatible: IE=edge < Cache-Control: no-cache, no-store, max-age=0, must-revalidate < Pragma: no-cache < Expires: Mon, 01 Jan 1990 00:00:00 GMT < Date: Sun, 09 Jun 2019 12:04:50 GMT < P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." < Strict-Transport-Security: max-age=31536000 < Content-Security-Policy: script-src 'nonce-15oeMA3A6tRGY5mQPJSejg' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/DotsSplashUi/cspreport;worker-src 'self' < Content-Security-Policy: script-src 'nonce-15oeMA3A6tRGY5mQPJSejg' 'self' 'unsafe-eval' https://apis.google.com https://ssl.gstatic.com https://www.google.com https://www.gstatic.com https://youtube.com https://www.youtube.com https://youtube.googleapis.com https://*.ytimg.com https://www.google-analytics.com/analytics.js https://www.googleapis.com/appsmarket/v2/installedApps/;report-uri /_/DotsSplashUi/cspreport < Server: ESF < X-XSS-Protection: 0 < X-Frame-Options: SAMEORIGIN < X-Content-Type-Options: nosniff < Set-Cookie: NID=185=qtK1bewKdiUWvf7iJ9VV27D_FX4HFRzLlL3SIHGxgXYv0p9hFl1Hc7AukPl3KyOO4UOxhSz7C_24AJ9E6RuxB6RpCZo4AWT-bHMHMv4hJgJoRP10WObbVCOpavdQ8TStnmG6FSX5-1agF1D4Dzv1j6COWITYSiCpIT_45XY42JA;Domain=.google.com;Path=/;Expires=Mon, 09-Dec-2019 12:04:50 GMT;HttpOnly < Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39" < Accept-Ranges: none < Vary: Accept-Encoding < Transfer-Encoding: chunked < { [1433 bytes data] 100 380k 0 380k 0 0 643k 0 --:--:-- --:--:-- --:--:-- 643k * Connection #1 to host news.google.com left intact -- END SNIPPET -- ----------------------------------------------------------------------------------------- 2019-06-09 13:13:43 btasker ----------------------------------------------------------------------------------------- However, it looks like they *are* making AMP specific changes on news.google.com If you load the page with a non-AMP UA, then switch the UA to be a mobile one and click the link, you end up being served a proper page rather than the AMP one. Looking at it with Developer Tools, just like in FKAMP-2, what Google are actually doing is rewriting the page with Javascript and then updating the address bar. ----------------------------------------------------------------------------------------- 2019-06-09 13:33:35 btasker ----------------------------------------------------------------------------------------- Things were far less obfuscated in FKAMP-2 though, Google really seem to have gone out of their way on the news site to try and make it an impenetrable wall of bollocks. I don't say this nearly enough, *fuck Google* and what they've become. So, actually looking at the source for news.google.co.uk as requested with that user-agent, they're already feeding you their half-breed HTML, which is probably why it looks a lot like it's been obfuscated by someone on crack. To be honest, at this point I'm a little inclined to do the unthinkable and just have the script redirect Google News to Bing News (which doesn't currently have the same issue). Never thought I'd find myself actually preferring anything Bing to Google's offering.... ----------------------------------------------------------------------------------------- 2019-06-09 14:05:53 btasker ----------------------------------------------------------------------------------------- -- BEGIN QUOTE -- Repo: RemoveAMP Commit: 5a127c52a58797f40bf8183725ceedec5204b372 Author: B Tasker