########################################################################################## MISC-37: Kodi can't update library with TheTVDB ########################################################################################## Issue Type: Bug ----------------------------------------------------------------------------------------- Issue Information ==================== Priority: Major Status: Closed Resolution: Fixed (2020-01-24 10:26:06) Project: Miscellaneous (MISC) Reported By: btasker Assigned To: btasker Environment:Kodi 18.5 Kodi 17.1 TVDB Media Provider 3.2.3 Labels: Kodi, TVDB, Time Estimate: 0 minutes Time Logged: 0 minutes ----------------------------------------------------------------------------------------- Issue Description ================== Kodi's update library won't currently add any TV series. Media info source is thetvdb.com Have verified that newly added directories are properly named to rule out it being that it just can't find them in the DB. ----------------------------------------------------------------------------------------- Issue Relations ================ - Kodi Forum (https://forum.kodi.tv/showthread.php?tid=349385&page=13) ----------------------------------------------------------------------------------------- Activity ========== ----------------------------------------------------------------------------------------- 2020-01-23 10:37:29 btasker ----------------------------------------------------------------------------------------- This is consistent between Kodi boxes running different versions of the software (18.5 and 17.1 confirmed), so it doesn't seem to be something that's been recently broken within Kodi Looking the kodi log, we get -- BEGIN SNIPPET -- 2020-01-23 10:31:04.676 T:1649124064 ERROR: Run: Unable to parse web site 2020-01-23 10:31:04.690 T:1649124064 ERROR: Previous line repeats 1 times. 2020-01-23 10:31:04.690 T:1649124064 WARNING: No information found for item 'http://series.kat.home:80/Magic%5FKnight%5FRayearth/', it won't be added to the library. 2020-01-23 10:31:05.233 T:1649124064 WARNING: Skipping item 'http://series.kat.home:80/manual/' with '.nomedia' file in parent directory, it won't be added to the library. 2020-01-23 10:31:05.300 T:1649124064 ERROR: Run: Unable to parse web site 2020-01-23 10:31:05.315 T:1649124064 ERROR: Previous line repeats 1 times. 2020-01-23 10:31:05.315 T:1649124064 WARNING: No information found for item 'http://series.kat.home:80/Manyuu%5FHiken%2DChou/', it won't be added to the library. -- END SNIPPET -- Need to look into it more, but I'm guess the TVDB's API has changed ----------------------------------------------------------------------------------------- 2020-01-23 11:20:21 btasker ----------------------------------------------------------------------------------------- I edited .kodi/addons/metadata.tvdb.com/tvdb.xml to use HTTP URLs for the TVDB API calls so I could more easily intercept and extract creds for troubleshooting. Each of the calls results in a 302 to HTTPS so we don't get to see responses, but it's enough to allow me to recreate the flow with curl Login flow works fine -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl -v -H "content-type: application/json" -d '{"apikey":"[KEY]"}' https://api.thetvdb.com/login * Trying 13.35.178.200... * Connected to api.thetvdb.com (13.35.178.200) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.thetvdb.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: CN=*.thetvdb.com * start date: Thu, 23 May 2019 00:00:00 GMT * expire date: Tue, 23 Jun 2020 12:00:00 GMT * issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon * compression: NULL * ALPN, server accepted to use http/1.1 > POST /login HTTP/1.1 > Host: api.thetvdb.com > User-Agent: curl/7.47.0 > Accept: */* > content-type: application/json > Content-Length: 29 > * upload completely sent off: 29 out of 29 bytes < HTTP/1.1 200 OK < Content-Type: application/json; charset=utf-8 < Content-Length: 462 < Connection: keep-alive < Date: Thu, 23 Jan 2020 11:12:07 GMT < Vary: Accept-Language < X-Powered-By: Thundar! < X-Thetvdb-Api-Version: 3.0.0 < X-Cache: Miss from cloudfront < Via: 1.1 5ebef4b6eba4f6ee211859e315b4fa16.cloudfront.net (CloudFront) < X-Amz-Cf-Pop: FJR50-C1 < X-Amz-Cf-Id: 9eNAEmMHbTtgXo9uyhk6Jpxzn-05vhZNphEeHRm3qaUk7Cnbbn_ApQ== < * Connection #0 to host api.thetvdb.com left intact {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI3MjcsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3OTI3fQ.3KbEW1oQq1J8eTIv1QEiFC3ebhN15jUKwF1G2A4er32B4wce_b1VRAUc_9_qgQXzxt0JI5sp1xuNeU7mW0aoT6WDiDA6RRtljdCJvNhDAeUsJ_i4jgkBEuY8kckNM7MX63-vgPRXm1AK4tC2ANGy-h0po1JGBD4hcLz0QnrFrdgZ50XcybVgiCDcHU2FRFiqwTcBKlkt3F0qsoSylURXCZDssJXEySZ1JjFZ1FwzE_1TgaMh1kB2eb08VYHaENcovAgekzpnXK1eSc7Qw0E6ErqINgMTm1gK8ml5Arsu91hak_DsP3P4cNAf0L7RRvD7_3sxzvTjMRssbxfHZqQSQQ"} -- END SNIPPET -- Can see Kodi searching later, and it too obviously got a token back as it's provided in the auth header -- BEGIN SNIPPET -- GET /search/series?name=Magic%20Knight%20Rayearth HTTP/1.1 Range: bytes=0- User-Agent: OSMC (Linux; OSMC 2017.03-1) Host: api.thetvdb.com Accept: */* Accept-Charset: UTF-8,*;q=0.8 Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A accept-language: en -- END SNIPPET -- If we place that request with curl -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl -v -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A" -H "accept-language: en" "https://api.thetvdb.com/search/series?name=Magic%20Knight%20Rayearth" * Trying 99.86.241.252... * Connected to api.thetvdb.com (99.86.241.252) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.thetvdb.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: CN=*.thetvdb.com * start date: Thu, 23 May 2019 00:00:00 GMT * expire date: Tue, 23 Jun 2020 12:00:00 GMT * issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon * compression: NULL * ALPN, server accepted to use http/1.1 > GET /search/series?name=Magic%20Knight%20Rayearth HTTP/1.1 > Host: api.thetvdb.com > User-Agent: curl/7.47.0 > Accept: */* > Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A > accept-language: en > < HTTP/1.1 200 OK < Content-Type: application/json; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < Date: Thu, 23 Jan 2020 11:14:30 GMT < Cache-Control: private, max-age=86400 < X-Powered-By: Thundar! < X-Thetvdb-Api-Version: 3.0.0 < Vary: Accept-Language < X-Cache: Miss from cloudfront < Via: 1.1 b48fca327a980187d93a198e7530195c.cloudfront.net (CloudFront) < X-Amz-Cf-Pop: VIE50-C1 < X-Amz-Cf-Id: pLlRFaHSkAkQAViOz73GsOTZJv51BO8sUXeLD6BdJsSuEtTjyxUP5A== < {"data":[{"aliases":["Una porta socchiusa ai confini del sole"],"banner":"/banners/graphical/71634-g3.jpg","firstAired":"1994-6-2","id":71634,"image":"/banners/posters/71634-5.jpg","network":"YTV (JP)","overview":"Three young girls, Hikaru, Umi, and Fuu, are transported to a magical world called Cephiro during a field trip to Tokyo Tower. They are soon greeted by Master Mage Clef, who explains to them that they have been summoned to become the Legendary Magic Knights and save Cephiro. The girls are less than enthusiastic about this idea, and only want to return home. Clef further explains that they must seek out the three Rune Gods to help them fight. He bestows armor and magical powers to each of them. They learn from Clef that High Priest Zagato has kidnapped the Pillar of Cephiro, Princess Emeraude. The Pillar of Cephiro has the sole responsibility of keeping Cephiro alive and in balance with her prayers. Without Princess Emeraude, Cephiro would fall into ruin. Hikaru, Umi, and Fuu must fight off Zagato's henchman and find the Rune Gods if they ever want to get back home. They soon learn that friendship and loyalty are the only things they can rely on in the crumbling Cephiro.","poster":"/banners/posters/71634-5.jpg","seriesName":"Magic Knight Rayearth","slug":"magic-knight-rayearth","status":"Ended"},{"aliases":["Magic Knight Rayearth OVA"],"banner":null,"firstAired":"1997-7-25","id":358557,"image":"/banners/series/358557/posters/62034069.jpg","network":null,"overview":"The Earth lies darkened and deserted. Despite the best efforts of Hikaru, the evil-filled castle of Cephiro has taken over the world and stands in the centre of a lifeless Tokyo. Before the ice-witch Alcyone can take her revenge on Hikaru though, she is rescued by Lantis, a young man who seals her into a magical maze where she is challenged to prove her powers while her friends are hunted down by Cephiro's warlocks.\r\n\r\nFleeing through the silent city, Umi and Fuu must seek out the sleeping Mashin — godlike creatures that will fuse with t* Connection #0 to host api.thetvdb.com left intact hem to form giant warriors of awesome destructive power. But the Magic Knights are afraid, untrained and alone... will even the Mashin be enough to save them from the evil acts of Cephiro?","poster":null,"seriesName":"Rayearth (1997)","slug":"rayearth","status":"Ended"}]} -- END SNIPPET -- we get JSON back. passing it through python -m json.tool confirms it's valid. So, Kodi is ostensibly getting data back but complaining it's unable to parse it ----------------------------------------------------------------------------------------- 2020-01-23 11:21:54 btasker ----------------------------------------------------------------------------------------- There are complaints about this on the Kodi forum too. One of the devs has complained -- BEGIN QUOTE -- We cannot diagnose from 3 lines of a log. Please provide the full log. -- END QUOTE -- It's not _quite_ worth the hassle of creating an account to point out that if they want more loglines maybe they should make their application log more than Unable to parse website by default. ----------------------------------------------------------------------------------------- 2020-01-23 11:26:49 btasker ----------------------------------------------------------------------------------------- Unfortunately the error message Unable to parse website sits in the middle of a binary - kodi.bin - media addons are essentially just XML files containing a bunch of regexes. Lets get debug logging turned on on a Kodi box and see if that yields anything extra. My guess is going to be that the TVDB's JSON has changed slightly - based on previous Kodi bugs I've seen I don't have a massive amount of faith in some of their implementation. ----------------------------------------------------------------------------------------- 2020-01-23 11:33:40 btasker ----------------------------------------------------------------------------------------- -- BEGIN SNIPPET -- 11:28:01.504 T:1549005808 DEBUG: FindMovie: Searching for 'Magic_Knight_Rayearth' using The TVDB scraper (path: '/home/osmc/.kodi/addons/metadata.tvdb.com', content: 'tvshows', version: '3.2.3') 11:28:01.504 T:1549005808 DEBUG: scraper: CreateSearchUrl returned http://api.thetvdb.com/login?{"apikey":"439DFEBA9D3059C6"}|Content-Type=application/json 11:28:01.507 T:1549005808 DEBUG: scraper: GetSearchResults returned http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth|Authorization=Bearer%20eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODM2NzQsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc4ODc0fQ.buhlqlAGCeGazN2Jq9rMgj9Wv5xgT1Zha8qmVSVlKmA0wSgpgkRcjPc9lcy36xFA71fzGKPDlleMTy7xp5zwo07-Dd9B0hLog1k_Gd-2NjvM4PDWbZHjYF5rH0OavotO6wBsYANLJCmJCgL7xbscaHax1BWcTHAKIaxvwBUjMDG6Ni6lmS-7Tuz5yPuu6QGAoQQfCKo5CDfugu2uZDm-faJ7B9T-hJaerbb_hLk7UJI31jO0Y2bs_eQR7oK9BbBn6sMXhSPiUvX_O3cRpZ6VhiKik-KwDrScESvcjWAcFozM4wsnFNtmOvpPNdVpVWxlvUddJrFt3ZbfLEwE1r33kQ&accept-language=en 11:28:01.507 T:1549005808 DEBUG: CurlFile::ParseAndCorrectUrl() adding custom header option 'Authorization: ***********' 11:28:01.507 T:1549005808 DEBUG: CurlFile::ParseAndCorrectUrl() adding custom header option 'accept-language: en' 11:28:01.508 T:1549005808 DEBUG: CurlFile::Open(0x727e76e0) http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth 11:28:01.936 T:1549005808 DEBUG: CCurlFile::Open - effective URL: 11:28:01.937 T:1549005808 DEBUG: Get: Using "UTF-8" charset for "http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth|Authorization=Bearer%20eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODM2NzQsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc4ODc0fQ.buhlqlAGCeGazN2Jq9rMgj9Wv5xgT1Zha8qmVSVlKmA0wSgpgkRcjPc9lcy36xFA71fzGKPDlleMTy7xp5zwo07-Dd9B0hLog1k_Gd-2NjvM4PDWbZHjYF5rH0OavotO6wBsYANLJCmJCgL7xbscaHax1BWcTHAKIaxvwBUjMDG6Ni6lmS-7Tuz5yPuu6QGAoQQfCKo5CDfugu2uZDm-faJ7B9T-hJaerbb_hLk7UJI31jO0Y2bs_eQR7oK9BbBn6sMXhSPiUvX_O3cRpZ6VhiKik-KwDrScESvcjWAcFozM4wsnFNtmOvpPNdVpVWxlvUddJrFt3ZbfLEwE1r33kQ&accept-language=en" 11:28:01.942 T:1549005808 DEBUG: could not parse input XML 11:28:01.943 T:1549005808 DEBUG: XSLT: xslt transformation failed 11:28:01.944 T:1549005808 ERROR: Run: Unable to parse web site 11:28:01.945 T:1549005808 WARNING: No information found for item 'http://series.kat.home:80/Magic%5FKnight%5FRayearth/', it won't be added to the library. -- END SNIPPET -- could not parse input XML.... sigh, is it getting JSON back when it expects XML? On the other hand, it's correctly submitting auth creds using JSON, which'd suggest the API has always been JSON (quick scan of https://api.thetvdb.com/swagger would support that theory), so is being passed to the wrong parser at Kodi's end? ----------------------------------------------------------------------------------------- 2020-01-23 11:42:00 btasker ----------------------------------------------------------------------------------------- Content-Type is correct in the response headers -- BEGIN SNIPPET -- > GET /search/series?name=Magic%20Knight%20Rayearth HTTP/1.1 > Host: api.thetvdb.com > User-Agent: curl/7.47.0 > Accept: */* > Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A > accept-language: en > < HTTP/1.1 200 OK < Content-Type: application/json; charset=utf-8 < Transfer-Encoding: chunked < Connection: keep-alive < Date: Thu, 23 Jan 2020 11:31:24 GMT < Cache-Control: private, max-age=86400 < X-Powered-By: Thundar! < X-Thetvdb-Api-Version: 3.0.0 < Vary: Accept-Language < X-Cache: Miss from cloudfront < Via: 1.1 12568db103eebfea9d7f7cb5f0a8e524.cloudfront.net (CloudFront) < X-Amz-Cf-Pop: FJR50-C1 < X-Amz-Cf-Id: LJIIAzQAZPHfTwsUm8zA3sV3pWX8VpkczVcyyWOYlCKS19F00ap35A== < -- END SNIPPET -- Adding Accept:application/vnd.thetvdb.v2 makes no difference to the response body, so I don't think it's an upgrade at the TVDB's end ----------------------------------------------------------------------------------------- 2020-01-23 12:01:47 btasker ----------------------------------------------------------------------------------------- I wonder if this is something stupid like them having recently added charset=utf-8 to Content-Type and Kodi doing a string check rather than parsing the headers properly. That'd be very poor, but it's probably worth checking. A quick bit of experimenting with Nginx config and the hosts file would suggest that's not the issue though. Realised my trac credentials work with Kodi's forum, so have posted what I found there - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918567#pid2918567 ----------------------------------------------------------------------------------------- 2020-01-23 12:02:06 btasker ----------------------------------------------------------------------------------------- I'm going to bump this issue over to MISC as it's clearly not an issue that's specific to my home network ----------------------------------------------------------------------------------------- 2020-01-23 12:04:21 ----------------------------------------------------------------------------------------- btasker changed Project from 'Home LAN' to 'Miscellaneous' ----------------------------------------------------------------------------------------- 2020-01-23 12:04:21 ----------------------------------------------------------------------------------------- btasker changed Key from 'LAN-138' to 'MISC-37' ----------------------------------------------------------------------------------------- 2020-01-23 12:04:37 ----------------------------------------------------------------------------------------- btasker added 'Kodi TVDB' to labels ----------------------------------------------------------------------------------------- 2020-01-23 12:20:53 btasker ----------------------------------------------------------------------------------------- There's a post by TVDB quoted on the Kodi site - https://koditips.com/kodi-not-working/ (actual source seems to be - https://forums.thetvdb.com/viewtopic.php?f=3&t=61687 ) -- BEGIN QUOTE -- One issue that we encountered was that some older versions of Python (mostly 2.7) experienced issues due to a bug in their bundled url libraries. This caused some old scrapers and players to experience a SSL handshake failure due to outdated/unsupported TLS support -- END QUOTE -- We can see from the captures that that's not the case here, there's definitely content coming down over the HTTPS connections ----------------------------------------------------------------------------------------- 2020-01-23 12:48:21 btasker ----------------------------------------------------------------------------------------- Ahhh, v1 of thetvdb API used to be XML. That's probably why the add-ons are trying to parse XML not JSON. I don't see anything in the requests that attempts to specify API v1, but the documented Accept header (https://api.thetvdb.com/swagger#Versioning) doesn't appear to work anyway - There's a fairly frustrated dev here https://old.reddit.com/r/Addons4Kodi/comments/esdi7l/tv_calendar_in_venom_and_others_just_hangs_since/ffadfh5/ who seems to be thinking along similar lines. - There seem to be issues left right and centre with the new API though, there's someone here reporting it just gives empty responses from time to time - https://forums.thetvdb.com/viewtopic.php?f=16&p=169321#p169321 - Another complaint about not being able to add anything - https://old.reddit.com/r/Addons4Kodi/comments/ervcfq/openmeta_issues/ ----------------------------------------------------------------------------------------- 2020-01-23 13:21:34 btasker ----------------------------------------------------------------------------------------- I think that's probably end of the line for me for now - it looks like TheTVDB have probably made a change which means search results come back in JSON instead of XML - at least insomuch as Kodi seems to be expecting XML here. I guess we just have to wait for either Kodi/TVDB guys to fix it. I've created a topic on the TVDB forum too - https://forums.thetvdb.com/viewtopic.php?f=16&t=62085 - though I don't expect to get very much back that I can do anything with. If we're lucky, maybe they made a change that can be rolled back, but based on an outsiders view I suspect the answer is that Kodi should really be using the JSON API and not relying on XML ----------------------------------------------------------------------------------------- 2020-01-24 09:47:39 btasker ----------------------------------------------------------------------------------------- Turns out that by default, neither forum emails you when a reply is posted. - Kodi have released a fix - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918674#pid2918674 - TheTVDB fairly unhelpfully replied to say that API v1 should be used instead - which kind of misses the point they've changed _something_ and broken downstream Kodi's fix is here - https://github.com/xbmc/repo-scrapers/pull/172/commits/013054e9c8d0f1ff44214cfb4a4812a79b42d874 ----------------------------------------------------------------------------------------- 2020-01-24 09:48:46 btasker ----------------------------------------------------------------------------------------- FWIW whilst TheTVDB's apparent inability to maintain a stable API is problematic/concerning, Kodi don't exactly help themselves here using - in effect - regular expressions to parse API responses rather than actually parsing the JSON ----------------------------------------------------------------------------------------- 2020-01-24 10:08:07 btasker ----------------------------------------------------------------------------------------- One of the Kodi devs has been a bit of a dick about this -- BEGIN QUOTE -- Please stop spreading this ignorant rumor about JSON vs XML. Kodi scraper addons are scraping JSON APIs, then transforming the collected results into an XML which is then gets parsed by Kodi core. -- END QUOTE -- Having reviewed the fix I've a few issues with what he's said, so I've replied https://forum.kodi.tv/showthread.php?tid=349385&pid=2918789#pid2918789 We can see in the commit that what's changed is the order of the attributes in the JSON - https://github.com/xbmc/repo-scrapers/pull/172/commits/013054e9c8d0f1ff44214cfb4a4812a79b42d874#diff-acaf350f27091a0fe065522129e3542e Basically, the attributes network and image have swapped places, and as a result the regex no longer matches. If they'd been parsing the JSON properly it wouldn't break a thing. When doing it with a regex you're entirely subject to things in changing like ordering. FAOD RFC 7159 - https://tools.ietf.org/html/rfc7159.html - states that attribute order does not matter within objects -- BEGIN QUOTE -- An object is an unordered collection of zero or more name/value pairs, where a name is a string and a value is a string, number, boolean, null, object, or array. -- END QUOTE -- ----------------------------------------------------------------------------------------- 2020-01-24 10:25:57 btasker ----------------------------------------------------------------------------------------- Triggering a scan to see if the update's helped -- BEGIN SNIPPET -- ben@milleniumfalcon:~$ curl -v --data-binary '{ "jsonrpc": "2.0", "method": "VideoLibrary.Scan", "id": "mybash"}' -H 'content-type: application/json;' http://osmc2.home/jsonrpc -- END SNIPPET -- That failed, but looking at it the box hasn't pulled down 3.2.4 yet. Manually updating the XML now (as I'm doing this remotely via SSH I can't tell Kodi to go off and update the addon, so literally overwriting tvdb.xml) Triggered another scan, looks like it's worked this time Confirmed then, v3.2.4 of metadata.tvdb.com fixes it. ----------------------------------------------------------------------------------------- 2020-01-24 10:26:06 ----------------------------------------------------------------------------------------- btasker changed status from 'Open' to 'Resolved' ----------------------------------------------------------------------------------------- 2020-01-24 10:26:06 ----------------------------------------------------------------------------------------- btasker added 'Fixed' to resolution ----------------------------------------------------------------------------------------- 2020-01-24 10:26:10 ----------------------------------------------------------------------------------------- btasker changed status from 'Resolved' to 'Closed' ----------------------------------------------------------------------------------------- 2020-01-26 10:09:08 btasker ----------------------------------------------------------------------------------------- Someone else with a dev background concurred with me that parsing JSON with regex is fragile, and again the dev seems very defensive about it - https://forum.kodi.tv/showthread.php?tid=349385&pid=2919044#pid2919044 I've stopped replying because, honestly, I'm not really looking for an argument. All I wanted was the fix, and the whole discussion afterwards only arose because it'd be _very_ nice for Kodi not to keep breaking because it's parsing an API in a fragile manner. Sounds like a change to that effect is on the cards, so there's no real value/logic in continuing to bruise the ego of a dev who seems to bruise quite easily. My first foray into the Kodi forums has been quite an unwelcoming experience really: I experienced an issue with the software, found a related thread where people were being asked to provide logs but hadn't yet done so. Provided the debug log, and based on what it said said that it suggested Kodi was expecting XML (Although that seems to have led people to assume the TVDB API was returning the wrong format, that was not my original assumption/conclusion - https://projects.bentasker.co.uk/jira_projects/browse/MISC-37.html#comment2693978). I think that's entirely defensible given the loglines (we now know what actually failed is the regex - it didn't match and so won't have returned any data. There _should_ have been an error logged then). I also asked on TheTVDB's forum if any recent changes to the API had been made. Could I have waited? Maybe, but as a user I wanted my software working again ASAP, and frankly have a pretty limited amount of time available to fiddle around with this stuff before I need to get on with other things (like work). I don't really see any harm in having asked if anything's changed - I certainly didn't pop up there and suggest the API should be returning XML. The result of all this was a Kodi dev getting bent out of shape and starting a post with -- BEGIN QUOTE -- Please stop spreading this ignorant rumor about JSON vs XML. -- END QUOTE -- Maybe I read too much tone into it, maybe not (given this - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918831#pid2918831 - I suspect not). I've been using Kodi for years, and often felt that I should contribute back more than the few bugs I've highlighted/raised. This experience hasn't exactly motivated me in that respect. It's not my battle to fight, but it still strikes me as grossly unfair to be letting the community think the TVDB fucked up (like here - https://forums.thetvdb.com/viewtopic.php?p=169605#p169605) when they didn't. If they decided for some reason to randomise the order of attributes in their JSON objects, that'd be 100% RFC compliant, and no RFC compliant client would complain/fail. Regex based implementations, of course, would: As far as JSON is concerned, each of these are identical -- BEGIN SNIPPET -- {"a":1,"b":2,"c":[1,2,3]} {"b":2,"a":1,"c":[1,2,3]} {"c":[1,2,3],"b":2,"a":1,} {"a":1,"c":[1,2,3],"b":2,} -- END SNIPPET -- The fascination with using regex to parse complex structured data goes back quite a way though, with multiple historic posts imploring people not to try and "parse" HTML with regex (example: https://boingboing.net/2011/11/24/why-you-shouldnt-parse-html.html), even doing something seemingly simple like validating URLs with Regex isn't straightforward (https://stackoverflow.com/questions/827557/how-do-you-validate-a-url-with-a-regular-expression-in-python). I get why Regex might initially have been chosen, even if I don't agree with that choice, but it seems to me a bit dishonest to then be blaming an upstream API for the consequences of your _own_ development decisions/assumptions. That TVDB's API hasn't changed/moved around for much of a decade is the result of luck (and a lack of inertia in terms of developing new stuff on their part). I don't want this comment to turn into a rant, or a character assassination, so I'll leave it here and close out with the immortal words of Jamie Zawinski (yes, it really was him - http://regex.info/blog/2006-09-15/247) -- BEGIN QUOTE -- Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems. -- END QUOTE --