MISC-37: Kodi can't update library with TheTVDB



Issue Information

Issue Type: Bug
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: Miscellaneous (MISC)
Resolution: Fixed (2020-01-24 10:26:06)
Labels: Kodi, TVDB,
Environment: Kodi 18.5
Kodi 17.1
TVDB Media Provider 3.2.3

Created: 2020-01-23 10:35:40
Time Spent Working


Description
Kodi's update library won't currently add any TV series. Media info source is thetvdb.com

Have verified that newly added directories are properly named to rule out it being that it just can't find them in the DB.


Issue Links

Kodi Forum
Toggle State Changes

Activity


This is consistent between Kodi boxes running different versions of the software (18.5 and 17.1 confirmed), so it doesn't seem to be something that's been recently broken within Kodi

Looking the kodi log, we get
2020-01-23 10:31:04.676 T:1649124064   ERROR: Run: Unable to parse web site
2020-01-23 10:31:04.690 T:1649124064   ERROR: Previous line repeats 1 times.
2020-01-23 10:31:04.690 T:1649124064 WARNING: No information found for item 'http://series.kat.home:80/Magic%5FKnight%5FRayearth/', it won't be added to the library.
2020-01-23 10:31:05.233 T:1649124064 WARNING: Skipping item 'http://series.kat.home:80/manual/' with '.nomedia' file in parent directory, it won't be added to the library.
2020-01-23 10:31:05.300 T:1649124064   ERROR: Run: Unable to parse web site
2020-01-23 10:31:05.315 T:1649124064   ERROR: Previous line repeats 1 times.
2020-01-23 10:31:05.315 T:1649124064 WARNING: No information found for item 'http://series.kat.home:80/Manyuu%5FHiken%2DChou/', it won't be added to the library.


Need to look into it more, but I'm guess the TVDB's API has changed
I edited .kodi/addons/metadata.tvdb.com/tvdb.xml to use HTTP URLs for the TVDB API calls so I could more easily intercept and extract creds for troubleshooting. Each of the calls results in a 302 to HTTPS so we don't get to see responses, but it's enough to allow me to recreate the flow with curl

Login flow works fine
ben@milleniumfalcon:~$ curl -v -H "content-type: application/json" -d '{"apikey":"[KEY]"}' https://api.thetvdb.com/login
*   Trying 13.35.178.200...
* Connected to api.thetvdb.com (13.35.178.200) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.thetvdb.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=*.thetvdb.com
* 	 start date: Thu, 23 May 2019 00:00:00 GMT
* 	 expire date: Tue, 23 Jun 2020 12:00:00 GMT
* 	 issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> POST /login HTTP/1.1
> Host: api.thetvdb.com
> User-Agent: curl/7.47.0
> Accept: */*
> content-type: application/json
> Content-Length: 29
> 
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Content-Length: 462
< Connection: keep-alive
< Date: Thu, 23 Jan 2020 11:12:07 GMT
< Vary: Accept-Language
< X-Powered-By: Thundar!
< X-Thetvdb-Api-Version: 3.0.0
< X-Cache: Miss from cloudfront
< Via: 1.1 5ebef4b6eba4f6ee211859e315b4fa16.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: FJR50-C1
< X-Amz-Cf-Id: 9eNAEmMHbTtgXo9uyhk6Jpxzn-05vhZNphEeHRm3qaUk7Cnbbn_ApQ==
< 
* Connection #0 to host api.thetvdb.com left intact
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI3MjcsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3OTI3fQ.3KbEW1oQq1J8eTIv1QEiFC3ebhN15jUKwF1G2A4er32B4wce_b1VRAUc_9_qgQXzxt0JI5sp1xuNeU7mW0aoT6WDiDA6RRtljdCJvNhDAeUsJ_i4jgkBEuY8kckNM7MX63-vgPRXm1AK4tC2ANGy-h0po1JGBD4hcLz0QnrFrdgZ50XcybVgiCDcHU2FRFiqwTcBKlkt3F0qsoSylURXCZDssJXEySZ1JjFZ1FwzE_1TgaMh1kB2eb08VYHaENcovAgekzpnXK1eSc7Qw0E6ErqINgMTm1gK8ml5Arsu91hak_DsP3P4cNAf0L7RRvD7_3sxzvTjMRssbxfHZqQSQQ"}


Can see Kodi searching later, and it too obviously got a token back as it's provided in the auth header
GET /search/series?name=Magic%20Knight%20Rayearth HTTP/1.1
Range: bytes=0-
User-Agent: OSMC (Linux; OSMC 2017.03-1)
Host: api.thetvdb.com
Accept: */*
Accept-Charset: UTF-8,*;q=0.8
Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A
accept-language: en


If we place that request with curl
ben@milleniumfalcon:~$ curl -v -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A" -H "accept-language: en" "https://api.thetvdb.com/search/series?name=Magic%20Knight%20Rayearth"
*   Trying 99.86.241.252...
* Connected to api.thetvdb.com (99.86.241.252) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.thetvdb.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=*.thetvdb.com
* 	 start date: Thu, 23 May 2019 00:00:00 GMT
* 	 expire date: Tue, 23 Jun 2020 12:00:00 GMT
* 	 issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /search/series?name=Magic%20Knight%20Rayearth HTTP/1.1
> Host: api.thetvdb.com
> User-Agent: curl/7.47.0
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A
> accept-language: en
> 
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Thu, 23 Jan 2020 11:14:30 GMT
< Cache-Control: private, max-age=86400
< X-Powered-By: Thundar!
< X-Thetvdb-Api-Version: 3.0.0
< Vary: Accept-Language
< X-Cache: Miss from cloudfront
< Via: 1.1 b48fca327a980187d93a198e7530195c.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: VIE50-C1
< X-Amz-Cf-Id: pLlRFaHSkAkQAViOz73GsOTZJv51BO8sUXeLD6BdJsSuEtTjyxUP5A==
< 
{"data":[{"aliases":["Una porta socchiusa ai confini del sole"],"banner":"/banners/graphical/71634-g3.jpg","firstAired":"1994-6-2","id":71634,"image":"/banners/posters/71634-5.jpg","network":"YTV (JP)","overview":"Three young girls, Hikaru, Umi, and Fuu, are transported to a magical world called Cephiro during a field trip to Tokyo Tower. They are soon greeted by Master Mage Clef, who explains to them that they have been summoned to become the Legendary Magic Knights and save Cephiro. The girls are less than enthusiastic about this idea, and only want to return home. Clef further explains that they must seek out the three Rune Gods to help them fight. He bestows armor and magical powers to each of them. They learn from Clef that High Priest Zagato has kidnapped the Pillar of Cephiro, Princess Emeraude. The Pillar of Cephiro has the sole responsibility of keeping Cephiro alive and in balance with her prayers. Without Princess Emeraude, Cephiro would fall into ruin. Hikaru, Umi, and Fuu must fight off Zagato's henchman and find the Rune Gods if they ever want to get back home. They soon learn that friendship and loyalty are the only things they can rely on in the crumbling Cephiro.","poster":"/banners/posters/71634-5.jpg","seriesName":"Magic Knight Rayearth","slug":"magic-knight-rayearth","status":"Ended"},{"aliases":["Magic Knight Rayearth OVA"],"banner":null,"firstAired":"1997-7-25","id":358557,"image":"/banners/series/358557/posters/62034069.jpg","network":null,"overview":"The Earth lies darkened and deserted. Despite the best efforts of Hikaru, the evil-filled castle of Cephiro has taken over the world and stands in the centre of a lifeless Tokyo. Before the ice-witch Alcyone can take her revenge on Hikaru though, she is rescued by Lantis, a young man who seals her into a magical maze where she is challenged to prove her powers while her friends are hunted down by Cephiro's warlocks.\r\n\r\nFleeing through the silent city, Umi and Fuu must seek out the sleeping Mashin — godlike creatures that will fuse with t* Connection #0 to host api.thetvdb.com left intact
hem to form giant warriors of awesome destructive power. But the Magic Knights are afraid, untrained and alone... will even the Mashin be enough to save them from the evil acts of Cephiro?","poster":null,"seriesName":"Rayearth (1997)","slug":"rayearth","status":"Ended"}]}


we get JSON back. passing it through python -m json.tool confirms it's valid.

So, Kodi is ostensibly getting data back but complaining it's unable to parse it
There are complaints about this on the Kodi forum too.

One of the devs has complained
We cannot diagnose from 3 lines of a log. Please provide the full log.


It's not quite worth the hassle of creating an account to point out that if they want more loglines maybe they should make their application log more than Unable to parse website by default.
Unfortunately the error message Unable to parse website sits in the middle of a binary - kodi.bin - media addons are essentially just XML files containing a bunch of regexes.

Lets get debug logging turned on on a Kodi box and see if that yields anything extra. My guess is going to be that the TVDB's JSON has changed slightly - based on previous Kodi bugs I've seen I don't have a massive amount of faith in some of their implementation.
11:28:01.504 T:1549005808   DEBUG: FindMovie: Searching for 'Magic_Knight_Rayearth' using The TVDB scraper (path: '/home/osmc/.kodi/addons/metadata.tvdb.com', content: 'tvshows', version: '3.2.3')
11:28:01.504 T:1549005808   DEBUG: scraper: CreateSearchUrl returned <url post="yes" cache="auth.json">http://api.thetvdb.com/login?{"apikey":"439DFEBA9D3059C6"}|Content-Type=application/json</url>
11:28:01.507 T:1549005808   DEBUG: scraper: GetSearchResults returned <results><url function="GetSearchResultsAuth" cache="search-Magic_Knight_Rayearth-en.json">http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth|Authorization=Bearer%20eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODM2NzQsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc4ODc0fQ.buhlqlAGCeGazN2Jq9rMgj9Wv5xgT1Zha8qmVSVlKmA0wSgpgkRcjPc9lcy36xFA71fzGKPDlleMTy7xp5zwo07-Dd9B0hLog1k_Gd-2NjvM4PDWbZHjYF5rH0OavotO6wBsYANLJCmJCgL7xbscaHax1BWcTHAKIaxvwBUjMDG6Ni6lmS-7Tuz5yPuu6QGAoQQfCKo5CDfugu2uZDm-faJ7B9T-hJaerbb_hLk7UJI31jO0Y2bs_eQR7oK9BbBn6sMXhSPiUvX_O3cRpZ6VhiKik-KwDrScESvcjWAcFozM4wsnFNtmOvpPNdVpVWxlvUddJrFt3ZbfLEwE1r33kQ&accept-language=en</url></results>
11:28:01.507 T:1549005808   DEBUG: CurlFile::ParseAndCorrectUrl() adding custom header option 'Authorization: ***********'
11:28:01.507 T:1549005808   DEBUG: CurlFile::ParseAndCorrectUrl() adding custom header option 'accept-language: en'
11:28:01.508 T:1549005808   DEBUG: CurlFile::Open(0x727e76e0) http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth
11:28:01.936 T:1549005808   DEBUG: CCurlFile::Open - effective URL: <https://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth>
11:28:01.937 T:1549005808   DEBUG: Get: Using "UTF-8" charset for "http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth|Authorization=Bearer%20eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODM2NzQsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc4ODc0fQ.buhlqlAGCeGazN2Jq9rMgj9Wv5xgT1Zha8qmVSVlKmA0wSgpgkRcjPc9lcy36xFA71fzGKPDlleMTy7xp5zwo07-Dd9B0hLog1k_Gd-2NjvM4PDWbZHjYF5rH0OavotO6wBsYANLJCmJCgL7xbscaHax1BWcTHAKIaxvwBUjMDG6Ni6lmS-7Tuz5yPuu6QGAoQQfCKo5CDfugu2uZDm-faJ7B9T-hJaerbb_hLk7UJI31jO0Y2bs_eQR7oK9BbBn6sMXhSPiUvX_O3cRpZ6VhiKik-KwDrScESvcjWAcFozM4wsnFNtmOvpPNdVpVWxlvUddJrFt3ZbfLEwE1r33kQ&accept-language=en"
11:28:01.942 T:1549005808   DEBUG: could not parse input XML
11:28:01.943 T:1549005808   DEBUG: XSLT: xslt transformation failed
11:28:01.944 T:1549005808   ERROR: Run: Unable to parse web site
11:28:01.945 T:1549005808 WARNING: No information found for item 'http://series.kat.home:80/Magic%5FKnight%5FRayearth/', it won't be added to the library.


could not parse input XML.... sigh, is it getting JSON back when it expects XML?

On the other hand, it's correctly submitting auth creds using JSON, which'd suggest the API has always been JSON (quick scan of https://api.thetvdb.com/swagger would support that theory), so is being passed to the wrong parser at Kodi's end?
Content-Type is correct in the response headers
> GET /search/series?name=Magic%20Knight%20Rayearth HTTP/1.1
> Host: api.thetvdb.com
> User-Agent: curl/7.47.0
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A
> accept-language: en
> 
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Thu, 23 Jan 2020 11:31:24 GMT
< Cache-Control: private, max-age=86400
< X-Powered-By: Thundar!
< X-Thetvdb-Api-Version: 3.0.0
< Vary: Accept-Language
< X-Cache: Miss from cloudfront
< Via: 1.1 12568db103eebfea9d7f7cb5f0a8e524.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: FJR50-C1
< X-Amz-Cf-Id: LJIIAzQAZPHfTwsUm8zA3sV3pWX8VpkczVcyyWOYlCKS19F00ap35A==
< 


Adding Accept:application/vnd.thetvdb.v2 makes no difference to the response body, so I don't think it's an upgrade at the TVDB's end
I wonder if this is something stupid like them having recently added charset=utf-8 to Content-Type and Kodi doing a string check rather than parsing the headers properly. That'd be very poor, but it's probably worth checking.

A quick bit of experimenting with Nginx config and the hosts file would suggest that's not the issue though.

Realised my trac credentials work with Kodi's forum, so have posted what I found there - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918567#pid2918567
I'm going to bump this issue over to MISC as it's clearly not an issue that's specific to my home network
btasker changed Project from 'Home LAN' to 'Miscellaneous'
btasker changed Key from 'LAN-138' to 'MISC-37'
btasker added 'Kodi TVDB' to labels
There's a post by TVDB quoted on the Kodi site - https://koditips.com/kodi-not-working/ (actual source seems to be - https://forums.thetvdb.com/viewtopic.php?f=3&t=61687 )
One issue that we encountered was that some older versions of Python (mostly 2.7) experienced issues due to a bug in their bundled url libraries. This caused some old scrapers and players to experience a SSL handshake failure due to outdated/unsupported TLS support


We can see from the captures that that's not the case here, there's definitely content coming down over the HTTPS connections
Ahhh, v1 of thetvdb API used to be XML. That's probably why the add-ons are trying to parse XML not JSON.

I don't see anything in the requests that attempts to specify API v1, but the documented Accept header (https://api.thetvdb.com/swagger#Versioning) doesn't appear to work anyway


- There's a fairly frustrated dev here https://old.reddit.com/r/Addons4Kodi/comments/esdi7l/tv_calendar_in_venom_and_others_just_hangs_since/ffadfh5/ who seems to be thinking along similar lines.
- There seem to be issues left right and centre with the new API though, there's someone here reporting it just gives empty responses from time to time - https://forums.thetvdb.com/viewtopic.php?f=16&p=169321#p169321
- Another complaint about not being able to add anything - https://old.reddit.com/r/Addons4Kodi/comments/ervcfq/openmeta_issues/
I think that's probably end of the line for me for now - it looks like TheTVDB have probably made a change which means search results come back in JSON instead of XML - at least insomuch as Kodi seems to be expecting XML here.

I guess we just have to wait for either Kodi/TVDB guys to fix it. I've created a topic on the TVDB forum too - https://forums.thetvdb.com/viewtopic.php?f=16&t=62085 - though I don't expect to get very much back that I can do anything with. If we're lucky, maybe they made a change that can be rolled back, but based on an outsiders view I suspect the answer is that Kodi should really be using the JSON API and not relying on XML
Turns out that by default, neither forum emails you when a reply is posted.

- Kodi have released a fix - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918674#pid2918674
- TheTVDB fairly unhelpfully replied to say that API v1 should be used instead - which kind of misses the point they've changed something and broken downstream

Kodi's fix is here - https://github.com/xbmc/repo-scrapers/pull/172/commits/013054e9c8d0f1ff44214cfb4a4812a79b42d874
FWIW whilst TheTVDB's apparent inability to maintain a stable API is problematic/concerning, Kodi don't exactly help themselves here using - in effect - regular expressions to parse API responses rather than actually parsing the JSON
One of the Kodi devs has been a bit of a dick about this
Please stop spreading this ignorant rumor about JSON vs XML.
Kodi scraper addons are scraping JSON APIs, then transforming the collected results into an XML which is then gets parsed by Kodi core.


Having reviewed the fix I've a few issues with what he's said, so I've replied https://forum.kodi.tv/showthread.php?tid=349385&pid=2918789#pid2918789

We can see in the commit that what's changed is the order of the attributes in the JSON - https://github.com/xbmc/repo-scrapers/pull/172/commits/013054e9c8d0f1ff44214cfb4a4812a79b42d874#diff-acaf350f27091a0fe065522129e3542e

Basically, the attributes network and image have swapped places, and as a result the regex no longer matches.

If they'd been parsing the JSON properly it wouldn't break a thing. When doing it with a regex you're entirely subject to things in changing like ordering.

FAOD RFC 7159 - https://tools.ietf.org/html/rfc7159.html - states that attribute order does not matter within objects
An object is an unordered collection of zero or more name/value
pairs, where a name is a string and a value is a string, number,
boolean, null, object, or array.


Triggering a scan to see if the update's helped
ben@milleniumfalcon:~$ curl -v --data-binary '{ "jsonrpc": "2.0", "method": "VideoLibrary.Scan", "id": "mybash"}' -H 'content-type: application/json;' http://osmc2.home/jsonrpc

That failed, but looking at it the box hasn't pulled down 3.2.4 yet. Manually updating the XML now (as I'm doing this remotely via SSH I can't tell Kodi to go off and update the addon, so literally overwriting tvdb.xml)

Triggered another scan, looks like it's worked this time

Confirmed then, v3.2.4 of metadata.tvdb.com fixes it.
btasker changed status from 'Open' to 'Resolved'
btasker added 'Fixed' to resolution
btasker changed status from 'Resolved' to 'Closed'
Someone else with a dev background concurred with me that parsing JSON with regex is fragile, and again the dev seems very defensive about it - https://forum.kodi.tv/showthread.php?tid=349385&pid=2919044#pid2919044

I've stopped replying because, honestly, I'm not really looking for an argument. All I wanted was the fix, and the whole discussion afterwards only arose because it'd be very nice for Kodi not to keep breaking because it's parsing an API in a fragile manner. Sounds like a change to that effect is on the cards, so there's no real value/logic in continuing to bruise the ego of a dev who seems to bruise quite easily.

My first foray into the Kodi forums has been quite an unwelcoming experience really:

I experienced an issue with the software, found a related thread where people were being asked to provide logs but hadn't yet done so. Provided the debug log, and based on what it said said that it suggested Kodi was expecting XML (Although that seems to have led people to assume the TVDB API was returning the wrong format, that was not my original assumption/conclusion - https://projects.bentasker.co.uk/jira_projects/browse/MISC-37.html#comment2693978). I think that's entirely defensible given the loglines (we now know what actually failed is the regex - it didn't match and so won't have returned any data. There should have been an error logged then).

I also asked on TheTVDB's forum if any recent changes to the API had been made. Could I have waited? Maybe, but as a user I wanted my software working again ASAP, and frankly have a pretty limited amount of time available to fiddle around with this stuff before I need to get on with other things (like work). I don't really see any harm in having asked if anything's changed - I certainly didn't pop up there and suggest the API should be returning XML.

The result of all this was a Kodi dev getting bent out of shape and starting a post with
Please stop spreading this ignorant rumor about JSON vs XML.


Maybe I read too much tone into it, maybe not (given this - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918831#pid2918831 - I suspect not).

I've been using Kodi for years, and often felt that I should contribute back more than the few bugs I've highlighted/raised. This experience hasn't exactly motivated me in that respect.

It's not my battle to fight, but it still strikes me as grossly unfair to be letting the community think the TVDB fucked up (like here - https://forums.thetvdb.com/viewtopic.php?p=169605#p169605) when they didn't. If they decided for some reason to randomise the order of attributes in their JSON objects, that'd be 100% RFC compliant, and no RFC compliant client would complain/fail. Regex based implementations, of course, would:

As far as JSON is concerned, each of these are identical
{"a":1,"b":2,"c":[1,2,3]}
{"b":2,"a":1,"c":[1,2,3]}
{"c":[1,2,3],"b":2,"a":1,}
{"a":1,"c":[1,2,3],"b":2,}


The fascination with using regex to parse complex structured data goes back quite a way though, with multiple historic posts imploring people not to try and "parse" HTML with regex (example: https://boingboing.net/2011/11/24/why-you-shouldnt-parse-html.html), even doing something seemingly simple like validating URLs with Regex isn't straightforward (https://stackoverflow.com/questions/827557/how-do-you-validate-a-url-with-a-regular-expression-in-python).

I get why Regex might initially have been chosen, even if I don't agree with that choice, but it seems to me a bit dishonest to then be blaming an upstream API for the consequences of your own development decisions/assumptions. That TVDB's API hasn't changed/moved around for much of a decade is the result of luck (and a lack of inertia in terms of developing new stuff on their part).

I don't want this comment to turn into a rant, or a character assassination, so I'll leave it here and close out with the immortal words of Jamie Zawinski (yes, it really was him - http://regex.info/blog/2006-09-15/247)
Some people, when confronted with a problem, think "I know,
I'll use regular expressions." Now they have two problems.