This is consistent between Kodi boxes running different versions of the software (18.5 and 17.1 confirmed), so it doesn't seem to be something that's been recently broken within Kodi
Looking the kodi log, we get
2020-01-23 10:31:04.676 T:1649124064 ERROR: Run: Unable to parse web site
2020-01-23 10:31:04.690 T:1649124064 ERROR: Previous line repeats 1 times.
2020-01-23 10:31:04.690 T:1649124064 WARNING: No information found for item 'http://series.kat.home:80/Magic%5FKnight%5FRayearth/', it won't be added to the library.
2020-01-23 10:31:05.233 T:1649124064 WARNING: Skipping item 'http://series.kat.home:80/manual/' with '.nomedia' file in parent directory, it won't be added to the library.
2020-01-23 10:31:05.300 T:1649124064 ERROR: Run: Unable to parse web site
2020-01-23 10:31:05.315 T:1649124064 ERROR: Previous line repeats 1 times.
2020-01-23 10:31:05.315 T:1649124064 WARNING: No information found for item 'http://series.kat.home:80/Manyuu%5FHiken%2DChou/', it won't be added to the library.
Need to look into it more, but I'm guess the TVDB's API has changed
I edited .kodi/addons/metadata.tvdb.com/tvdb.xml to use HTTP URLs for the TVDB API calls so I could more easily intercept and extract creds for troubleshooting. Each of the calls results in a 302 to HTTPS so we don't get to see responses, but it's enough to allow me to recreate the flow with curl
Login flow works fine
ben@milleniumfalcon:~$ curl -v -H "content-type: application/json" -d '{"apikey":"[KEY]"}' https://api.thetvdb.com/login
* Trying 13.35.178.200...
* Connected to api.thetvdb.com (13.35.178.200) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: *.thetvdb.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=*.thetvdb.com
* start date: Thu, 23 May 2019 00:00:00 GMT
* expire date: Tue, 23 Jun 2020 12:00:00 GMT
* issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
* compression: NULL
* ALPN, server accepted to use http/1.1
> POST /login HTTP/1.1
> Host: api.thetvdb.com
> User-Agent: curl/7.47.0
> Accept: */*
> content-type: application/json
> Content-Length: 29
>
* upload completely sent off: 29 out of 29 bytes
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Content-Length: 462
< Connection: keep-alive
< Date: Thu, 23 Jan 2020 11:12:07 GMT
< Vary: Accept-Language
< X-Powered-By: Thundar!
< X-Thetvdb-Api-Version: 3.0.0
< X-Cache: Miss from cloudfront
< Via: 1.1 5ebef4b6eba4f6ee211859e315b4fa16.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: FJR50-C1
< X-Amz-Cf-Id: 9eNAEmMHbTtgXo9uyhk6Jpxzn-05vhZNphEeHRm3qaUk7Cnbbn_ApQ==
<
* Connection #0 to host api.thetvdb.com left intact
{"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI3MjcsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3OTI3fQ.3KbEW1oQq1J8eTIv1QEiFC3ebhN15jUKwF1G2A4er32B4wce_b1VRAUc_9_qgQXzxt0JI5sp1xuNeU7mW0aoT6WDiDA6RRtljdCJvNhDAeUsJ_i4jgkBEuY8kckNM7MX63-vgPRXm1AK4tC2ANGy-h0po1JGBD4hcLz0QnrFrdgZ50XcybVgiCDcHU2FRFiqwTcBKlkt3F0qsoSylURXCZDssJXEySZ1JjFZ1FwzE_1TgaMh1kB2eb08VYHaENcovAgekzpnXK1eSc7Qw0E6ErqINgMTm1gK8ml5Arsu91hak_DsP3P4cNAf0L7RRvD7_3sxzvTjMRssbxfHZqQSQQ"}
Can see Kodi searching later, and it too obviously got a token back as it's provided in the auth header
ben@milleniumfalcon:~$ curl -v -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A" -H "accept-language: en" "https://api.thetvdb.com/search/series?name=Magic%20Knight%20Rayearth"
* Trying 99.86.241.252...
* Connected to api.thetvdb.com (99.86.241.252) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 603 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: *.thetvdb.com (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=*.thetvdb.com
* start date: Thu, 23 May 2019 00:00:00 GMT
* expire date: Tue, 23 Jun 2020 12:00:00 GMT
* issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET /search/series?name=Magic%20Knight%20Rayearth HTTP/1.1
> Host: api.thetvdb.com
> User-Agent: curl/7.47.0
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI1MDUsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3NzA1fQ.D70tll5WTBPt-s6Krh3_-JOnmY-H2YRLMe0CyqoQIYN72h3rzBBET430wCcrmE028MfkCpdjkLVASZBlALlntqsWytYWDdbR-hLEoKIiSH-OZIp6Ockhl59QyyY-_d4yTcVRH2KrqiZx0sM28TUF3KPpKn1cw2M11siYM_TbX5MJGMGZLjFmSJN2fahHBcLLM_QfaeDyvn7pQxvPzy_voRkSgtp-avHyNefZvrcqVE6BbrtHI7lzR5RUhyc2gT3-arx-W-UhpWZpoHHSkQs-Rwp4PfJh1bIA5kaczCHMvnIqC5d6VHydH-dJuAXID6AB7J1vw8Is7zr7amiXDbpq4A
> accept-language: en
>
< HTTP/1.1 200 OK
< Content-Type: application/json; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Thu, 23 Jan 2020 11:14:30 GMT
< Cache-Control: private, max-age=86400
< X-Powered-By: Thundar!
< X-Thetvdb-Api-Version: 3.0.0
< Vary: Accept-Language
< X-Cache: Miss from cloudfront
< Via: 1.1 b48fca327a980187d93a198e7530195c.cloudfront.net (CloudFront)
< X-Amz-Cf-Pop: VIE50-C1
< X-Amz-Cf-Id: pLlRFaHSkAkQAViOz73GsOTZJv51BO8sUXeLD6BdJsSuEtTjyxUP5A==
<
{"data":[{"aliases":["Una porta socchiusa ai confini del sole"],"banner":"/banners/graphical/71634-g3.jpg","firstAired":"1994-6-2","id":71634,"image":"/banners/posters/71634-5.jpg","network":"YTV (JP)","overview":"Three young girls, Hikaru, Umi, and Fuu, are transported to a magical world called Cephiro during a field trip to Tokyo Tower. They are soon greeted by Master Mage Clef, who explains to them that they have been summoned to become the Legendary Magic Knights and save Cephiro. The girls are less than enthusiastic about this idea, and only want to return home. Clef further explains that they must seek out the three Rune Gods to help them fight. He bestows armor and magical powers to each of them. They learn from Clef that High Priest Zagato has kidnapped the Pillar of Cephiro, Princess Emeraude. The Pillar of Cephiro has the sole responsibility of keeping Cephiro alive and in balance with her prayers. Without Princess Emeraude, Cephiro would fall into ruin. Hikaru, Umi, and Fuu must fight off Zagato's henchman and find the Rune Gods if they ever want to get back home. They soon learn that friendship and loyalty are the only things they can rely on in the crumbling Cephiro.","poster":"/banners/posters/71634-5.jpg","seriesName":"Magic Knight Rayearth","slug":"magic-knight-rayearth","status":"Ended"},{"aliases":["Magic Knight Rayearth OVA"],"banner":null,"firstAired":"1997-7-25","id":358557,"image":"/banners/series/358557/posters/62034069.jpg","network":null,"overview":"The Earth lies darkened and deserted. Despite the best efforts of Hikaru, the evil-filled castle of Cephiro has taken over the world and stands in the centre of a lifeless Tokyo. Before the ice-witch Alcyone can take her revenge on Hikaru though, she is rescued by Lantis, a young man who seals her into a magical maze where she is challenged to prove her powers while her friends are hunted down by Cephiro's warlocks.\r\n\r\nFleeing through the silent city, Umi and Fuu must seek out the sleeping Mashin — godlike creatures that will fuse with t* Connection #0 to host api.thetvdb.com left intact
hem to form giant warriors of awesome destructive power. But the Magic Knights are afraid, untrained and alone... will even the Mashin be enough to save them from the evil acts of Cephiro?","poster":null,"seriesName":"Rayearth (1997)","slug":"rayearth","status":"Ended"}]}
we get JSON back. passing it through python -m json.tool confirms it's valid.
So, Kodi is ostensibly getting data back but complaining it's unable to parse it
There are complaints about this on the Kodi forum too.
One of the devs has complained
We cannot diagnose from 3 lines of a log. Please provide the full log.
It's not quite worth the hassle of creating an account to point out that if they want more loglines maybe they should make their application log more than Unable to parse website by default.
Unfortunately the error message Unable to parse website sits in the middle of a binary - kodi.bin - media addons are essentially just XML files containing a bunch of regexes.
Lets get debug logging turned on on a Kodi box and see if that yields anything extra. My guess is going to be that the TVDB's JSON has changed slightly - based on previous Kodi bugs I've seen I don't have a massive amount of faith in some of their implementation.
11:28:01.504 T:1549005808 DEBUG: FindMovie: Searching for 'Magic_Knight_Rayearth' using The TVDB scraper (path: '/home/osmc/.kodi/addons/metadata.tvdb.com', content: 'tvshows', version: '3.2.3')
11:28:01.504 T:1549005808 DEBUG: scraper: CreateSearchUrl returned <url post="yes" cache="auth.json">http://api.thetvdb.com/login?{"apikey":"439DFEBA9D3059C6"}|Content-Type=application/json</url>
11:28:01.507 T:1549005808 DEBUG: scraper: GetSearchResults returned <results><url function="GetSearchResultsAuth" cache="search-Magic_Knight_Rayearth-en.json">http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth|Authorization=Bearer%20eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODM2NzQsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc4ODc0fQ.buhlqlAGCeGazN2Jq9rMgj9Wv5xgT1Zha8qmVSVlKmA0wSgpgkRcjPc9lcy36xFA71fzGKPDlleMTy7xp5zwo07-Dd9B0hLog1k_Gd-2NjvM4PDWbZHjYF5rH0OavotO6wBsYANLJCmJCgL7xbscaHax1BWcTHAKIaxvwBUjMDG6Ni6lmS-7Tuz5yPuu6QGAoQQfCKo5CDfugu2uZDm-faJ7B9T-hJaerbb_hLk7UJI31jO0Y2bs_eQR7oK9BbBn6sMXhSPiUvX_O3cRpZ6VhiKik-KwDrScESvcjWAcFozM4wsnFNtmOvpPNdVpVWxlvUddJrFt3ZbfLEwE1r33kQ&accept-language=en</url></results>
11:28:01.507 T:1549005808 DEBUG: CurlFile::ParseAndCorrectUrl() adding custom header option 'Authorization: ***********'
11:28:01.507 T:1549005808 DEBUG: CurlFile::ParseAndCorrectUrl() adding custom header option 'accept-language: en'
11:28:01.508 T:1549005808 DEBUG: CurlFile::Open(0x727e76e0) http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth
11:28:01.936 T:1549005808 DEBUG: CCurlFile::Open - effective URL: <https://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth>
11:28:01.937 T:1549005808 DEBUG: Get: Using "UTF-8" charset for "http://api.thetvdb.com/search/series?name=Magic_Knight_Rayearth|Authorization=Bearer%20eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODM2NzQsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc4ODc0fQ.buhlqlAGCeGazN2Jq9rMgj9Wv5xgT1Zha8qmVSVlKmA0wSgpgkRcjPc9lcy36xFA71fzGKPDlleMTy7xp5zwo07-Dd9B0hLog1k_Gd-2NjvM4PDWbZHjYF5rH0OavotO6wBsYANLJCmJCgL7xbscaHax1BWcTHAKIaxvwBUjMDG6Ni6lmS-7Tuz5yPuu6QGAoQQfCKo5CDfugu2uZDm-faJ7B9T-hJaerbb_hLk7UJI31jO0Y2bs_eQR7oK9BbBn6sMXhSPiUvX_O3cRpZ6VhiKik-KwDrScESvcjWAcFozM4wsnFNtmOvpPNdVpVWxlvUddJrFt3ZbfLEwE1r33kQ&accept-language=en"
11:28:01.942 T:1549005808 DEBUG: could not parse input XML
11:28:01.943 T:1549005808 DEBUG: XSLT: xslt transformation failed
11:28:01.944 T:1549005808 ERROR: Run: Unable to parse web site
11:28:01.945 T:1549005808 WARNING: No information found for item 'http://series.kat.home:80/Magic%5FKnight%5FRayearth/', it won't be added to the library.
could not parse input XML.... sigh, is it getting JSON back when it expects XML?
On the other hand, it's correctly submitting auth creds using JSON, which'd suggest the API has always been JSON (quick scan of https://api.thetvdb.com/swagger would support that theory), so is being passed to the wrong parser at Kodi's end?
I wonder if this is something stupid like them having recently added charset=utf-8 to Content-Type and Kodi doing a string check rather than parsing the headers properly. That'd be very poor, but it's probably worth checking.
A quick bit of experimenting with Nginx config and the hosts file would suggest that's not the issue though.
One issue that we encountered was that some older versions of Python (mostly 2.7) experienced issues due to a bug in their bundled url libraries. This caused some old scrapers and players to experience a SSL handshake failure due to outdated/unsupported TLS support
We can see from the captures that that's not the case here, there's definitely content coming down over the HTTPS connections
Ahhh, v1 of thetvdb API used to be XML. That's probably why the add-ons are trying to parse XML not JSON.
I don't see anything in the requests that attempts to specify API v1, but the documented Accept header (https://api.thetvdb.com/swagger#Versioning) doesn't appear to work anyway
I think that's probably end of the line for me for now - it looks like TheTVDB have probably made a change which means search results come back in JSON instead of XML - at least insomuch as Kodi seems to be expecting XML here.
I guess we just have to wait for either Kodi/TVDB guys to fix it. I've created a topic on the TVDB forum too - https://forums.thetvdb.com/viewtopic.php?f=16&t=62085 - though I don't expect to get very much back that I can do anything with. If we're lucky, maybe they made a change that can be rolled back, but based on an outsiders view I suspect the answer is that Kodi should really be using the JSON API and not relying on XML
FWIW whilst TheTVDB's apparent inability to maintain a stable API is problematic/concerning, Kodi don't exactly help themselves here using - in effect - regular expressions to parse API responses rather than actually parsing the JSON
One of the Kodi devs has been a bit of a dick about this
Please stop spreading this ignorant rumor about JSON vs XML.
Kodi scraper addons are scraping JSON APIs, then transforming the collected results into an XML which is then gets parsed by Kodi core.
Basically, the attributes network and image have swapped places, and as a result the regex no longer matches.
If they'd been parsing the JSON properly it wouldn't break a thing. When doing it with a regex you're entirely subject to things in changing like ordering.
An object is an unordered collection of zero or more name/value
pairs, where a name is a string and a value is a string, number,
boolean, null, object, or array.
That failed, but looking at it the box hasn't pulled down 3.2.4 yet. Manually updating the XML now (as I'm doing this remotely via SSH I can't tell Kodi to go off and update the addon, so literally overwriting tvdb.xml)
Triggered another scan, looks like it's worked this time
Confirmed then, v3.2.4 of metadata.tvdb.com fixes it.
I've stopped replying because, honestly, I'm not really looking for an argument. All I wanted was the fix, and the whole discussion afterwards only arose because it'd be very nice for Kodi not to keep breaking because it's parsing an API in a fragile manner. Sounds like a change to that effect is on the cards, so there's no real value/logic in continuing to bruise the ego of a dev who seems to bruise quite easily.
My first foray into the Kodi forums has been quite an unwelcoming experience really:
I experienced an issue with the software, found a related thread where people were being asked to provide logs but hadn't yet done so. Provided the debug log, and based on what it said said that it suggested Kodi was expecting XML (Although that seems to have led people to assume the TVDB API was returning the wrong format, that was not my original assumption/conclusion - https://projects.bentasker.co.uk/jira_projects/browse/MISC-37.html#comment2693978). I think that's entirely defensible given the loglines (we now know what actually failed is the regex - it didn't match and so won't have returned any data. There should have been an error logged then).
I also asked on TheTVDB's forum if any recent changes to the API had been made. Could I have waited? Maybe, but as a user I wanted my software working again ASAP, and frankly have a pretty limited amount of time available to fiddle around with this stuff before I need to get on with other things (like work). I don't really see any harm in having asked if anything's changed - I certainly didn't pop up there and suggest the API should be returning XML.
The result of all this was a Kodi dev getting bent out of shape and starting a post with
Please stop spreading this ignorant rumor about JSON vs XML.
I've been using Kodi for years, and often felt that I should contribute back more than the few bugs I've highlighted/raised. This experience hasn't exactly motivated me in that respect.
It's not my battle to fight, but it still strikes me as grossly unfair to be letting the community think the TVDB fucked up (like here - https://forums.thetvdb.com/viewtopic.php?p=169605#p169605) when they didn't. If they decided for some reason to randomise the order of attributes in their JSON objects, that'd be 100% RFC compliant, and no RFC compliant client would complain/fail. Regex based implementations, of course, would:
As far as JSON is concerned, each of these are identical
I get why Regex might initially have been chosen, even if I don't agree with that choice, but it seems to me a bit dishonest to then be blaming an upstream API for the consequences of your own development decisions/assumptions. That TVDB's API hasn't changed/moved around for much of a decade is the result of luck (and a lack of inertia in terms of developing new stuff on their part).
I don't want this comment to turn into a rant, or a character assassination, so I'll leave it here and close out with the immortal words of Jamie Zawinski (yes, it really was him - http://regex.info/blog/2006-09-15/247)
Some people, when confronted with a problem, think Â"I know,
I'll use regular expressions.Â" Now they have two problems.
Activity
2020-01-23 10:37:29
Looking the kodi log, we get
Need to look into it more, but I'm guess the TVDB's API has changed
2020-01-23 11:20:21
Login flow works fine
ben@milleniumfalcon:~$ curl -v -H "content-type: application/json" -d '{"apikey":"[KEY]"}' https://api.thetvdb.com/login * Trying 13.35.178.200... * Connected to api.thetvdb.com (13.35.178.200) port 443 (#0) * found 148 certificates in /etc/ssl/certs/ca-certificates.crt * found 603 certificates in /etc/ssl/certs * ALPN, offering http/1.1 * SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256 * server certificate verification OK * server certificate status verification SKIPPED * common name: *.thetvdb.com (matched) * server certificate expiration date OK * server certificate activation date OK * certificate public key: RSA * certificate version: #3 * subject: CN=*.thetvdb.com * start date: Thu, 23 May 2019 00:00:00 GMT * expire date: Tue, 23 Jun 2020 12:00:00 GMT * issuer: C=US,O=Amazon,OU=Server CA 1B,CN=Amazon * compression: NULL * ALPN, server accepted to use http/1.1 > POST /login HTTP/1.1 > Host: api.thetvdb.com > User-Agent: curl/7.47.0 > Accept: */* > content-type: application/json > Content-Length: 29 > * upload completely sent off: 29 out of 29 bytes < HTTP/1.1 200 OK < Content-Type: application/json; charset=utf-8 < Content-Length: 462 < Connection: keep-alive < Date: Thu, 23 Jan 2020 11:12:07 GMT < Vary: Accept-Language < X-Powered-By: Thundar! < X-Thetvdb-Api-Version: 3.0.0 < X-Cache: Miss from cloudfront < Via: 1.1 5ebef4b6eba4f6ee211859e315b4fa16.cloudfront.net (CloudFront) < X-Amz-Cf-Pop: FJR50-C1 < X-Amz-Cf-Id: 9eNAEmMHbTtgXo9uyhk6Jpxzn-05vhZNphEeHRm3qaUk7Cnbbn_ApQ== < * Connection #0 to host api.thetvdb.com left intact {"token":"eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1ODAzODI3MjcsImlkIjoiS29kaSIsIm9yaWdfaWF0IjoxNTc5Nzc3OTI3fQ.3KbEW1oQq1J8eTIv1QEiFC3ebhN15jUKwF1G2A4er32B4wce_b1VRAUc_9_qgQXzxt0JI5sp1xuNeU7mW0aoT6WDiDA6RRtljdCJvNhDAeUsJ_i4jgkBEuY8kckNM7MX63-vgPRXm1AK4tC2ANGy-h0po1JGBD4hcLz0QnrFrdgZ50XcybVgiCDcHU2FRFiqwTcBKlkt3F0qsoSylURXCZDssJXEySZ1JjFZ1FwzE_1TgaMh1kB2eb08VYHaENcovAgekzpnXK1eSc7Qw0E6ErqINgMTm1gK8ml5Arsu91hak_DsP3P4cNAf0L7RRvD7_3sxzvTjMRssbxfHZqQSQQ"}Can see Kodi searching later, and it too obviously got a token back as it's provided in the auth header
If we place that request with curl
we get JSON back. passing it through
So, Kodi is ostensibly getting data back but complaining it's unable to parse it
2020-01-23 11:21:54
One of the devs has complained
It's not quite worth the hassle of creating an account to point out that if they want more loglines maybe they should make their application log more than
2020-01-23 11:26:49
Lets get debug logging turned on on a Kodi box and see if that yields anything extra. My guess is going to be that the TVDB's JSON has changed slightly - based on previous Kodi bugs I've seen I don't have a massive amount of faith in some of their implementation.
2020-01-23 11:33:40
On the other hand, it's correctly submitting auth creds using JSON, which'd suggest the API has always been JSON (quick scan of https://api.thetvdb.com/swagger would support that theory), so is being passed to the wrong parser at Kodi's end?
2020-01-23 11:42:00
Adding
2020-01-23 12:01:47
A quick bit of experimenting with Nginx config and the hosts file would suggest that's not the issue though.
Realised my trac credentials work with Kodi's forum, so have posted what I found there - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918567#pid2918567
2020-01-23 12:02:06
2020-01-23 12:04:21
2020-01-23 12:04:21
2020-01-23 12:04:37
2020-01-23 12:20:53
We can see from the captures that that's not the case here, there's definitely content coming down over the HTTPS connections
2020-01-23 12:48:21
I don't see anything in the requests that attempts to specify API v1, but the documented
- There's a fairly frustrated dev here https://old.reddit.com/r/Addons4Kodi/comments/esdi7l/tv_calendar_in_venom_and_others_just_hangs_since/ffadfh5/ who seems to be thinking along similar lines.
- There seem to be issues left right and centre with the new API though, there's someone here reporting it just gives empty responses from time to time - https://forums.thetvdb.com/viewtopic.php?f=16&p=169321#p169321
- Another complaint about not being able to add anything - https://old.reddit.com/r/Addons4Kodi/comments/ervcfq/openmeta_issues/
2020-01-23 13:21:34
I guess we just have to wait for either Kodi/TVDB guys to fix it. I've created a topic on the TVDB forum too - https://forums.thetvdb.com/viewtopic.php?f=16&t=62085 - though I don't expect to get very much back that I can do anything with. If we're lucky, maybe they made a change that can be rolled back, but based on an outsiders view I suspect the answer is that Kodi should really be using the JSON API and not relying on XML
2020-01-24 09:47:39
- Kodi have released a fix - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918674#pid2918674
- TheTVDB fairly unhelpfully replied to say that API v1 should be used instead - which kind of misses the point they've changed something and broken downstream
Kodi's fix is here - https://github.com/xbmc/repo-scrapers/pull/172/commits/013054e9c8d0f1ff44214cfb4a4812a79b42d874
2020-01-24 09:48:46
2020-01-24 10:08:07
Having reviewed the fix I've a few issues with what he's said, so I've replied https://forum.kodi.tv/showthread.php?tid=349385&pid=2918789#pid2918789
We can see in the commit that what's changed is the order of the attributes in the JSON - https://github.com/xbmc/repo-scrapers/pull/172/commits/013054e9c8d0f1ff44214cfb4a4812a79b42d874#diff-acaf350f27091a0fe065522129e3542e
Basically, the attributes
If they'd been parsing the JSON properly it wouldn't break a thing. When doing it with a regex you're entirely subject to things in changing like ordering.
FAOD RFC 7159 - https://tools.ietf.org/html/rfc7159.html - states that attribute order does not matter within objects
2020-01-24 10:25:57
ben@milleniumfalcon:~$ curl -v --data-binary '{ "jsonrpc": "2.0", "method": "VideoLibrary.Scan", "id": "mybash"}' -H 'content-type: application/json;' http://osmc2.home/jsonrpcThat failed, but looking at it the box hasn't pulled down 3.2.4 yet. Manually updating the XML now (as I'm doing this remotely via SSH I can't tell Kodi to go off and update the addon, so literally overwriting
Triggered another scan, looks like it's worked this time
Confirmed then, v3.2.4 of
2020-01-24 10:26:06
2020-01-24 10:26:06
2020-01-24 10:26:10
2020-01-26 10:09:08
I've stopped replying because, honestly, I'm not really looking for an argument. All I wanted was the fix, and the whole discussion afterwards only arose because it'd be very nice for Kodi not to keep breaking because it's parsing an API in a fragile manner. Sounds like a change to that effect is on the cards, so there's no real value/logic in continuing to bruise the ego of a dev who seems to bruise quite easily.
My first foray into the Kodi forums has been quite an unwelcoming experience really:
I experienced an issue with the software, found a related thread where people were being asked to provide logs but hadn't yet done so. Provided the debug log, and based on what it said said that it suggested Kodi was expecting XML (Although that seems to have led people to assume the TVDB API was returning the wrong format, that was not my original assumption/conclusion - https://projects.bentasker.co.uk/jira_projects/browse/MISC-37.html#comment2693978). I think that's entirely defensible given the loglines (we now know what actually failed is the regex - it didn't match and so won't have returned any data. There should have been an error logged then).
I also asked on TheTVDB's forum if any recent changes to the API had been made. Could I have waited? Maybe, but as a user I wanted my software working again ASAP, and frankly have a pretty limited amount of time available to fiddle around with this stuff before I need to get on with other things (like work). I don't really see any harm in having asked if anything's changed - I certainly didn't pop up there and suggest the API should be returning XML.
The result of all this was a Kodi dev getting bent out of shape and starting a post with
Maybe I read too much tone into it, maybe not (given this - https://forum.kodi.tv/showthread.php?tid=349385&pid=2918831#pid2918831 - I suspect not).
I've been using Kodi for years, and often felt that I should contribute back more than the few bugs I've highlighted/raised. This experience hasn't exactly motivated me in that respect.
It's not my battle to fight, but it still strikes me as grossly unfair to be letting the community think the TVDB fucked up (like here - https://forums.thetvdb.com/viewtopic.php?p=169605#p169605) when they didn't. If they decided for some reason to randomise the order of attributes in their JSON objects, that'd be 100% RFC compliant, and no RFC compliant client would complain/fail. Regex based implementations, of course, would:
As far as JSON is concerned, each of these are identical
{"a":1,"b":2,"c":[1,2,3]} {"b":2,"a":1,"c":[1,2,3]} {"c":[1,2,3],"b":2,"a":1,} {"a":1,"c":[1,2,3],"b":2,}The fascination with using regex to parse complex structured data goes back quite a way though, with multiple historic posts imploring people not to try and "parse" HTML with regex (example: https://boingboing.net/2011/11/24/why-you-shouldnt-parse-html.html), even doing something seemingly simple like validating URLs with Regex isn't straightforward (https://stackoverflow.com/questions/827557/how-do-you-validate-a-url-with-a-regular-expression-in-python).
I get why Regex might initially have been chosen, even if I don't agree with that choice, but it seems to me a bit dishonest to then be blaming an upstream API for the consequences of your own development decisions/assumptions. That TVDB's API hasn't changed/moved around for much of a decade is the result of luck (and a lack of inertia in terms of developing new stuff on their part).
I don't want this comment to turn into a rant, or a character assassination, so I'll leave it here and close out with the immortal words of Jamie Zawinski (yes, it really was him - http://regex.info/blog/2006-09-15/247)