utilities/file_location_listing#62: File types aren't correctly recognised if the server includes a charset

The crawler relies on the server's Content-Type header when identifying whether (and how) to extract URLs from a URL.

For example:

if pd["content-type"] == "text/html":
                pagecontent = BeautifulSoup(pg.content, features="lxml")
                links, title, tags, published = extractUrlsFromHTML(pagecontent, url)

The attribute content-type is a copy of the server's content-type header caste to lowercase.

However, the server might include details of the characterset in use, for example

Content-Type: text/html; charset=utf8

This will result in pd["content-type"] no longer matching these checks and so not extracting URLs (or metadata) from that URL