project Utilities / File Location Listing avatar

utilities/file_location_listing#2: Create Crawler

Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: proof-of-concept
Created: 28-Dec-23 12:11


The crawler should read a list of predefined domains from config and crawl pages on those domains. It should store

  • scheme
  • domain
  • path
  • Filename
  • Last modified (if provided)
  • Content-type (if provided)

Toggle State Changes


assigned to @btasker

mentioned in issue #1


mentioned in commit 68b856ea787c19ad0cfc1bafd0b3a0e4fd959803

Commit: 68b856ea787c19ad0cfc1bafd0b3a0e4fd959803 
Author: B Tasker                            
Date: 2023-12-28T13:06:14.000+00:00 


Start creating a crawler (utilities/file_location_listing#2)

This fetches a single page and calculates an index file entry for it.

+302 -0 (302 lines changed)

mentioned in issue #3