project Utilities / File Location Listing avatar

utilities/file_location_listing#2: Create Crawler

Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: proof-of-concept
Created: 28-Dec-23 12:11



Description

The crawler should read a list of predefined domains from config and crawl pages on those domains. It should store

  • scheme
  • domain
  • path
  • Filename
  • Last modified (if provided)
  • Content-type (if provided)


Toggle State Changes

Activity


28-Dec-23 12:11

assigned to @btasker


28-Dec-23 12:11

mentioned in issue #1


28-Dec-23 13:29
verified

mentioned in commit 68b856ea787c19ad0cfc1bafd0b3a0e4fd959803

Commit: 68b856ea787c19ad0cfc1bafd0b3a0e4fd959803 
Author: B Tasker                            
                            
Date: 2023-12-28T13:06:14.000+00:00

Message

Start creating a crawler (utilities/file_location_listing#2)

This fetches a single page and calculates an index file entry for it.

+302 -0 (302 lines changed)

28-Dec-23 16:39

mentioned in issue #3