delete_url.py
Used to delete a URL (or URLs) from the database.
Usage:
./utilities/delete_url.py [-p] [url]
By default, the utility will delete any entry that exactly matches the provided URL.
If the prefix (-p) flag is used, however, it will delete any entry that starts with the provided URL, allowing recursive deletes.
Warning: the utility does not request confirmation
reindex.py
Used to rebuild the indexes
Usage:
./reindex.py
revalidate.py
Usage:
export REVAL_COUNT=500
./revalidate.py
Selects REVAL_COUNT URLs from the index and triggers non-recursive crawls of them to check that those urls are still valid. If they are not, they'll be removed from the database.