project-management-only/scraper-snitch-bot#9: Allow subnet matching

whois gives the following information for this block

inet6num:       2a03:2880:300::/40
netname:        EAG
country:        US
admin-c:        RD4299-RIPE
tech-c:         RD4299-RIPE
status:         ASSIGNED
mnt-by:         fb-neteng
mnt-by:         facebook-neteng

Just for confirmation, we don't currently have the ability to exclude via subnet - it looks for exact matches:

        if bot['ip'] in config['exclude_ips']:
            # Allowlisted IP
            print(f"Skipping {bot['ip']}, present in allowlist")
            continue

I wonder whether the answer is to have calcFileName do the swap?

def calcFileName(ip):
    ''' Turn the bot's IP into a filesystem and url safe filename
    '''
    # generate the filename
    fname = ip.replace(".", "-").replace(":", "-")

    # Play it safe and assume someone found a way to get an invalid IP into the db
    # strip chars so they can't pull filesystem shenanigans
    fname = re.sub('([^(a-f|A-F|\-|0-9)]+)', '', fname)
    return fname

I could have a manual config list of prefixes and have the function override the filename if there's a match.

That would also prevent toots from being sent, the state file will exist.

If I do ^ we also need to think about what will happen during receipt regeneration.

It looks like it should be OK, but we need to make sure that the attribute ip in the state file is an IP rather than the subnet (otherwise various lookups may fail)

def regenerate():
    ''' Look for IPs that we know about and regenerate information for them

    [misc/python-mastodon-snitch-bot#2](/issue/misc/python-mastodon-snitch-bot/2.html)
    '''

    # Load the main config
    config = loadConfig(CONFIG_FILE)

    conn, cursor = getCursor(config['flightsql']['host'], 
                             config['flightsql']['port'], 
                             config['flightsql']['token'],
                             config['flightsql']['bucket'])
    runtime = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S (UTC)')


    # Get a list of state files
    dir_path = r'{}/state-*.txt'.format(config['state_dir'])
    res = glob.glob(dir_path)

    for statefile in res:
        with open(statefile, "r") as fh:
            try:
                bot = yaml.safe_load(fh)
            except:
                print(f"Invalid YAML in {statefile}")
                continue

        receipt_file = f"{config['receipt_dir']}/{bot['fname']}.txt"

        if bot['ip'] in config['exclude_ips']:
            # Allowlisted IP, remove the report/state
            print(f"Removing {bot['ip']}, present in allowlist")

            if os.path.exists(receipt_file):
                os.remove(receipt_file)

            os.remove(statefile)
            continue


        # Get new receipts
        new = {}
        new['receipts'] = getReceipts(cursor, bot['ip'], bot['queried_period'], config['flightsql']['forever_filter'], config['robots_disallow_all'].upper())
        new['extended_info'] = getIPInfo(bot['ip'])
        bot['last_checked'] = runtime

        merged_bot = mergeReceipts(bot, new)

        # Write the updated receipt
        writeReceiptFile(merged_bot, config['receipt_dir'], bot['fname'], runtime, config['notes_dir'])

        # Write the latest state to the statefile
        with open(statefile, "w") as fh:
            yaml.dump(merged_bot, fh)

The basic functionality is implemented, it's possible to provide a list of string prefixes:

grouped_prefixes:
  - 2a03:2880:3

This isn't going to be massively useful to people though. The aim of that config is to block 2a03:2880:300::/40.

I deliberately went with string matching rather than IP parsing to keep the check cheap, but we need a way to communicate to users which subnet they should be blocking.

One option would be to list subnets and check whether the IP is in it, the other would be to make the config more verbose

grouped_prefixes:
  - name: 2a03:2880:300::/40
    pref: 2a03:2880:3

I think it's probably best to do it properly and parse IPs and Networks - otherwise we'll only end up overblocking by accident

OK, the logic is in place then - what we need to look at now is how to go about exposing information to the user.

The state and receipt filename has a -subnet suffix (e.g. state-2a03-2880-300--40-subnet.txt), but we probably want the receipt contents to note that it's a subnet match.

We should also hide various bits of information

• The request count will be incorrect (it'll be for whatever IP last triggered it)
• user_agents will be correct but may be incomplete (if different UA's are used by different IPs within the subnet)

Receipt files now reflect that they are for a subnet match:

# Suspected Mastodon Scraper: 2a03:2880:300::/40

File first generated: 2025-07-08 10:05:45 (UTC)
File (re)generated: 2025-07-08 10:05:45 (UTC)


### IP Information

Subnet: 2a03:2880:300::/40
rDNS: None
ASN: [32934](https://ipinfo.io/AS32934)

Tor Exit Node: False

----

### Overview

Warning: the following stats are likely to be incorrect because this match is part of a known wider subnet

Observed Requests: 14
First Seen: 2025-07-06 13:40:55 (UTC)
Last Seen:  2025-07-07 05:12:43 (UTC)

Average number of daily requests: 7.0

We should probably adjust the toot text too.

I think we should probably log a flag too, that way people can look it up on the wiki and understand why the receipt file is different.

Cool, this seems to be working - I've tested against the last 48hrs of events.

Wiki has been updated

Closing ready to cut a release

Just for my own reference in future, when adding a subnet to the config, it's also possible to pre-stage some additional notes.

For example, for 2a03:2880:300::/40 I did

nano notes/2a03-2880-300--40-subnet.txt

FTR, other observed facebook subnets are

• 2a03:2880:1100::/40
• 2a03:2880:1300::/40
• 2a03:2880:2700::/40
• 2a03:2880:200::/40
• 2a03:2880:3100::/40
• 2a03:2880:3200::/40

Realistically, we probably want to group that into 2a03:2880::/32