project project-management-only / Scraper Snitch Bot avatar

project-management-only/scraper-snitch-bot#7: Receipt regeneration duplicates flags and paths

Issue Information

Issue Type: issue
Status: closed
Reported By: btasker
Assigned To: btasker

Milestone: v0.14
Created: 14-Feb-23 18:35



Description

It looks like flags can become duplicated during regeneration:

### Flags
  - Does-not-fetch-robots.txt
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']
  - ['Acquires-tokens', 'Does-not-fetch-robots.txt']

Have observed the same happening with paths



Toggle State Changes

Activity


14-Feb-23 18:35

assigned to @btasker


14-Feb-23 18:36

I assume what's happening is we're accidentally writing a list of lists back rather than a single list (i.e. something's not being merged), so we end up with a list of lists in the YAML


14-Feb-23 18:38

Yup:

receipts:
  avg_daily_requests: 11.590909090909092
  first_seen: 2022-12-31 07:34:06 (UTC)
  flags:
  - Does-not-fetch-robots.txt
  - - Acquires-tokens
    - Does-not-fetch-robots.txt
  - - Acquires-tokens
    - Does-not-fetch-robots.txt
  - - Acquires-tokens
    - Does-not-fetch-robots.txt
  - - Acquires-tokens
    - Does-not-fetch-robots.txt

Yup....

def appendToListIfNew(old, new):
    ''' Take old and new

    Treat old as a CSV, if new doesn't appear
    append it

    '''

    merged = old

    for newua in new:
        if newua and newua not in merged:
            merged.append(new)

    return merged

We should be appending newua there not new


14-Feb-23 18:39
verified

mentioned in commit misc/python-mastodon-snitch-bot@3bff77d2a0fc85d61546bb652c6cc7ec064a7704

Commit: misc/python-mastodon-snitch-bot@3bff77d2a0fc85d61546bb652c6cc7ec064a7704 
Author: B Tasker                            
                            
Date: 2023-02-14T18:39:33.000+00:00

Message

bugfix: Append correct entity (project-management-only/scraper-snitch-bot#7)

+1 -1 (2 lines changed)

14-Feb-23 18:40

I'll cut a release to get this fix out.


14-Feb-23 18:59

v0.14 contains this fix. I've not marked as released yet as we need to do some tidying of the state files first


14-Feb-23 19:08

Man I'm glad I went with YAML as the format for state files, it made correction a hell of a lot easier.