utilities/tp-link-to-influxdb#7: Cannot poll Tapo devices with firmware >= 1.2.1

The tracker for the P100 module does suggest that this is a thing.

The last commit in that repo was a long time ago though, so it's unlikely to be fixed there. There's a more recent fork at https://github.com/almottier/TapoP100 which claims to have fixed support.

However, when I ran

pip3 install git+https://github.com/almottier/TapoP100.git@main

it broke comms with my plugs (which aren't running the new firmware).

There's a timeout hardcoded into the library which the plugs keep hitting. Manually hacking that resolved it.

The new module returns information in a slightly different format, so I've staged some changes to handle it (commit 3661a5c521f0eac07c8183882b22ebebda9b90a7).

It sounds like this actually broke a couple of months back.

My devices weren't affected because they didn't get the firmware update:

As noted here I block the devices from accessing the internet (apart from NTP, which they need for tracking daily totals) precisely because TP-Link have form for breaking (or removing) functionality with firmware updates.

I guess I should probably think about having a canary that isn't blocked so that I notice stuff like this in future

I'm not ready to merge quite yet, but have put PR 4 up in draft mode.

In an ideal world, I'd like to move to only supporting the newer module, but unfortunately it's not quite that straightforward:

• they use the same name, but functions return in different formats, so if any existing user were to update the script they'd start getting some quite confusing errors.
• I really don't like relying on pip install somerepo@main (sounds like that may be being fixed)
• That timeout is a problem

It's worth noting, too, that it looks like this can also impact Kasa devices - the python-kasa project have added support for the new KLAP Auth

That won't need a change of library though, just an update of python-kasa

The user's reported that it isn't working with their plugs

So, I've taken the Tapo off our slow cooker to use for testing.

Test setup:

Config file at ~/tmp/readings_collect/config.yml

---

# List tapo devices
tapo:
    # Tapo devices require that you log in with the credentials
    # that you use to log into the app
    #
    user: "<redacted>"
    passw: "<redacted>"
    devices:
        -
            name: "slow-cooker"
            ip: 192.168.3.164

influxdb:
    - 
        name: "local"
        url: "http://192.168.3.84:8086"
        token: ""
        org: ""
        bucket: "testing_db"

Running

$ git status
On branch p100-lib-change

$ export CONF_FILE=~/tmp/readings_collect/config.yml
$ ./app/collect.py 
Plug: slow-cooker using 0.0W, today: Not Supplied
Wrote 1 points to local

The Tapo app says that this plug is currently running firmware version 1.0.7 Build 210629 Rel.174901

The app says that it will update to 1.3.0 Build 230905 Rel. 152200 which brings with it "Enhanced local communication security."

Need to remove the firewall wall the stops the plug from connecting to the internet - without it, it won't update.

The plug got itself in a weird state, refusing to update with the app reporting "Network is unstable. Please try again".

It only worked once I unplugged it and plugged it back in.

The app says the firmware is up to date, so

$ git status
On branch p100-lib-change
nothing to commit, working tree clean

$ ./app/collect.py 
Error: {'error_code': 1003}
Failed to initialize protocol OldProtocol
Traceback (most recent call last):
  File "/home/ben/.local/lib/python3.10/site-packages/PyP100/PyP100.py", line 32, in _initialize
    protocol.Initialize()
  File "/home/ben/.local/lib/python3.10/site-packages/PyP100/auth_protocol.py", line 244, in Initialize
    result = self._request_raw("handshake", {"key": public_key})
  File "/home/ben/.local/lib/python3.10/site-packages/PyP100/auth_protocol.py", line 173, in _request_raw
    raise Exception(f"Error code: {data['error_code']}")
Exception: Error code: 1003
Failed at reading stage
Failed to communicate with device slow-cooker

OK, so we're now where the user in Issue 3 is.

Failed at reading stage

I missed this in their screenshot - that means the initial setup has worked because we log that a little later

    # If we got this far, we've connected to the device successfully
    # get the readings
    try:
        usage_dict = p110.getEnergyUsage()
    except:
        print("Failed at reading stage")
        return False, False

If I do this

    '''
    try:
        # Try using the new signature to communicate with an older device
        # if this fails, we'll fall through to using the invocation that works
        # with both libraries and newer devices (if the new library is in use)
        p110 = PyP110.P110(ip, user, passw, preferred_protocol="old")
        # The new library runs handshake/login automatically, so we don't need 
        # to invoke those if we know we're using it
        use_old_proto = True
        print("Running with old proto")
    except:
        pass
    '''

    if not use_old_proto:
        print("Using new")
        try:
            p110 = PyP110.P110(ip, user, passw)
            p110.handshake() #Creates the cookies required for further methods
            p110.login() #Sends credentials to the plug and creates AES Key and IV for further methods                    
        except:
            return False, False

    # If we got this far, we've connected to the device successfully
    # get the readings
    #try:
    if 1 == 1:
        usage_dict = p110.getEnergyUsage()

We get readings back.

Without the first try commented out, the library uses the old protocol even though the plug doesn't support it - I had thought it'd throw an exception (allowing us to enter the other conditional) but clearly not.

OK, so, path forward on this:

• Add support for a config option to denote what auth protocol to use
• Test getting energy usage with the new mode and fall back to old if that fails

It's probably best to do both - add automatic testing, but allow override via config.

OK, the first part of this is implemented

• 28d16229cf9668c07249d783b2a10cc36c98d55f : try each auth protocol in turn

Using the test config from earlier, but with an older firmware device added, we get this:

$ ./app/collect.py 
Plug: big-fridge using 170.197W, today: 0.735 kWh
Plug: slow-cooker using 0.0W, today: Not Supplied
Wrote 3 points to local

That Not Supplied concerned me, so I had a look - it's because the plug is reporting 0 for consumption, leading to a truthiness check failing. It's out of scope for this issue, so I'll look at fixing that (#8) once these changes have been merged

Adding support for an auth attribute in plug config

Valid values are

• all: the default behaviour, try the mechanisms in turn
• package_defaults: use whatever the default is for the PyP100 module in use (so new for almottier and the old mechanism for the PyPi originated module)
• old: Use the almottier old (will not function correctly if the PyPi module is in use)

With debug logs turned on and nothing set for either plug, we get

ben@optimus:~/Documents/src.old/tplink_smartplug_to_influxdb$ ./app/collect.py 
DEBUG:__main__:Auth mode is set to all
DEBUG:__main__:NewAuth Failed at login stage Failed to initialize protocol
Plug: big-fridge using 5.385W, today: 0.743 kWh
DEBUG:__main__:Auth mode is set to all
Plug: slow-cooker using 0.0W, today: Not Supplied
Wrote 3 points to local

With things configured

    devices:
        - 
            name: "big-fridge"
            ip : 192.168.3.153
            auth: "almottier_old"

        -
            name: "slow-cooker"
            ip: 192.168.3.164
            auth: "package_defaults"

Running results in

$ ./app/collect.py 
DEBUG:__main__:Auth mode is set to almottier_old
Plug: big-fridge using 5.317W, today: 0.744 kWh
DEBUG:__main__:Auth mode is set to package_defaults
Plug: slow-cooker using 0.0W, today: Not Supplied
Wrote 3 points to local

Committed in 69169adbc68d339a118b2790fa49917d75160e30

Before I merge this, I want to see whether it's possible to do anything about those timeouts.

I'd like to move the docker container to using the newer module (at some point, I'm probably going to end up with new plugs arriving that are already on a newer firmware), but that's a non-starter if polls are going to fail because of overly conservative timeouts.

Actually, scrap that.

Realistically, overriding the timeout is going to involve monkey patching. There's no good reason to dirty this work up with that, so I'll pursue that under a separate ticket + branch (raised as #9)

PR 4 is now merged