project Utilities / zepp_to_influxdb avatar

utilities/zepp_to_influxdb#4: Collect Heart Rate

Issue Information

Issue Type: issue
Status: closed
Reported By: btasker

Milestone: v0.4
Created: 03-Aug-23 15:24



Description

It's not currently possible to capture Heart Rate data, raising this ticket to log the information I've collected so far.

heartRate endpoint

The app places requests to a heart rate endpoint:

  • Domain: api-mifit-de2.zepp.com
  • Path: /users//heartRate

Query args:

device=android_21
v=2.0
device_type=android_phone
cv=50724_6.7.7
country=GB
lang=en_GB
appid=428135909242707968
endTime=1691103599999
startTime=1691017200000
type=2
channel=play
userid=<redacted>
timezone=Europe%2FLondon

However this always seems to result in an empt dataset

{
  "items": []
}

band_data detail view

When the heart rate graphs are loaded in the app, a call seems to be made to the detail view of band_data.json:

  • Domain: api-mifit-de2.zepp.com
  • Path: /v1/data/band_data.json

Query params:

userid=<redacted>
appid=428135909242707968
byteLength=8
channel=play
country=GB
cv=50724_6.7.7
device=android_21
device_type=android_phone
from_date=2023-07-31
lang=en_GB
query_type=detail
timezone=Europe%2FLondon
to_date=2023-07-31
v=2.0

This results in a response structure similar to the existing band_data.json calls. However, each entry's dict also has a data attribute.

It appears to be a base64 encoded binary blob:

"data": "fgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAAfgAA
..snip..

One possibility is that this is a blob of raw values. If we look at a graph where readings were only available later in the day and then hexdump the resulting blob we do seem to get a long stream of 0's

ben@optimus:~/Documents/src.old/zepp_to_influxdb$ xxd /tmp/blob 
00000000: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..~
00000010: 0000 7e00 007e 0000 7e00 007e 0000 7e00  ..~..~..~..~..~.
00000020: 007e 0000 7e00 007e 0000 7e00 007e 0000  .~..~..~..~..~..
00000030: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..~
00000040: 0000 7e00 007e 0000 7e00 007e 0000 7e00  ..~..~..~..~..~.
00000050: 007e 0000 7e00 007e 0000 7e00 007e 0000  .~..~..~..~..~..
00000060: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..~
00000070: 0000 7e00 007e 0000 7e00 007e 0000 7e00  ..~..~..~..~..~.
00000080: 007e 0000 7e00 007e 0000 7e00 007e 0000  .~..~..~..~..~..
00000090: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..~
000000a0: 0000 7e00 007e 0000 7e00 007e 0000 7e00  ..~..~..~..~..~.
000000b0: 007e 0000 7e00 007e 0000 7e00 007e 0000  .~..~..~..~..~..
000000c0: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..~
000000d0: 0000 7e00 007e 0000 7e00 007e 0000 7e00  ..~..~..~..~..~.
000000e0: 007e 0000 7e00 007e 0000 7e00 007e 0000  .~..~..~..~..~..
000000f0: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..~
00000100: 0000 7e00 007e 0000 7e00 007e 0000 7e00  ..~..~..~..~..~.
00000110: 007e 0000 7e00 007e 0000 7e00 007e 0000  .~..~..~..~..~..
00000120: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..~
00000130: 0000 7e00 007e 0000 7e00 007e 0000 7e00  ..~..~..~..~..~.
00000140: 007e 0000 7e00 007e 0000 7e00 007e 0000  .~..~..~..~..~..
00000150: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..~
00000160: 0000 7e00 007e 0000 7e00 007e 0000 7e00  ..~..~..~..~..~.
00000170: 007e 0000 7e00 007e 0000 7e00 007e 0000  .~..~..~..~..~..
00000180: 7e00 007e 0000 7e00 007e 0000 7e00 007e  ~..~..~..~..~..

The graph seems to group into 10 minute windows, so it may be worth iterating through to see whether the number of values aligns.

I had initially thought that 7e might be a delimiter, but it stops appearing part way through the data:

000006e0: 0050 1400 5019 0050 0c00 5012 0050 1500  .P..P..P..P..P..
000006f0: 5007 0050 0b00 5a00 0050 0300 5013 0050  P..P..Z..P..P..P
00000700: 2400 5048 0460 660e 6047 0b01 4d29 1040  $.PH.`f.`G..M).@
00000710: 0960 3400 0158 1060 4300 6063 0d01 7566  .`4..X.`C.`c..uf
00000720: 016f 6201 7163 0173 6101 5b35 0182 6310  .ob.qc.sa.[5..c.
00000730: 3a09 604d 0c01 522c 104b 0860 3400 0181  :.`M..R,.K.`4...
00000740: 6701 4f30 014d 1d01 4c1c 0171 3401 703c  g.O0.M..L..q4.p<
00000750: 0167 4810 4707 015a 1b01 6a5d 017d 4c01  .gH.G..Z..j].}L.
00000760: 7d67 0172 4a01 501f 0177 6410 1600 6029  }g.rJ.P..wd...`)
00000770: 0001 653c 1034 0060 2900 6019 0060 2600  ..e<.4.`).`..`&.

Within the JSON there's also a data_hr attribute which is similarly odd

"data_hr":"/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7+/v7/////Wf////////9cTEZu////Xl9cYGthUEz/YF7/d3VyTl5JTFf/Z2RpbWRkb/9gY/////9T/////////1JbZ0hLV2P/////UmL/U/////////9T////////////Sv///////////0P///////////9T////////////VP///////////zz///////////9L////////////S////////////1P//

Much like the data blob it starts with 0s

ben@optimus:~/Documents/src.old/zepp_to_influxdb$ xxd /tmp/blob2 
00000000: fa00 00f0 0000 f000 00f0 0000 f000 00fa  ................
00000010: 0000 fa00 00f0 0000 f000 00f0 0000 f000  ................
00000020: 00f0 0000 f000 00f0 0000 f000 00f0 0000  ................
00000030: f911 00f0 0000 f000 00f0 0000 f000 00f0  ................
00000040: 0000 f000 00f0 0000 f000 00f0 0000 f000  ................
00000050: 00f0 0000 f000 00f0 0000 f000 00f0 0000  ................
00000060: f000 00f0 0000 f000 00f0 0000 f000 00f0  ................
00000070: 0000 f000 00f0 0000 f000 00f0 0000 f000  ................
00000080: 00f0 0000 f000 00f0 0000 f000 00f0 0a00  ................
00000090: f000 00f0 0000 f000 00f0 0000 f002 00f0  ................
000000a0: 0000 f000 00fa 0000 f000 00f0 0000 f000  ................
000000b0: 00f0 0500 f01e 00f9 0e00 f000 00f0 0000  ................
000000c0: fa00 00f0 0200 f000 00fa 0000 f000 00f0  ................
000000d0: 0000 700e 007a 0000 7a00 007a 0000 700d  ..p..z..z..z..p.

It'll probably turn out to be really simple, but I just don't have the time/energy to hammer at it at the moment



Toggle State Changes

Activity


04-Aug-23 13:11

I flashed and rooted android 9 on to a spare phone and installed the Zepp app so that I could MiTM it.

The requests are exactly that same as those made by Zepp life (presumably they're using a common SDK), so unfortunately it hasn't really revealed any additional information.


07-Aug-23 12:30

Quick bit of lunchtime hacking: I wrote a small python script to read the binary values, convert to int and output lp

#!/usr/bin/env python3
import time

int_vals = []
with open("/tmp/blob", "rb") as f:
    while(byte := f.read(1)):
        int_vals.append(int.from_bytes(byte, "big"))

ts = 1691362800
for val in int_vals:
    ts += 60
    print(f"zeppblobtest4,foo=bar val={val} {ts}000000000")

The Zepp app heartrate for that particular day looks like this

Screenshot_2023-08-07-12-32-36-683_com.huami.watch.hmwatchmanager

The LP import looks like this (with some transforms tidying shit up)

Screenshot_20230807_131541

So, I think I'm probably on the right track but am still missing something.

I'm wondering if the answer might actually be to ignore some of the entries.

There are a least a couple of bits the app exposes which don't seem to correlate with API calls (other than this detail one):

  • Heart rate
  • PAI score

The opening values are

[126, 0, 0, 126, 0, 0, 126, 0, 0,

There's clearly a repeating pattern there, so perhaps what we're looking at is actually 3 columns of data?

#!/usr/bin/env python3
import time

int_vals = {1: [],
         2: [],
         3: []
         }
x = 1
with open("/tmp/blob", "rb") as f:
    while(byte := f.read(1)):
        int_vals[x].append(int.from_bytes(byte, "big"))
        x += 1
        if x == 4:
            x = 1

for x in [1,2,3]:
    ts = 1691362800
    for val in int_vals[x]:
        ts += 60
        print(f"zeppblobtest5,foo=bar val{x}={val} {ts}000000000")

val1:

Screenshot_20230807_132809

val2:

Screenshot_20230807_132738

val3:

Screenshot_20230807_132832

The values don't seem to align with the heart-rate graph but the timing on val2 does.

I think this probably deserves more investigation, I'll spin out a ticket later


07-Aug-23 19:02

Re-opening, as it doesn't make sense to fragment the information and I've just figured this out.

I was looking along the right lines, and needed to figure out the final dots to put them together.

So, the detail view of band_data.json returns a JSON object with the following structure

{
  "code": 1,
  "message": "success",
  "data": [
    {
      "uid": "<redacted>",
      "data_type": 0,
      "date_time": "2023-08-01",
      "source": 256,
      "summary": "<base64'd json object>",
      "uuid": "null",
      "data": "<base64'd binary blob>",
      "data_hr" :  "<base64'd binary blob>",
   }
 ]
}

Each object within the array data relates to a single day.

In each of those objects, data_hr is a binary stream of values representing 1 heart rate measurement per minute.

The crucial bit that I was missing above is the type of these values - the app (and presumably) the API both use Java, and the developer has used a variable of type short.

So, we need to read 2 bytes for each value.

The following python reads a base64 decoded dump off disk and converts it into a line of line protocol per reading

#!/usr/bin/env python3
import time
import sys

adjusted_vals = []
x = 1
b=b''
with open("/tmp/blob2", "rb") as f:
    while(byte := f.read(1)):
        #int_vals[x].append(int.from_bytes(byte, "big"))
        x += 1
        b += byte
        if x == 2:
            # Reset the counter
            x = 1

            # Convert to an int
            v = int(b.hex(), 16)
            if v < 200:
                adjusted_vals.append(v)
            else:
                # Set an initialisation value
                adjusted_vals.append(9999)

            # Reset the byte string
            b = b''


ts = 1691362800
for x in adjusted_vals:
    ts += 60
    if x == 9999:
        continue
    print(f"zeppheartrate,foo=bar hr={x} {ts}000000000")

When queried:

Screenshot_20230807_192051

For reference, the app graph for that day

Screenshot_20230807_200033

So, we've now got a PoC, just need to translate it into code to read from the API response.

As a follow up action, need to run data through the same logic and generate a graph - that way I can try and see what the shape of it matches up to in the app in order to identify what data it represents.

Update: it seems to be 0 or 126 throughout that day... so maybe whatever it is is something my Bip doesn't do (or perhaps it isn't a short)


08-Aug-23 07:46

The following python successfully loads the JSON dump I've got on disk and translates the binary blob into the exact same figures as the PoC script does from the dumped binary

#!/usr/bin/env python3
import datetime as dt
import base64
import json


def translate_stats(daydata):
    ''' Extract the heart rate data blob from the JSON
    and convert to a list of stats

    TODO: work out and append times
    '''

    # Create a datetime object from the date specified in JSON
    # this will be midnight.
    nowtime = dt.datetime.strptime(daydata['date_time'], "%Y-%m-%d")

    number_blob = bytearray(base64.b64decode(daydata['data_hr']))
    #print(number_blob)
    adjusted_vals = []

    # Initialise values
    x = 1
    b=b''    

    # Iterate through the bytestring
    for byte_i in number_blob:
        # iterating over leads to us fetching ints
        # not bytes, so convert back
        byte = byte_i.to_bytes(length=1, byteorder="big")

        # Concatenate this byte onto the previous
        b += byte

        # Move the marker to the right
        x += 1

        # The data is a java short, so every
        # 2 bytes, convert it to an integer
        if x == 2:            
            # Convert the bytestring to an int
            v = int(b.hex(), 16)

            # Adjust the timestamp forward 1 minute
            nowtime = nowtime + dt.timedelta(minutes=1)

            # They seem to use a high initialisation
            # value to indicate lack of data. If it's
            # higher than 200 set it really high so
            # we know to skip it
            if v < 200:
                adjusted_vals.append([nowtime.strftime('%s'), v])
            else:
                # Set an initialisation value
                adjusted_vals.append([nowtime.strftime('%s'), 9999])

            # Reset the byte string
            b = b''
            # Reset the counter
            x = 1

    print(adjusted_vals)


# Load the JSON from disk
fh = open("json.txt", "r")
j = json.load(fh)
fh.close()

#print(j)

for dayentry in j['data']:
    print(translate_stats(dayentry))

So, it just needs adjusting and adding to the codebase.

The detail call to band_data.json includes the summary attribute so, I think, rather than adding a second API call we can just adjust the one that's currently used to retrieve step counts etc.


08-Aug-23 07:59
verified

mentioned in commit 51fd5a2c46738dc043ebb6fa17860d168f1342c5

Commit: 51fd5a2c46738dc043ebb6fa17860d168f1342c5 
Author: B Tasker                            
                            
Date: 2023-08-08T08:58:41.000+01:00

Message

Extract heart rate data from the band_data.json detail view (utilities/zepp_to_influxdb#4)

+70 -2 (72 lines changed)

08-Aug-23 08:00

I've successfully retrieved HR data for the past few days using the commit above Screenshot_20230808_090007


08-Aug-23 08:02

This change adds one new field and one new tag

  • tag: hr_measure - currently static (periodic)
  • field: heart_rate, integer

The tag denotes that it's a heart-rate measurement taken as a result of the watch's continuous monitoring. If/when we get to looking at activity specific stats, I imagine there'll be another set of heartrate stats associated with those.


08-Aug-23 12:09

changed title from Heart Rate to {+Collect +}Heart Rate