ADBLK-19: Xiaomi Data-Grubbing



Issue Information

Issue Type: Bug
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: Adblock Lists (ADBLK)
Resolution: Done (2020-05-02 12:05:06)
Labels: MIUI, Xiaomi,

Created: 2020-05-01 18:31:00
Time Spent Working


Description
Xiaomi are using various analytics on their devices - there seem to be some new domains associated with "SensorsData" (www.sensorsdata.cn)

Domain being blocked here is sa.api.intl.miui.com

More info at the following links

- https://www.forbes.com/sites/thomasbrewster/2020/04/30/exclusive-warning-over-chinese-mobile-giant-xiaomi-recording-millions-of-peoples-private-web-and-phone-use/
- https://twitter.com/cybergibbons/status/1255969996641112066


Issue Links

Toggle State Changes

Activity



Repo: adblocklists
Commit: f292a66f08e75e2c4ed726f8a6a659af344723ff
Author: B Tasker <github@<Domain Hidden>>

Date: Fri May 01 18:32:19 2020 +0100
Commit Message: ADBLK-19 Block Xiaomi's SensorData domain



Modified (-)(+)
-------
config/manualblock.txt




Webhook User-Agent

GitHub-Hookshot/7431eee


View Commit

There's a in-depth teardown of the claims in that article here - https://twitter.com/cybergibbons/status/1256275170802745345
The increased scrutiny Xiaomi have brought to their domains in my DNS logs shows that our devices are regularly trying to resolve metok-ccc.intl.xiaomi.com.home which NXDOMAINs.

I won't add it to the adlists, but I am going to blacklist it locally:

- it's extremely regular so creates log noise
- It's regularity suggests it might be a (currently failed) attempt to phone home

But, what else are the devices trying to look up?
root@PIHRP1:~# grep xiaomi /var/log/pihole.log | grep query | cut -d\  -f7 | sort | uniq -c
      4 fr.app.chat.global.xiaomi.net
      3 fr-app-chat-global-xiaomi-net1-1667981913.eu-central-1.elb.amazonaws.com
      1 fr-app-chat-global-xiaomi-net-1516654448.eu-central-1.elb.amazonaws.com
     14 metok-ccc.intl.xiaomi.com
     51 metok-ccc.intl.xiaomi.com.home
      8 metok-ccc.intl.xiaomi.com.home.home
     11 resolver.msg.global.xiaomi.net
      4 xiaomi.eu
      8 zeus.ad.intl.xiaomi.com.home


and, in yesterday's log (i.e. a full day of logs)
root@PIHRP1:~# grep xiaomi /var/log/pihole.log.1 | grep query | cut -d\  -f7 | sort | uniq -c
      4 api.ad.intl.xiaomi.com
      3 data.mistat.intl.xiaomi.com
     11 fr.app.chat.global.xiaomi.net
      1 fr-app-chat-global-xiaomi-net1-1667981913.eu-central-1.elb.amazonaws.com
     10 fr-app-chat-global-xiaomi-net-1516654448.eu-central-1.elb.amazonaws.com
     70 metok-ccc.intl.xiaomi.com.home
     19 resolver.msg.global.xiaomi.net
     25 sdkconfig.ad.intl.xiaomi.com
      2 zeus.ad.intl.xiaomi.com
     14 zeus.ad.intl.xiaomi.com.home


The ones ending .home are because the device didn't like the answer (domain blocked, most likely) and so used the LAN's search domain instead to see if there was a record for that.

So just for completeness, the following were blocked
root@PIHRP1:~# grep xiaomi /var/log/pihole.log.1 | grep gravity | cut -d\  -f7 | sort | uniq -c
      4 api.ad.intl.xiaomi.com
      3 data.mistat.intl.xiaomi.com
     25 sdkconfig.ad.intl.xiaomi.com
      2 zeus.ad.intl.xiaomi.com


Of those queries yesterday, the following were not blocked

- fr.app.chat.global.xiaomi.net
- resolver.msg.global.xiaomi.net

We can see from here - https://xiaomi.eu/community/threads/calls-home-to-the-maintainers.43699/ - that resolver.msg.global.xiaomi.net is used to find a host to phone home to, it's essentially a http-dns service giving JSON response bodies.

I've never used Mi Message so it shouldn't be enabled

That page also gives a list of identified domains, so lets add those

- app.chat.xiaomi.net
- data.mistat.xiaomi.com
- data.mistat.intl.xiaomi.com
- ccc.sys.miui.com
- ccc.sys.intl.miui.com
- connect.rom.miui.com
- sdkconfig.ad.xiaomi.com
- sdkconfig.ad.intl.xiaomi.com
- api.sec.intl.miui.com
- api.sec.miui.com
- auth.be.sec.miui.com
- auth.be.sec.intl.miui.com
- weatherapi.market.xioami.com
- resolver.msg.xiaomi.net

That list is missing the "chat" endpoint I hit though, so we'll include app.chat.xiaomi.net as a zone rather than a straight domain

Repo: adblocklists
Commit: 7cdead185804388616a120b5f50afa1ef2ee2f98
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 10:16:18 2020 +0100
Commit Message: ADBLK-19 Block some more Xiaomi domains



Modified (-)(+)
-------
config/manualblock.txt
config/manualzones.txt




Webhook User-Agent

GitHub-Hookshot/7431eee


View Commit


Repo: adblocklists
Commit: cf1a8d8dfed997519647caf2ff770382d1845ad6
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 10:19:30 2020 +0100
Commit Message: ADBLK-19 Consume the excellent Xiaomi domain list - https://gist.githubusercontent.com/unknownFalleN/3f38e2daa8a98caff1b0d965c2b89b25/raw - when building lists



Modified (-)(+)
-------
bin/update_addomains.sh




Webhook User-Agent

GitHub-Hookshot/7431eee


View Commit


Repo: adblocklists
Commit: aa42b6b230e804596371834c32618cb922e9bd95
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 10:26:02 2020 +0100
Commit Message: ADBLK-19 Upstream list seems to be missing the russian endpoint. See https://twitter.com/craiu/status/1256508158161367040



Modified (-)(+)
-------
config/manualblock.txt




Webhook User-Agent

GitHub-Hookshot/7431eee


View Commit


Repo: adblocklists
Commit: 2334c071a21188e94b79f16205d81d99fb63ca7e
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:06:50 2020 +0100
Commit Message: ADBLK-19 Block the zones ad.[locations] for Xiaomi



Modified (-)(+)
-------
config/manualzones.txt




Webhook User-Agent

GitHub-Hookshot/7431eee


View Commit

So, finally, lets run a list off from all logs
root@PIHRP1:~# zgrep xiaomi /var/log/pihole.log.*.gz | grep query | cut -d\  -f6 | sort | uniq
api.ad.intl.xiaomi.com
data.mistat.intl.xiaomi.com
fr.app.chat.global.xiaomi.net
fr-app-chat-global-xiaomi-net1-1667981913.eu-central-1.elb.amazonaws.com
fr-app-chat-global-xiaomi-net-1516654448.eu-central-1.elb.amazonaws.com
globalapi.ad.xiaomi.com
metok-ccc.intl.xiaomi.com.home
resolver.msg.global.xiaomi.net
sdkconfig.ad.intl.xiaomi.com
zeus.ad.intl.xiaomi.com
zeus.ad.intl.xiaomi.com.home


All of those should now be blocked (except the .home and Amazon's ELB anyway)
Just for completeness, checking for miui domains too
root@PIHRP1:~# zgrep miui /var/log/pihole.log.*.gz | grep query | cut -d\  -f6 | sort | uniq
connect.rom.miui.com
tracking.intl.miui.com


The domain tracking.intl.miui.com was blocked quite a while back - https://github.com/bentasker/adblocklists/commit/95183702 - but, given what we've seen under xiaomi.com it's probably prudent to check if versions without the intl exist
ben@milleniumfalcon:~$ host tracking.rus.miui.com
tracking.rus.miui.com has address 107.155.53.108
ben@milleniumfalcon:~$ host tracking.india.miui.com
tracking.india.miui.com is an alias for tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com.
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 15.206.87.36
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 13.235.208.92
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 13.235.204.1
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 3.6.193.53
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 15.206.180.196
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 15.206.39.100
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 13.235.181.120
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 15.206.118.188
ben@milleniumfalcon:~$ host tracking.miui.com
tracking.miui.com has address 0.0.0.0
tracking.miui.com has IPv6 address ::
ben@milleniumfalcon:~$ host tracking.gb.miui.com
ben@milleniumfalcon:~$ 


Let's block those too then.

Now, the interesting thing to note with Xiaomi is the difference between an empty DNS response and a NXDOMAIN. If we look for a made up name, we get an NXDOMAIN
ben@milleniumfalcon:~$ host tracking.fr.miui.com
Host tracking.fr.miui.com not found: 3(NXDOMAIN)


This means there's no zone fr.miui.com. But, there obviously is a gb.miui.com because we don't get an empty response with NOERROR
ben@milleniumfalcon:~$ dig tracking.gb.miui.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> tracking.gb.miui.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53897
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;tracking.gb.miui.com.		IN	A

;; AUTHORITY SECTION:
miui.com.		155	IN	SOA	ns3.dnsv5.com. enterprise3dnsadmin.dnspod.com. 1588144648 3600 180 1209600 180

;; Query time: 1 msec
;; SERVER: 192.168.3.5#53(192.168.3.5)
;; WHEN: Sat May 02 11:20:25 BST 2020
;; MSG SIZE  rcvd: 122


So, should probably think about blocking those too in case they come in use later

Repo: adblocklists
Commit: 24650de5b5b4b09b0d92817d45c1aa43d3f3f7bd
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:21:21 2020 +0100
Commit Message: ADBLK-19 some of the region specific Miui tracking domains



Modified (-)(+)
-------
config/manualblocks/xiaomi.txt




Webhook User-Agent

GitHub-Hookshot/7431eee


View Commit

Actually, what we could really do with is creating a new type of list - a regex list. That can then be fed into pihole, so that we can block (say) tracking\..+\.miui.com and catch whatever regional zones they come up with

Doing that in ADBLK-21
ADBLK-21 is now implemented, so we can use regex's to block.

A Xiaomi domain was added as part of the initial config for that
^tracking\..+\.miui.com$


looking back at their earlier list of domains, we almost certainly want to do

^.+\.ad\..+\.xiaomi\.com$
^.+\.mistat\..+\.xiaomi\.com$
^adv\.sec\..+\.xiaomi.com$

Repo: adblocklists
Commit: 44b1451ff3f9abfc35abf3d6f418e20a1d623548
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:52:36 2020 +0100
Commit Message: ADBLK-19 Use regex functionality to block some Xiaomi tracking domains



Modified (-)(+)
-------
config/regexes/xiaomi.txt




Webhook User-Agent

GitHub-Hookshot/7431eee


View Commit

I think we're done here - any new domains they add can be added under new issues (and will probably be picked up by the external list first anyway)
btasker added 'MIUI Xiaomi' to labels
btasker changed status from 'Open' to 'Resolved'
btasker added 'Done' to resolution
btasker changed status from 'Resolved' to 'Closed'
Looks like Xiaomi have updated their blog in the meantime - https://twitter.com/cybergibbons/status/1256524577406140416

Repo: adblocklists
Commit: aa6c0659db6bb43ba3a6bc8efa7fa434552a77f6
Author: B Tasker <github@<Domain Hidden>>

Date: Sun May 03 09:33:13 2020 +0100
Commit Message: ADBLK-19 Add a new regex to handle Xiaomi's "resolver" service - knew there'd be another label appear at some point. This time it's "global" - resolver.msg.global.xiaomi.net



Modified (-)(+)
-------
config/regexes/xiaomi.txt




Webhook User-Agent

GitHub-Hookshot/7431eee


View Commit

Picked up that domain whilst checking the Pihole logs from overnight.

The other thing that stands out, is that Xiaomi's hardware seems to have been designed based on Toy Story - it comes alive whilst you're asleep. Interesting to see just how regularly attempts were made to resolve tracking.intl.miui.com
May  3 00:14:51 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:14:51 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:14:51 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:14:51 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:15:03 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:15:03 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:15:03 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:15:03 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:16:27 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:16:27 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:16:27 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:16:27 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:19:07 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:19:07 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:19:07 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:19:07 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:21:49 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:21:49 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:21:49 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:21:49 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:21:59 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:21:59 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:21:59 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:21:59 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:22:09 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:22:09 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:22:09 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:22:09 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:22:19 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:22:19 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:22:19 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:22:19 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:23:27 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:23:27 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:23:27 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:23:27 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is 0.0.0.0
May  3 00:23:37 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78