##########################################################################################

       ADBLK-19: Xiaomi Data-Grubbing

##########################################################################################


Issue Type: Bug 
-----------------------------------------------------------------------------------------

Issue Information
====================

Priority: Major					Status:      Closed
Resolution:  Done (2020-05-02 12:05:06)
Project: Adblock Lists (ADBLK)


Reported By: btasker					
Assigned To: btasker




Labels: MIUI, Xiaomi, 

Time Estimate: 0 minutes
Time Logged:   0 minutes


-----------------------------------------------------------------------------------------

Issue Description
==================

Xiaomi are using various analytics on their devices - there seem to be some new domains
associated with "SensorsData" (www.sensorsdata.cn)

Domain being blocked here is sa.api.intl.miui.com

More info at the following links

-
https://www.forbes.com/sites/thomasbrewster/2020/04/30/exclusive-warning-over-chinese-mobile-giant-xiaomi-recording-millions-of-peoples-private-web-and-phone-use/
- https://twitter.com/cybergibbons/status/1255969996641112066

-----------------------------------------------------------------------------------------

Issue Relations
================
	
	- Discovered ADBLK-21: Expose list of Regexes
	- relates to ADBLK-22: Xiaomi global zone


-----------------------------------------------------------------------------------------

Activity
==========

	
-----------------------------------------------------------------------------------------
2020-05-01 18:34:16              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: f292a66f08e75e2c4ed726f8a6a659af344723ff
Author: B Tasker <github@<Domain Hidden>>

Date: Fri May 01 18:32:19 2020 +0100
Commit Message: ADBLK-19 Block Xiaomi's SensorData domain



Modified (-)(+)
-------
config/manualblock.txt



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/f292a66f08e75e2c4ed726f8a6a659af344723ff


	
-----------------------------------------------------------------------------------------
2020-05-01 18:38:27              btasker
-----------------------------------------------------------------------------------------

Xiaomi have responded (poorly IMO) to the article here -
http://blog.mi.com/en/2020/05/02/live-post-evidence-and-statement-in-response-to-media-coverage-on-our-privacy-policy/

	
-----------------------------------------------------------------------------------------
2020-05-01 18:45:34              btasker
-----------------------------------------------------------------------------------------

There's a in-depth teardown of the claims in that article here -
https://twitter.com/cybergibbons/status/1256275170802745345
	
-----------------------------------------------------------------------------------------
2020-05-02 09:47:23              btasker
-----------------------------------------------------------------------------------------

There's a fantastic collection of Xiaomi tracking domains here -
https://gist.githubusercontent.com/unknownFalleN/3f38e2daa8a98caff1b0d965c2b89b25/raw/53035c634a699817f3ab0800102b267253912114/xiaomi_dns_block.lst


	
-----------------------------------------------------------------------------------------
2020-05-02 10:07:20              btasker
-----------------------------------------------------------------------------------------

The increased scrutiny Xiaomi have brought to their domains in my DNS logs shows that our
devices are regularly trying to resolve metok-ccc.intl.xiaomi.com.home which NXDOMAINs.

I won't add it to the adlists, but I _am_ going to blacklist it locally:

- it's extremely regular so creates log noise
- It's regularity suggests it might be a (currently failed) attempt to phone home

But, what else are the devices trying to look up?

-- BEGIN SNIPPET --

root@PIHRP1:~# grep xiaomi /var/log/pihole.log | grep query | cut -d\  -f7 | sort | uniq
-c
      4 fr.app.chat.global.xiaomi.net
      3 fr-app-chat-global-xiaomi-net1-1667981913.eu-central-1.elb.amazonaws.com
      1 fr-app-chat-global-xiaomi-net-1516654448.eu-central-1.elb.amazonaws.com
     14 metok-ccc.intl.xiaomi.com
     51 metok-ccc.intl.xiaomi.com.home
      8 metok-ccc.intl.xiaomi.com.home.home
     11 resolver.msg.global.xiaomi.net
      4 xiaomi.eu
      8 zeus.ad.intl.xiaomi.com.home

 -- END SNIPPET --

and, in yesterday's log (i.e. a full day of logs)

-- BEGIN SNIPPET --

root@PIHRP1:~# grep xiaomi /var/log/pihole.log.1 | grep query | cut -d\  -f7 | sort | uniq
-c
      4 api.ad.intl.xiaomi.com
      3 data.mistat.intl.xiaomi.com
     11 fr.app.chat.global.xiaomi.net
      1 fr-app-chat-global-xiaomi-net1-1667981913.eu-central-1.elb.amazonaws.com
     10 fr-app-chat-global-xiaomi-net-1516654448.eu-central-1.elb.amazonaws.com
     70 metok-ccc.intl.xiaomi.com.home
     19 resolver.msg.global.xiaomi.net
     25 sdkconfig.ad.intl.xiaomi.com
      2 zeus.ad.intl.xiaomi.com
     14 zeus.ad.intl.xiaomi.com.home

 -- END SNIPPET --

The ones ending .home are because the device didn't like the answer (domain blocked, most
likely) and so used the LAN's search domain instead to see if there was a record for
that.

So just for completeness, the following were blocked

-- BEGIN SNIPPET --

root@PIHRP1:~# grep xiaomi /var/log/pihole.log.1 | grep gravity | cut -d\  -f7 | sort |
uniq -c
      4 api.ad.intl.xiaomi.com
      3 data.mistat.intl.xiaomi.com
     25 sdkconfig.ad.intl.xiaomi.com
      2 zeus.ad.intl.xiaomi.com

 -- END SNIPPET --


	
-----------------------------------------------------------------------------------------
2020-05-02 10:17:10              btasker
-----------------------------------------------------------------------------------------

Of those queries yesterday, the following were not blocked

- fr.app.chat.global.xiaomi.net
- resolver.msg.global.xiaomi.net

We can see from here -
https://xiaomi.eu/community/threads/calls-home-to-the-maintainers.43699/ - that
resolver.msg.global.xiaomi.net is used to find a host to phone home to, it's essentially a
http-dns service giving JSON response bodies.

I've never used Mi Message so it shouldn't be enabled

That page also gives a list of identified domains, so lets add those

- app.chat.xiaomi.net
- data.mistat.xiaomi.com
- data.mistat.intl.xiaomi.com
- ccc.sys.miui.com
- ccc.sys.intl.miui.com
- connect.rom.miui.com
- sdkconfig.ad.xiaomi.com
- sdkconfig.ad.intl.xiaomi.com
- api.sec.intl.miui.com
- api.sec.miui.com
- auth.be.sec.miui.com
- auth.be.sec.intl.miui.com
- weatherapi.market.xioami.com
- resolver.msg.xiaomi.net

That list is missing the "chat" endpoint I hit though, so we'll include
app.chat.xiaomi.net as a zone rather than a straight domain
	
-----------------------------------------------------------------------------------------
2020-05-02 10:18:14              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: 7cdead185804388616a120b5f50afa1ef2ee2f98
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 10:16:18 2020 +0100
Commit Message: ADBLK-19 Block some more Xiaomi domains



Modified (-)(+)
-------
config/manualblock.txt
config/manualzones.txt



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/7cdead185804388616a120b5f50afa1ef2ee2f98


	
-----------------------------------------------------------------------------------------
2020-05-02 10:20:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: cf1a8d8dfed997519647caf2ff770382d1845ad6
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 10:19:30 2020 +0100
Commit Message: ADBLK-19 Consume the excellent Xiaomi domain list -
https://gist.githubusercontent.com/unknownFalleN/3f38e2daa8a98caff1b0d965c2b89b25/raw -
when building lists



Modified (-)(+)
-------
bin/update_addomains.sh



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/cf1a8d8dfed997519647caf2ff770382d1845ad6


	
-----------------------------------------------------------------------------------------
2020-05-02 10:28:14              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: aa42b6b230e804596371834c32618cb922e9bd95
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 10:26:02 2020 +0100
Commit Message: ADBLK-19 Upstream list seems to be missing the russian endpoint. See
https://twitter.com/craiu/status/1256508158161367040



Modified (-)(+)
-------
config/manualblock.txt



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/aa42b6b230e804596371834c32618cb922e9bd95


	
-----------------------------------------------------------------------------------------
2020-05-02 11:08:14              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: 2334c071a21188e94b79f16205d81d99fb63ca7e
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:06:50 2020 +0100
Commit Message: ADBLK-19 Block the zones ad.[locations] for Xiaomi



Modified (-)(+)
-------
config/manualzones.txt



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/2334c071a21188e94b79f16205d81d99fb63ca7e


	
-----------------------------------------------------------------------------------------
2020-05-02 11:12:57              btasker
-----------------------------------------------------------------------------------------

So, finally, lets run a list off from all logs

-- BEGIN SNIPPET --

root@PIHRP1:~# zgrep xiaomi /var/log/pihole.log.*.gz | grep query | cut -d\  -f6 | sort |
uniq
api.ad.intl.xiaomi.com
data.mistat.intl.xiaomi.com
fr.app.chat.global.xiaomi.net
fr-app-chat-global-xiaomi-net1-1667981913.eu-central-1.elb.amazonaws.com
fr-app-chat-global-xiaomi-net-1516654448.eu-central-1.elb.amazonaws.com
globalapi.ad.xiaomi.com
metok-ccc.intl.xiaomi.com.home
resolver.msg.global.xiaomi.net
sdkconfig.ad.intl.xiaomi.com
zeus.ad.intl.xiaomi.com
zeus.ad.intl.xiaomi.com.home

 -- END SNIPPET --

All of those should now be blocked (except the .home and Amazon's ELB anyway)
	
-----------------------------------------------------------------------------------------
2020-05-02 11:16:32              btasker
-----------------------------------------------------------------------------------------

Just for completeness, checking for miui domains too

-- BEGIN SNIPPET --

root@PIHRP1:~# zgrep miui /var/log/pihole.log.*.gz | grep query | cut -d\  -f6 | sort |
uniq
connect.rom.miui.com
tracking.intl.miui.com

 -- END SNIPPET --

The domain tracking.intl.miui.com was blocked quite a while back -
https://github.com/bentasker/adblocklists/commit/95183702 - but, given what we've seen
under xiaomi.com it's probably prudent to check if versions without the intl exist

-- BEGIN SNIPPET --

ben@milleniumfalcon:~$ host tracking.rus.miui.com
tracking.rus.miui.com has address 107.155.53.108
ben@milleniumfalcon:~$ host tracking.india.miui.com
tracking.india.miui.com is an alias for
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com.
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 15.206.87.36
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address
13.235.208.92
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 13.235.204.1
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address 3.6.193.53
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address
15.206.180.196
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address
15.206.39.100
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address
13.235.181.120
tracking-india-miui-com1-1835355922.ap-south-1.elb.amazonaws.com has address
15.206.118.188
ben@milleniumfalcon:~$ host tracking.miui.com
tracking.miui.com has address 0.0.0.0
tracking.miui.com has IPv6 address ::
ben@milleniumfalcon:~$ host tracking.gb.miui.com
ben@milleniumfalcon:~$ 

 -- END SNIPPET --

Let's block those too then.

Now, the interesting thing to note with Xiaomi is the difference between an empty DNS
response and a NXDOMAIN. If we look for a made up name, we get an NXDOMAIN

-- BEGIN SNIPPET --

ben@milleniumfalcon:~$ host tracking.fr.miui.com
Host tracking.fr.miui.com not found: 3(NXDOMAIN)

 -- END SNIPPET --

This means there's no zone fr.miui.com. But, there obviously is a gb.miui.com because we
don't get an empty response with NOERROR

-- BEGIN SNIPPET --

ben@milleniumfalcon:~$ dig tracking.gb.miui.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> tracking.gb.miui.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53897
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;tracking.gb.miui.com.		IN	A

;; AUTHORITY SECTION:
miui.com.		155	IN	SOA	ns3.dnsv5.com. enterprise3dnsadmin.dnspod.com. 1588144648 3600 180
1209600 180

;; Query time: 1 msec
;; SERVER: 192.168.3.5#53(192.168.3.5)
;; WHEN: Sat May 02 11:20:25 BST 2020
;; MSG SIZE  rcvd: 122

 -- END SNIPPET --

So, should probably think about blocking those too in case they come in use later
	
-----------------------------------------------------------------------------------------
2020-05-02 11:22:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: 24650de5b5b4b09b0d92817d45c1aa43d3f3f7bd
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:21:21 2020 +0100
Commit Message: ADBLK-19 some of the region specific Miui tracking domains



Modified (-)(+)
-------
config/manualblocks/xiaomi.txt



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/24650de5b5b4b09b0d92817d45c1aa43d3f3f7bd


	
-----------------------------------------------------------------------------------------
2020-05-02 11:22:34              btasker
-----------------------------------------------------------------------------------------

Actually, what we could really do with is creating a new type of list - a regex list. That
can then be fed into pihole, so that we can block (say) tracking\..+\.miui.com and catch
whatever regional zones they come up with

Doing that in ADBLK-21
	
-----------------------------------------------------------------------------------------
2020-05-02 11:53:19              btasker
-----------------------------------------------------------------------------------------

ADBLK-21 is now implemented, so we can use regex's to block.

A Xiaomi domain was added as part of the initial config for that

-- BEGIN SNIPPET --

^tracking\..+\.miui.com$

 -- END SNIPPET --

looking back at their earlier list of domains, we almost certainly want to do

^.+\.ad\..+\.xiaomi\.com$
^.+\.mistat\..+\.xiaomi\.com$
^adv\.sec\..+\.xiaomi.com$

	
-----------------------------------------------------------------------------------------
2020-05-02 11:54:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: 44b1451ff3f9abfc35abf3d6f418e20a1d623548
Author: B Tasker <github@<Domain Hidden>>

Date: Sat May 02 11:52:36 2020 +0100
Commit Message: ADBLK-19 Use regex functionality to block some Xiaomi tracking domains



Modified (-)(+)
-------
config/regexes/xiaomi.txt



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/44b1451ff3f9abfc35abf3d6f418e20a1d623548


	
-----------------------------------------------------------------------------------------
2020-05-02 12:04:47              btasker
-----------------------------------------------------------------------------------------

I think we're done here - any new domains they add can be added under new issues (and will
probably be picked up by the external list first anyway)
	
-----------------------------------------------------------------------------------------
2020-05-02 12:04:58              
-----------------------------------------------------------------------------------------

btasker added 'MIUI Xiaomi' to labels
	
-----------------------------------------------------------------------------------------
2020-05-02 12:05:06              
-----------------------------------------------------------------------------------------

btasker changed status from 'Open' to 'Resolved'
	
-----------------------------------------------------------------------------------------
2020-05-02 12:05:06              
-----------------------------------------------------------------------------------------

btasker added 'Done' to resolution
	
-----------------------------------------------------------------------------------------
2020-05-02 12:05:09              
-----------------------------------------------------------------------------------------

btasker changed status from 'Resolved' to 'Closed'
	
-----------------------------------------------------------------------------------------
2020-05-02 12:07:48              btasker
-----------------------------------------------------------------------------------------

Looks like Xiaomi have updated their blog in the meantime -
https://twitter.com/cybergibbons/status/1256524577406140416
	
-----------------------------------------------------------------------------------------
2020-05-03 09:35:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: adblocklists
Commit: aa6c0659db6bb43ba3a6bc8efa7fa434552a77f6
Author: B Tasker <github@<Domain Hidden>>

Date: Sun May 03 09:33:13 2020 +0100
Commit Message: ADBLK-19 Add a new regex to handle Xiaomi's "resolver" service - knew
there'd be another label appear at some point. This time it's "global" -
resolver.msg.global.xiaomi.net



Modified (-)(+)
-------
config/regexes/xiaomi.txt



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/7431eee
 -- END SNIPPET --

https://github.com/bentasker/adblocklists/commit/aa6c0659db6bb43ba3a6bc8efa7fa434552a77f6


	
-----------------------------------------------------------------------------------------
2020-05-03 10:06:48              btasker
-----------------------------------------------------------------------------------------

Picked up that domain whilst checking the Pihole logs from overnight.

The other thing that stands out, is that Xiaomi's hardware seems to have been designed
based on Toy Story - it comes alive whilst you're asleep. Interesting to see just how
regularly attempts were made to resolve tracking.intl.miui.com

-- BEGIN SNIPPET --

May  3 00:14:51 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:14:51 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:14:51 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:14:51 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:15:03 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:15:03 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:15:03 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:15:03 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:16:27 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:16:27 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:16:27 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:16:27 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:19:07 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:19:07 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:19:07 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:19:07 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:21:49 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:21:49 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:21:49 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:21:49 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:21:59 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:21:59 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:21:59 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:21:59 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:22:09 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:22:09 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:22:09 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:22:09 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:22:19 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:22:19 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:22:19 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:22:19 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:23:27 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78
May  3 00:23:27 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:23:27 dnsmasq[26416]: query[A] tracking.intl.miui.com from 192.168.3.78
May  3 00:23:27 dnsmasq[26416]: /etc/pihole/gravity.list tracking.intl.miui.com is
0.0.0.0
May  3 00:23:37 dnsmasq[26416]: query[AAAA] tracking.intl.miui.com from 192.168.3.78

 -- END SNIPPET --