########################################################################################## HLS-35: Playback of encrypted streams fails on Macs (inc VLC) ########################################################################################## Issue Type: Bug ----------------------------------------------------------------------------------------- Issue Information ==================== Priority: Major Status: Closed Resolution: Fixed (2019-11-16 12:38:36) Project: HLS Stream Creator (HLS) Reported By: btasker Assigned To: btasker Affected Versions: - 1.0 Targeted for fix in version: - 1.0 Time Estimate: 0 minutes Time Logged: 0 minutes ----------------------------------------------------------------------------------------- Issue Description ================== Github #34 reported that it wasn't possible to playback an encrypted stream using Safari or VLC on a Mac. Reproducing the info in that ticket here for posterity, but this also needs fixing. *User Report* created encrypted parts using -- BEGIN SNIPPET -- hls-stream-creater.sh -i ./small.mp4 -e -s 10 ./ -- END SNIPPET -- it created successfully but now its not playing. m3u8 file contents: -- BEGIN SNIPPET -- #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-ALLOW-CACHE:YES #EXT-X-KEY:METHOD=AES-128,URI=small.mp4.key #EXT-X-TARGETDURATION:6 #EXTINF:5.600000, small.mp4_00000.ts #EXT-X-ENDLIST -- END SNIPPET -- Developer tools in Safari simply reports Failed to load resource: Plug-in handled load Investigation details are provided in comments below, but the root cause has boiled down to the keyfile not being in quotes. ----------------------------------------------------------------------------------------- Issue Relations ================ - Github #34 (https://github.com/bentasker/HLS-Stream-Creator/issues/34) - RFC 8216 (https://tools.ietf.org/html/rfc8216) ----------------------------------------------------------------------------------------- Activity ========== ----------------------------------------------------------------------------------------- 2019-11-16 12:16:18 btasker ----------------------------------------------------------------------------------------- VLC's log is -- BEGIN SNIPPET -- main debug: VLC media player - 3.0.8 Vetinari main debug: Copyright © 1996-2019 the VideoLAN team main debug: revision 3.0.8-0-gf350b6b5a7 main debug: configured with /Users/d-fu/vlc-3.0/extras/package/macosx/../../../configure '--prefix=/Users/d-fu/vlc-3.0/build/vlc_install_dir' '--enable-macosx' '--enable-merge-ffmpeg' '--enable-osx-notifications' '--enable-faad' '--enable-flac' '--enable-theora' '--enable-shout' '--enable-ncurses' '--enable-twolame' '--enable-realrtsp' '--enable-libass' '--enable-macosx-qtkit' '--enable-macosx-avfoundation' '--disable-skins2' '--disable-xcb' '--disable-caca' '--disable-pulse' '--disable-sdl-image' '--disable-vnc' '--build=x86_64-apple-darwin17' '--host=x86_64-apple-darwin17' '--with-macosx-version-min=10.7' '--with-macosx-sdk=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk' '--with-breakpad=https://mac.crashes.videolan.org' 'build_alias=x86_64-apple-darwin17' 'host_alias=x86_64-apple-darwin17' 'CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang' 'CFLAGS=-g' 'LDFLAGS=' 'CXX=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++' 'CXXFLAGS=-g' 'OBJC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang' 'OBJCFLAGS=-g' main debug: Creating an input for preparsing 'small.mp4.m3u8' main debug: processing request item: small.mp4.m3u8, node: Playlist, skip: 0 main debug: rebuilding array of current - root Playlist main debug: rebuild done - 1 items, index 0 main debug: starting playback of new item main debug: resyncing on small.mp4.m3u8 main debug: small.mp4.m3u8 is at 0 main debug: creating new input thread macosx debug: Enabling media key support macosx warning: Failed to enable media key support, likely app needs to be whitelisted in Security Settings. main debug: looking for meta fetcher module matching "any": 1 candidates main debug: Creating an input for 'small.mp4.m3u8' lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/fetcher lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/fetcher lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/fetcher main debug: no meta fetcher modules matched main debug: looking for art finder module matching "any": 2 candidates lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/art main debug: requesting art for new input thread lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/art lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/art main debug: using timeshift granularity of 50 MiB main debug: using default timeshift path main debug: `file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8' gives access `file' demux `any' path `/Users/Admin/Desktop/vid_hls/small.mp4.m3u8' lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/00_musicbrainz.luac main debug: creating demux: access='file' demux='any' location='/Users/Admin/Desktop/vid_hls/small.mp4.m3u8' file='/Users/Admin/Desktop/vid_hls/small.mp4.m3u8' main debug: looking for access_demux module matching "file": 14 candidates main debug: no access_demux modules matched main debug: creating access: file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8 main debug: (path: /Users/Admin/Desktop/vid_hls/small.mp4.m3u8) main debug: looking for access module matching "file": 23 candidates main debug: using access module "filesystem" main debug: looking for stream_filter module matching "prefetch,cache_read": 26 candidates cache_read debug: Using stream method for AStream* cache_read debug: starting pre-buffering cache_read debug: received first data after 0 ms cache_read debug: pre-buffering done 192 bytes in 0s - 2314 KiB/s main debug: using stream_filter module "cache_read" main debug: looking for stream_filter module matching "any": 26 candidates lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/playlist lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/playlist lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/playlist lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/00_musicbrainz.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/anevia_streams.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/01_googleimage.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/anevia_xml.luac lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/01_googleimage.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/02_frenchtv.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/appletrailers.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/bbc_co_uk.luac lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/02_frenchtv.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/03_lastfm.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/cue.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/dailymotion.luac lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/03_lastfm.luac main debug: no art finder modules matched lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/jamendo.luac main debug: looking for meta fetcher module matching "any": 1 candidates lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/fetcher lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/fetcher lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/fetcher main debug: no meta fetcher modules matched main debug: looking for art finder module matching "any": 2 candidates lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/art lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/art lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/art lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/00_musicbrainz.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/koreus.luac lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/00_musicbrainz.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/01_googleimage.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/liveleak.luac lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/01_googleimage.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/02_frenchtv.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/newgrounds.luac lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/02_frenchtv.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/03_lastfm.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/rockbox_fm_presets.luac lua debug: skipping script (unmatched scope) /Applications/VLC.app/Contents/MacOS/share/lua/meta/art/03_lastfm.luac main debug: no art finder modules matched lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/soundcloud.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/twitch.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/vimeo.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/vocaroo.luac lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/playlist/youtube.luac main debug: no stream_filter modules matched main debug: looking for stream_directory module matching "any": 1 candidates main debug: no stream_directory modules matched main debug: attachment of directory-extractor failed for file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8 main debug: looking for stream_filter module matching "record": 26 candidates main debug: using stream_filter module "record" main debug: creating demux: access='file' demux='any' location='/Users/Admin/Desktop/vid_hls/small.mp4.m3u8' file='/Users/Admin/Desktop/vid_hls/small.mp4.m3u8' main debug: looking for demux module matching "any": 55 candidates adaptive debug: Retrieving file:///Users/Admin/Desktop/vid_hls/mall.mp4.ke @0 main debug: creating access: file:///Users/Admin/Desktop/vid_hls/mall.mp4.ke main debug: (path: /Users/Admin/Desktop/vid_hls/mall.mp4.ke) main debug: looking for access module matching "file": 23 candidates filesystem error: cannot open file /Users/Admin/Desktop/vid_hls/mall.mp4.ke (No such file or directory) main debug: no access modules matched main error: no suitable access module for `file:///Users/Admin/Desktop/vid_hls/mall.mp4.ke' adaptive debug: Period adaptive debug: BaseAdaptationSet default_id#0 adaptive debug: Representation file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8 adaptive debug: Segment #1 url=file:///Users/Admin/Desktop/vid_hls/small.mp4_00000.ts duration 560 adaptive debug: opening playlist file (/Users/Admin/Desktop/vid_hls/small.mp4.m3u8) main debug: using demux module "adaptive" main debug: looking for a subtitle file in /Users/Admin/Desktop/vid_hls/ main debug: looking for meta reader module matching "any": 2 candidates adaptive error: Failed to create demuxer 0x0 Unknown lua debug: Trying Lua scripts in /Users/Admin/Library/Application Support/org.videolan.vlc/lua/meta/reader lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/lua/meta/reader lua debug: Trying Lua scripts in /Applications/VLC.app/Contents/MacOS/share/lua/meta/reader lua debug: Trying Lua playlist script /Applications/VLC.app/Contents/MacOS/share/lua/meta/reader/filename.luac main debug: no meta reader modules matched main debug: `file:///Users/Admin/Desktop/vid_hls/small.mp4.m3u8' successfully opened main debug: EOF reached main debug: removing module "adaptive" main debug: removing module "record" main debug: removing module "cache_read" main debug: removing module "filesystem" main debug: dead input main debug: changing item without a request (current 0/1) main debug: nothing to play -- END SNIPPET -- ----------------------------------------------------------------------------------------- 2019-11-16 12:17:24 btasker ----------------------------------------------------------------------------------------- User provided a copy of the stream and I was able to decrypt and play just fine on Linux Key-length is correct -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ wc -c small.mp4.key 16 small.mp4.key -- END SNIPPET -- Segment decrypts happily enough using the key -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ cp small.mp4_00000.ts test.ts ben@milleniumfalcon:~/tmp/HLS$ IV=$(printf '%032x' 00000) ben@milleniumfalcon:~/tmp/HLS$ KEY=$(cat small.mp4.key | hexdump -e '16/1 "%02x"') ben@milleniumfalcon:~/tmp/HLS$ openssl aes-128-cbc -d -in test.ts -out test.dc.ts -K $KEY -iv $IV ben@milleniumfalcon:~/tmp/HLS$ ffprobe -hide_banner -i test.dc.ts Input #0, mpegts, from 'test.dc.ts': Duration: 00:00:05.53, start: 1.466667, bitrate: 350 kb/s Program 1 Metadata: service_name : Service01 service_provider: FFmpeg Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 560x320, 30 fps, 30 tbr, 90k tbn, 60 tbc Stream #0:1[0x101](eng): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, mono, fltp, 72 kb/s -- END SNIPPET -- and actually, VLC playback works fine for me: -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ cvlc small.mp4.m3u8 VLC media player 2.2.2 Weatherwax (revision 2.2.2-0-g6259d80) [0000000000fc11d8] dummy interface: using the dummy interface module... [00007fb888c01518] httplive stream: HTTP Live Streaming (/home/ben/tmp/HLS/small.mp4.m3u8) [00007fb888c01fc8] ts demux: MPEG-4 descriptor not found for pid 0x101 type 0xf [00007fb888ddbb78] packetizer_mpeg4audio packetizer: AAC channels: 1 samplerate: 48000 [00007fb888c84718] avcodec decoder: Using NVIDIA VDPAU Driver Shared Library 340.104 Thu Sep 14 16:45:03 PDT 2017 for hardware decoding. -- END SNIPPET -- ----------------------------------------------------------------------------------------- 2019-11-16 12:19:09 btasker ----------------------------------------------------------------------------------------- I've managed to find an old macbook and there's something very weird going on here (VLC version is latest Vetinari). It's truncating letters off the key-filename: -- BEGIN SNIPPET -- no suitable access module for 'http://scratch.holly.home:80/HLS/mall.mp4.key' -- END SNIPPET -- I noticed truncation in users output but assumed it might've been copy paste. Verified on my webserver that it is requesting the truncated version -- BEGIN SNIPPET -- 192.168.1.187 - - [16/Nov/2019:12:05:10 +0000] "GET /HLS/mall.mp4.ke HTTP/1.0" 404 168 "-" "VLC/3.0.8 LibVLC/3.0.8" "-" "scratch.holly.home" CACHE_- 0.002 - -- END SNIPPET -- Dumping the file out as hex it _looks_ ok -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ xxd small.mp4.m3u8 00000000: 2345 5854 4d33 550a 2345 5854 2d58 2d56 #EXTM3U.#EXT-X-V 00000010: 4552 5349 4f4e 3a33 0a23 4558 542d 582d ERSION:3.#EXT-X- 00000020: 4d45 4449 412d 5345 5155 454e 4345 3a30 MEDIA-SEQUENCE:0 00000030: 0a23 4558 542d 582d 414c 4c4f 572d 4341 .#EXT-X-ALLOW-CA 00000040: 4348 453a 5945 530a 2345 5854 2d58 2d4b CHE:YES.#EXT-X-K 00000050: 4559 3a4d 4554 484f 443d 4145 532d 3132 EY:METHOD=AES-12 00000060: 382c 5552 493d 736d 616c 6c2e 6d70 342e 8,URI=small.mp4. 00000070: 6b65 790a 2345 5854 2d58 2d54 4152 4745 key.#EXT-X-TARGE 00000080: 5444 5552 4154 494f 4e3a 360a 2345 5854 TDURATION:6.#EXT 00000090: 494e 463a 352e 3630 3030 3030 2c0a 736d INF:5.600000,.sm 000000a0: 616c 6c2e 6d70 345f 3030 3030 302e 7473 all.mp4_00000.ts 000000b0: 0a23 4558 542d 582d 454e 444c 4953 540a .#EXT-X-ENDLIST. -- END SNIPPET -- But, if we take the relevant line and try and convert it back the issue becomes apparent -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ echo "382c 5552 493d 736d 616c 6c2e 6d70 342e 6b65 790a" | xxd -r URI=small.mp4.ke -- END SNIPPET -- The end of that looks ok, hex `79` is dec 121 ("y" in ascii) and `0a` is a line break. Let's iterate over the hex and convert to dec then -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ echo "382c 5552 493d 736d 616c 6c2e 6d70 342e 6b65 790a" | tr -d ' ' | python3 -c 'import sys,re; p=re.findall("..",sys.stdin.read()); [print(int(x,16))for x in p]' 56 44 85 82 73 61 115 109 97 108 108 46 109 112 52 46 107 101 121 10 -- END SNIPPET -- Dumping those to chars -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ echo "382c 5552 493d 736d 616c 6c2e 6d70 342e 6b65 790a" | tr -d ' ' | python3 -c 'import sys,re; p=re.findall("..",sys.stdin.read()); [print(chr(int(x,16)))for x in p]' 8 , U R I = s m a l l . m p 4 . k e y -- END SNIPPET -- Looks ok... Converting one way and back works fine -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ cat small.mp4.m3u8 | xxd > hex ben@milleniumfalcon:~/tmp/HLS$ cat hex | xxd -r #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-ALLOW-CACHE:YES #EXT-X-KEY:METHOD=AES-128,URI=small.mp4.key #EXT-X-TARGETDURATION:6 #EXTINF:5.600000, small.mp4_00000.ts #EXT-X-ENDLIST -- END SNIPPET -- Perhaps the earlier test was a false positive then? This is ridiculous. I hate macs. Going back to basics, what if we quote the keyfile name -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp/HLS$ cat ~/scratch/HLS/small.mp4.m3u8 #EXTM3U #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-ALLOW-CACHE:YES #EXT-X-KEY:METHOD=AES-128,URI="small.mp4.key" #EXT-X-TARGETDURATION:6 #EXTINF:5.600000, small.mp4_00000.ts #EXT-X-ENDLIST -- END SNIPPET -- VLC works. And so does Safari.... FFS ----------------------------------------------------------------------------------------- 2019-11-16 12:27:22 btasker ----------------------------------------------------------------------------------------- Running a mux with the current codebase -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp$ ~/Documents/src.old/HLS-Stream-Creator/HLS-Stream-Creator.sh -i ../Videos/lua_string_split.mp4 -e -s 10 -o before -- END SNIPPET -- Making the change -- BEGIN SNIPPET -- ben@milleniumfalcon:~/Documents/src.old/HLS-Stream-Creator$ git diff diff --git a/HLS-Stream-Creator.sh b/HLS-Stream-Creator.sh index d88fa6b..1d6a6c3 100755 --- a/HLS-Stream-Creator.sh +++ b/HLS-Stream-Creator.sh @@ -322,7 +322,7 @@ function encrypt(){ for manifest in ${OUTPUT_DIRECTORY}/*.m3u8 do # Insert the KEY at the 5'th line in the m3u8 file - $SED -i "5i #EXT-X-KEY:METHOD=AES-128,URI="${KEY_PREFIX}${KEY_NAME}.key "$manifest" + $SED -i "5i #EXT-X-KEY:METHOD=AES-128,URI=\"${KEY_PREFIX}${KEY_NAME}.key\"" "$manifest" done } -- END SNIPPET -- Running a new mux -- BEGIN SNIPPET -- ben@milleniumfalcon:~/tmp$ ~/Documents/src.old/HLS-Stream-Creator/HLS-Stream-Creator.sh -i ../Videos/lua_string_split.mp4 -e -s 10 -o after -- END SNIPPET -- Tested playback on a Mac - Playback of http://scratch.holly.home/before/lua_string_split.mp4.m3u8 fails in the same manner. - Playback of http://scratch.holly.home/after/lua_string_split.mp4.m3u8 fails in the same manner. Committing as 6fb42e9 ----------------------------------------------------------------------------------------- 2019-11-16 12:28:58 git ----------------------------------------------------------------------------------------- -- BEGIN QUOTE -- Repo: HLS-Stream-Creator Commit: 6fb42e9f937718ca8818016670f86664eed2ab74 Author: B Tasker