HLS-20: Reported issues with Encryption



Issue Information

Issue Type: Bug
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: HLS Stream Creator (HLS)
Resolution: Fixed (2017-04-26 22:20:00)
Components: CLI Options , Encryption ,
Labels: Encryption,

Created: 2017-04-26 19:56:23
Time Spent Working
Estimated:
  
60 minutes
Remaining:
 
0 minutes
Logged:
 
124 minutes


Description
A couple of users have reported (in Github #15) that when they enable encryption, nothing happens

An example call provided is
./HLS-Stream-Creator.sh -i trailer.mov -s 10 -e -o trailer -b 272,872,1372,1872


When the issue was originally raised I ran a test and it seemed to work, but the original user never provided the further info I asked for.

Another user has reported the same issue (in the same thread) and has provided the commandline above, so need to revisit this


Issue Links

Github #15
Original Merge bb3cce4f
Toggle State Changes

Activity


Turns out I still need to update the readme to note the existence of this flag, but the functionality was originally merged in bb3cce4f
OK, running a test against the latest master
# Looks like I've got an old version of ffmpeg, so will allow experimental codecs for now (and update later)

ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ FFMPEG_FLAGS='-strict -2'
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ export FFMPEG_FLAGS

# Verifying that it works without encryption
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ./HLS-Stream-Creator.sh -i /tmp/hlstest/big_buck_bunny_720p_stereo.avi -s 10 -o test1 -b 272,872,1372,1872


All looks fine so far.

Now with encryption enabled
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ./HLS-Stream-Creator.sh -i /tmp/hlstest/big_buck_bunny_720p_stereo.avi -s 10 -e -o test2 -b 272,872,1372,1872


That's running, but encryption doesn't happen until the end, so will need to wait for the transcodes to complete. Will have a look over the code while I wait
Transcodes have completed, but it doesn't look like encrypt has run at all

ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ grep X-KEY test2/*m3u8
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ls test2/*.key
ls: cannot access 'test2/*.key': No such file or directory


I don't see anything odd on either stdout or stderr

I wonder if it might be a weird product of creating an Adaptive stream. Will re-run with a single bitrate and see if encrypt then gets called
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ./HLS-Stream-Creator.sh -i /tmp/hlstest/big_buck_bunny_720p_stereo.avi -s 10 -e -o test3 -b 272


Odd, doesn't seem to be happening for a single bitrate stream either

ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ls test3/*.key
ls: cannot access 'test3/*.key': No such file or directory


Was definitely working before?
Ohhhh, I see it.

It's a result of the conditional at Line 353 ( https://github.com/bentasker/HLS-Stream-Creator/blob/4a9e0a019f3eed8036836b37c93e213a03c0fade/HLS-Stream-Creator.sh#L353 )

If bitrates have been specified in the argument list, we fork off down a different code path (because we need to spin up sub-processes etc). Processing effectively ends once awaitCompletion has done it's bit.

The call to encrypt happens in the other (smaller) code path, that we follow when bitrates haven't been explicitly specified, essentially everything below the else at line 438 ( https://github.com/bentasker/HLS-Stream-Creator/blob/4a9e0a019f3eed8036836b37c93e213a03c0fade/HLS-Stream-Creator.sh#L438 )

So the original pull request was wrong. Damn, I should've caught that
Post "fix" (commit a8898a3) testing

Specify a single bitrate on the CL
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ./HLS-Stream-Creator.sh -i /tmp/hlstest/big_buck_bunny_720p_stereo.avi -s 10 -e -o test6 -b 272

ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ls test6/*.key
test6/big_buck_bunny_720p_stereo.avi.key


Not perfect though, the m3u8 rewrite is using an incorrect manifest name
sed: can't read test6/big_buck_bunny_720p_stereo.avi.m3u8: No such file or directory


Thinking about it, the key will need to be written into the bitrate manifests (and not the master) anyway. I'm going to put that into a seperate commit


Test with no bitrate specified
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ./HLS-Stream-Creator.sh -i /tmp/hlstest/big_buck_bunny_720p_stereo.avi -s 10 -e -o test7
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ls test7/*.key
test7/big_buck_bunny_720p_stereo.avi.key

ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ grep X-KEY test7/big_buck_bunny_720p_stereo.avi.m3u8 
#EXT-X-KEY:METHOD=AES-128,URI=big_buck_bunny_720p_stereo.avi.key


And (expectedly) no complaints about not being able to find the manifest.

Checking playback
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ vlc http://scratch.holly.home/HLS/test7/big_buck_bunny_720p_stereo.avi.m3u8
....
httplive stream error: gcry_cipher_decrypt failed:  gcrypt/Invalid length


Playback fails.

Looking in the access logs, VLC is requesting a range
192.168.1.70 - - [26/Apr/2017:20:52:57 +0100] "GET /HLS/test7/big_buck_bunny_720p_stereo.avi.key HTTP/1.1" 206 16 "-" "VLC/2.2.2 LibVLC/2.2.2"
192.168.1.70 - - [26/Apr/2017:20:52:57 +0100] "GET /HLS/test7/big_buck_bunny_720p_stereo.avi_00000.ts HTTP/1.1" 206 3008564 "-" "VLC/2.2.2 LibVLC/2.2.2"
192.168.1.70 - - [26/Apr/2017:20:52:57 +0100] "GET /HLS/test7/big_buck_bunny_720p_stereo.avi.m3u8 HTTP/1.1" 206 3691 "-" "VLC/2.2.2 LibVLC/2.2.2"


The 206 is just because VLC requests uses "Range: bytes=0-"

Getting 16 bytes back, which should be right, the HLS spec says

When a key file is listed in the index file, the key file contains a cipher key that must be used to decrypt subsequent media files listed in the index file. Currently HTTP Live Streaming supports AES-128 encryption using 16-octet keys. The format of the key file is a packed array of these 16 octets in binary format.


OK, need to look into that then. No point encrypting if you can't play back (the ultimate DRM :D)
Openssl doesn't seem to like the key either

ben@milleniumfalcon:/tmp$ openssl aes-128-cbc -d -in big_buck_bunny_720p_stereo.avi_00045.ts -out test.ts -K `cat hlstest/HLS-Stream-Creator/test7/big_buck_bunny_720p_stereo.avi.key | hexdump -e '16/1 "%02x"'` -iv `printf '%032x' 45`
bad decrypt
140079633548952:error:0606506D:digital envelope routines:EVP_DecryptFinal_ex:wrong final block length:evp_enc.c:518:


Something's done horribly wrong there :(

OK, lets manually run an encrypt and see where we get. Will use one of the cleartext segments from the tests where encryption didn't run. Using the same key as the one above


ben@milleniumfalcon:/tmp/encrypt_test$ ls
big_buck_bunny_720p_stereo.avi_272_00001.ts  in.key

ben@milleniumfalcon:/tmp/encrypt_test$ INIT_VECTOR=$(printf '%032x' 1)
ben@milleniumfalcon:/tmp/encrypt_test$ ENCRYPTION_KEY=$(cat in.key | hexdump -e '16/1 "%02x"')
ben@milleniumfalcon:/tmp/encrypt_test$ 
ben@milleniumfalcon:/tmp/encrypt_test$ openssl aes-128-cbc -e -in big_buck_bunny_720p_stereo.avi_272_00001.ts -out encrypted_op.ts -nosalt -iv $INIT_VECTOR -K $ENCRYPTION_KEY

ben@milleniumfalcon:/tmp/encrypt_test$ ls
big_buck_bunny_720p_stereo.avi_272_00001.ts  encrypted_op.ts  in.key

# Will it decrypt?
ben@milleniumfalcon:/tmp/encrypt_test$ openssl aes-128-cbc -d -in encrypted_op.ts -out decrypted_op.ts -K $ENCRYPTION_KEY -iv $INIT_VECTOR

# Seems to have done
ben@milleniumfalcon:/tmp/encrypt_test$ ls -s
total 1828
608 big_buck_bunny_720p_stereo.avi_272_00001.ts  608 decrypted_op.ts  608 encrypted_op.ts    4 in.key


Which would suggest something's going wrong in script and affecting the segments themselves, rather than specifically being an issue with the key. As a starting point, have dropped an echo into the script and set a run going to see exactly what's being run
Playing around while that run completes, it seems it's much simpler than it seems.

If I remove the key definition from the manifest, the video plays, so the fragments aren't being encrypted. A quick check with xxd confirms
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ xxd test7/big_buck_bunny_720p_stereo.avi_00000.ts | head -n 25
00000000: 4740 1110 0042 f025 0001 c100 0000 01ff  G@...B.%........
00000010: 0001 fc80 1448 1201 0646 466d 7065 6709  .....H...FFmpeg.
00000020: 5365 7276 6963 6530 31a7 79a0 03ff ffff  Service01.y.....
00000030: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000040: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000050: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000060: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000070: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000080: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000090: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000a0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000b0: ffff ffff ffff ffff ffff ffff 4740 0010  ............G@..
000000c0: 0000 b00d 0001 c100 0000 01f0 002a b104  .............*..
000000d0: b2ff ffff ffff ffff ffff ffff ffff ffff  ................
000000e0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
000000f0: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000100: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000110: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000120: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000130: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000140: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000150: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000160: ffff ffff ffff ffff ffff ffff ffff ffff  ................
00000170: ffff ffff ffff ffff 4750 0010 0002 b017  ........GP......
00000180: 0001 c100 00e1 00f0 001b e100 f000 0fe1  ................


There's obviously something wrong with the for loop in encrypt. The complaints about invalid length are, of course, because the data's not padded (because it's not encrypted in the first place)
The for statement returns empty
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ OUTPUT_DIRECTORY='test9'
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ export OUTPUT_DIRECTORY
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ for file in $(ls ${OUTPUT_DIRECTORY}/*.ts | cut -f3 -d '/'); do echo "$file"; done


Let's take the subcommand out of it (I don't particularly like the variable being called file either, but will fix that once I've sorted the iteration out)
The iterator is now finding the files, but that loops still not happy
test10/test10/big_buck_bunny_720p_stereo.avi_00059.ts: No such file or directory
139653670704792:error:02001002:system library:fopen:No such file or directory:bss_file.c:398:fopen('test10/test10/big_buck_bunny_720p_stereo.avi_00059.ts','r')
139653670704792:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:400:
mv: cannot stat 'test10/big_buck_bunny_720p_stereo.avi_enc_59.ts': No such file or directory


Though the commands being run now look more or less right (aside from the input file path being incorrect)
Processing test10/big_buck_bunny_720p_stereo.avi_00058.ts
openssl aes-128-cbc -e -in test10/test10/big_buck_bunny_720p_stereo.avi_00058.ts -out test10/big_buck_bunny_720p_stereo.avi_enc_58.ts -nosalt -iv 0000000000000000000000000000003a -K 8ed38783231164928aceb5f42f970a45
Processing test10/big_buck_bunny_720p_stereo.avi_00059.ts
openssl aes-128-cbc -e -in test10/test10/big_buck_bunny_720p_stereo.avi_00059.ts -out test10/big_buck_bunny_720p_stereo.avi_enc_59.ts -nosalt -iv 0000000000000000000000000000003b -K 8ed38783231164928aceb5f42f970a45


Next test run underway. I really wish I'd checked that pull more carefully, I'm not convinced this should ever have worked fully, unless you specified an absolute path for the output directory
OK, this is looking far more hopeful
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ xxd test11/big_buck_bunny_720p_stereo.avi_00000.ts | head -n 25
00000000: 6969 bede 9aac 3913 564c 457b 42b9 6559  ii....9.VLE{B.eY
00000010: b4eb a4db 5089 a843 b0bb 09c4 7f94 6186  ....P..C......a.
00000020: 0056 9c11 6e3c 10c5 8408 bc45 22c0 c0cb  .V..n<.....E"...
00000030: 01b4 b548 4e78 2c9e b36f 8452 557b cf81  ...HNx,..o.RU{..
00000040: 2d38 06e5 8809 f27e cc89 32e5 b8bd 187c  -8.....~..2....|
00000050: 7d59 49ee 2579 cb9d 0fde 9ed9 8d9b 41d5  }YI.%y........A.
00000060: 23fa fb2e 4baa c783 a8ef e8ee 5d9d 288a  #...K.......].(.
00000070: e1c0 d5e0 932a c9e2 0006 ce92 7624 cf37  .....*......v$.7
00000080: b609 a817 a0e5 4239 328c dfa7 22ff 6ec4  ......B92...".n.
00000090: 6852 cd1f b784 8f20 3928 a3db 765a f912  hR..... 9(..vZ..
000000a0: b49f 2d04 9b3c e9bb 546a 46f4 c1df 9367  ..-..<..TjF....g
000000b0: 3a3b abc1 c33b b43b c1b1 395f f067 76a6  :;...;.;..9_.gv.
000000c0: 72e6 4f71 d94e 10ae 39a8 01db 6a18 c133  r.Oq.N..9...j..3
000000d0: e7dc 2148 5b17 b823 2b80 92db 6692 c0b0  ..!H..#+...f...
000000e0: 2f5d e226 3c51 1fb9 9afb 94ce 3fc2 399f  /].&<Q......?.9.
000000f0: 385a 8bb1 261f 61c2 12bd 827f f2f8 ff11  8Z..&.a.........
00000100: 9510 0fc8 a33b 88c7 7d3e a15d b80b 288e  .....;..}>.]..(.
00000110: d6fa 6f16 d7e4 cbe3 4b06 1c0e 3435 741e  ..o.....K...45t.
00000120: 7084 3000 93a5 905f 3bce 0462 98cc 5f03  p.0...._;..b.._.
00000130: c172 d286 52b3 412c f9a8 7e46 b278 85e2  .r..R.A,..~F.x..
00000140: a22f 7f53 2e14 df63 7dbe b279 61b2 7cb7  ./.S...c}..ya..Otv
00000170: b7b7 aa8b dd88 6100 841f 3e1a a4ed 334d  ......a...>...3M
00000180: 915a 5c50 06d8 2b9a 9619 4313 cdad fcb4  .Z\P..+...C.....


VLC no longer complains about the key, and ffplay can correctly play the stream back
ben@milleniumfalcon:~/Documents/Notes/Home_Network$ ffplay http://scratch.holly.home/HLS/test11/big_buck_bunny_720p_stereo.avi.m3u8


VLC also plays back.

Will commit the fixes made so far, and fix the manifest location issue in encrypt so that this should all work with adaptive streams (as well as single bitrate streams where only one bitrate has been specified)

Repo: HLS-Stream-Creator
Commit: a8898a33ad164abaccf9b922d050d7f0542e3879
Author: B Tasker <github@<Domain Hidden>>

Date: Wed Apr 26 20:40:42 2017 +0100
Commit Message: Partial fix for #15 and HLS-20

This commit adjusts the code path when bitrates are specified on the command line. to ensure that encrypt is called (wasn't before)

A second fix is needed to the manifest rewrite as it is currently looking for the wrong file (unless bitrates haven't been specified)



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent

GitHub-Hookshot/2cc07f9


View Commit


Repo: HLS-Stream-Creator
Commit: f9b6b7e646e1e8a81d9565c26ee06bb89c0c381c
Author: B Tasker <github@<Domain Hidden>>

Date: Wed Apr 26 21:43:32 2017 +0100
Commit Message: Fix issue with iterating over and encrypting segments when relative output dir is used. See HLS-20

Should still be considered a partial fix - need to correct a manifest filename



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent

GitHub-Hookshot/2cc07f9


View Commit

Triggering a run
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ ./HLS-Stream-Creator.sh -i /tmp/hlstest/big_buck_bunny_720p_stereo.avi -s 10 -e -o test12 -b 272,480,1024 2> /tmp/uhoh


No complaints this time
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ tail /tmp/uhoh 
libx264 @ 0x22072a0] coded y,uvDC,uvAC intra: 46.5% 55.1% 17.3% inter: 6.6% 8.7% 0.6%
[libx264 @ 0x22072a0] i16 v,h,dc,p: 31% 28% 12% 30%
[libx264 @ 0x22072a0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 26% 20% 32%  3%  4%  5%  4%  4%  4%
[libx264 @ 0x22072a0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 23% 18% 15%  5% 10% 10%  8%  6%  5%
[libx264 @ 0x22072a0] i8c dc,h,v,p: 51% 23% 20%  6%
[libx264 @ 0x22072a0] Weighted P-Frames: Y:2.0% UV:1.1%
[libx264 @ 0x22072a0] ref P L0: 69.4% 15.3% 11.8%  3.4%  0.0%
[libx264 @ 0x22072a0] ref B L0: 87.3% 10.8%  2.0%
[libx264 @ 0x22072a0] ref B L1: 96.7%  3.3%
[libx264 @ 0x22072a0] kb/s:1014.63


Playback works, and the segments are definitely encrypted
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ vlc http://scratch.holly.home/HLS/test12/big_buck_bunny_720p_stereo.avi_master.m3u8
ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ xxd test12/big_buck_bunny_720p_stereo.avi_272_00000.ts  coded y,uvDC,uvAC intra: 27.5% 43.5% 10.9% inter: 2.5% 3.3% 0.1%
[libx264 @ 0x8562a0] i16 v,h,dc,p: 32% 28% 12% 28%
[libx264 @ 0x8562a0] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 22% 15% 41%  3%  4%  4%  3%  4%  4%
[libx264 @ 0x8562a0] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 24% 18% 18%  5%  9%  8%  7%  6%  5%
[libx264 @ 0x8562a0] i8c dc,h,v,p: 62% 19% 15%  4%
[libx264 @ 0x8562a0] Weighted P-Frames: Y:2.0% UV:1.1%
[libx264 @ 0x8562a0] ref P L0: 66.1% 17.5% 12.4%  3.9%  0.0%
[libx264 @ 0x8562a0] ref B L0: 87.8% 10.2%  2.1%
[libx264 @ 0x8562a0] ref B L1: 97.1%  2.9%
[libx264 @ 0x8562a0] kb/s:472.45

ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ xxd test13/big_buck_bunny_720p_stereo.avi_480_00000.ts | head -n 3
00000000: 917d 5673 98dc 708f 094d c8e9 1c91 38e3  .}Vs..p..M....8.
00000010: 64d9 2d2b 1e68 5e9b 0060 9a54 af69 a3f0  d.-+.h^..`.T.i..
00000020: 7d8c 61e7 8415 6433 dc48 2f60 48f0 4a4a  }.a...d3.H/`H.JJ

ben@milleniumfalcon:/tmp/hlstest/HLS-Stream-Creator$ vlc http://scratch.holly.home/HLS/test13/big_buck_bunny_720p_stereo.avi_master.m3u8


I'm happy with that, will commit and push the changes.

Need to update the README to reference the encryption flag before closing this issue though

Repo: HLS-Stream-Creator
Commit: b65ec694c23c613c8a8e2589d7f2c9053d8b8d75
Author: B Tasker <github@<Domain Hidden>>

Date: Wed Apr 26 22:02:53 2017 +0100
Commit Message: Adjust manifest filename in encryption. See HLS-20 and #15

The encrypt function assumed a single bitrate stream (not generated using a -b argument), have changed the way manifest files are located when we write key information into them.

Could be improved - we'll currently write into the master playlist too, which is harmless but unnecessary. Just didn't seem worth putting a check in to stop it



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent

GitHub-Hookshot/2cc07f9


View Commit

btasker changed timespent from '0 minutes' to '124 minutes'

Repo: HLS-Stream-Creator
Commit: 42de7b6cd9a344d6e57d2544f1b18376aaef7bc1
Author: B Tasker <github@<Domain Hidden>>

Date: Wed Apr 26 22:07:19 2017 +0100
Commit Message: Updated README to reference encryption. See HLS-20



Modified (-)(+)
-------
README.md




Webhook User-Agent

GitHub-Hookshot/2cc07f9


View Commit

Could probably do with adding a section about encryption to the README but that can wait until another day.

Marking as fixed as all my tests have passed. Will re-open if the users report they're still have issues
btasker changed status from 'Open' to 'Resolved'
btasker added 'Fixed' to resolution
btasker changed status from 'Resolved' to 'Closed'
btasker removed 'Fixed' from resolution
btasker changed status from 'Closed' to 'Reopened'
btasker added 'Encryption' to labels
btasker changed status from 'Reopened' to 'Resolved'
btasker added 'Fixed' to resolution
btasker changed status from 'Resolved' to 'Closed'

Repo: HLS-Stream-Creator
Commit: 88e9f80a0cae78a705222274ffeae8f11cc6cd07
Author: B Tasker <github@<Domain Hidden>>

Date: Sat Apr 29 11:44:32 2017 +0100
Commit Message: Update README to contain more information on encrypted streams (HLS-20)



Modified (-)(+)
-------
README.md




Webhook User-Agent

GitHub-Hookshot/2cc07f9


View Commit

Work log


Ben Tasker
Permalink
2017-04-26 22:08:17

Time Spent: 124 minutes
Log Entry: Investigating, going further down the rabbit hole, testing fixing