HLS-33: Brew no longer supports --with-default-names

Issue Information

Issue Type: Bug
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: HLS Stream Creator (HLS)
Resolution: Fixed (2019-03-29 13:09:12)
Affects Version: 1.0,
Target version: 1.0,

Created: 2019-03-26 09:08:14
Time Spent Working


Description
It looks like brew has removed support for the --with-default-names argument. So when trying to install GNU sed Mac users will get 

Error: invalid option: --with-default-names


There are a bunch of commits from January like this one - https://github.com/Homebrew/homebrew-core/commit/43f2e19d93f04eddf24ec19187b3648f594b80e2

Can see that they've removed support for the option, and now print a caveat on install instructing the user to edit bashrc if they want the programs to run from their original names (i.e. sed instead of gsed).

I'm not sure that's particularly an improvement in user experience, but it is what it is.

Need to update the README to detail the new process. Did toy with the idea of including a test in the script to see if gsed exists (and use that if it does), but I don't really like the idea of having hacks in there for the sake of one (albeit popular) OS.

In Github #30 (where this was identified), user has provided the output from running `brew install grep` 
brew install grep
==> Downloading https://homebrew.bintray.com/bottles/grep-3.3.mojave.bottle.2.tar.gz
######################################################################## 100.0%
==> Pouring grep-3.3.mojave.bottle.2.tar.gz
==> Caveats
All commands have been installed with the prefix "g".
If you need to use these commands with their normal names, you
can add a "gnubin" directory to your PATH from your bashrc like:
PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"


That path can differ depending on flags passed in, but that should be the default.

Also need to find out whether this post (https://scriptingosx.com/2017/04/about-bash_profile-and-bashrc-on-macos/) about Macs being a bit stupid when it comes to bashrc and bash_profile is still accurate.


Issue Links

Github #30
Toggle State Changes

Activity


2019-03-26 17:21:21

Repo: HLS-Stream-Creator
Commit: 5807524b54017b429e77a357c757576d283baff5
Author: B Tasker <github@<Domain Hidden>>

Date: Tue Mar 26 17:18:23 2019 +0000
Commit Message: HLS-33 Test for presence of ggrep and gsed and use those if they exist

This is due to a change in `brew` that means you can no longer use `--with-default-names`, the replacement method (adjusting PATH in bashrc) seems to have inconsistent results.



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent 

GitHub-Hookshot/903858c


View Commit


2019-03-26 17:25:21

Repo: HLS-Stream-Creator
Commit: 4edc45061c8ffda73147d4a01a85be19492357df
Author: B Tasker <github@<Domain Hidden>>

Date: Tue Mar 26 17:24:25 2019 +0000
Commit Message: HLS-33

Not convinced I could have screwed that last commit up much more.

* Reverse the logic (override the var if the command completes successfully)
* Insert missing semicolons

In my defence, the commit was done to deploy onto my test system



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent 

GitHub-Hookshot/903858c


View Commit


2019-03-26 17:29:45
Feedback in the Github ticket, along with various resources online, suggest that relying on users to adjust bashrc is going to give fairly inconsistent results (depending on which terminal program they use), so decided to bite the bullet and write a check into the script instead.

There's a test encode running at the moment (seemingly successfully, but time will tell).

I'm going to update the README to remove the brew steps (so that they don't cause confusion). Recording the original content below for posterities sake

Segment encryption won't work out of the box on OS X as it relies on arguments which the BSD `grep` and `sed` commands don't support. In order to use encryption on OS X you must first install their GNU counterparts 

    brew install gnu-sed --with-default-names
    brew install grep --with-default-names



2019-03-26 17:33:21

Repo: HLS-Stream-Creator
Commit: 880b71da2f62b97700f5a5117210f0f8080d5c7c
Author: B Tasker <github@<Domain Hidden>>

Date: Tue Mar 26 17:32:03 2019 +0000
Commit Message: HLS-33 Update README



Modified (-)(+)
-------
README.md




Webhook User-Agent 

GitHub-Hookshot/903858c


View Commit


2019-03-26 17:39:46
Test encode completed successfully, no errors thrown. Segments are encrypted on disk, and the key line has been successfully written into the manifest.

Have asked the user who originally reported issues to try pulling the latest revision to run on their Mac. Other than creating aliases I don't have a good way to test how a Mac will behave

2019-03-28 16:09:00
Frustratingly it seems not to be working. The failure being reported appears when trying to insert the EXT-X-KEY directive into the manifest (exactly where we expect BSD-sed to fail), but the new test should be causing us to use gsed 
for manifest in ${OUTPUT_DIRECTORY}/*.m3u8
do
    # Insert the KEY at the 5'th line in the m3u8 file
    $SED -i "5i #EXT-X-KEY:METHOD=AES-128,URI="${KEY_PREFIX}${KEY_NAME}.key "$manifest"
done


The user has provided output which shows the test should work (in the sense that command functions as expected on a Mac) 
$ command -v gsed
/usr/local/bin/gsed

$ command -v sed
/usr/bin/sed


I wondered if perhaps it doesn't like the ternary operator used on that line in the script, so have asked for the output of the following 
SED='sed'
command -v gsed >/dev/null 2>&1
if [ "$?" == "0" ]
then
SED='gsed'
fi
echo $SED

command -v foobar >/dev/null 2>&1
echo $?

command -v command >/dev/null 2>&1
echo $?

If exit statuses are set correctly then it'd be worth adjusting to use those. If not, might need to revisit the original decision to include logic in the script to handle it. Sodding Macs.

2019-03-28 17:52:21

Repo: HLS-Stream-Creator
Commit: 765e5c26a8f423b5e6e5c3740cebe884ad1fcdda
Author: B Tasker <github@<Domain Hidden>>

Date: Thu Mar 28 17:49:18 2019 +0000
Commit Message: HLS-33 Macs appear not to like the ternary operator (see #30)

Swapping to a less concise syntax. Either I've missed something more fundamental, or this should work



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent 

GitHub-Hookshot/903858c


View Commit


2019-03-29 09:02:02
User has provided a video of them running HLS-Stream-Creator. Was originally uploaded to youtube, so I've put a copy here ( https://videos.bentasker.co.uk/Player.html?url=2018/03/201803_hls_33_sed_error/hls_33_rec_master.m3u8 ) so that can refer back if needed.

2019-03-29 09:54:36
Watching the video, this is actually a change in behaviour.

The key line isn't being written into the manifest, but there's also no error being thrown to stderr. With BSD sed we'd normally see something like 
sed: 1: "./output/dil.mp4.m3u8": invalid command code .


and that's not present here. Suggests gsed is being called but something else is going wrong. Need to try and lay hands on a Mac later to tinker with this some more

2019-03-29 11:12:21

Repo: HLS-Stream-Creator
Commit: 2114885fc19d3dd80db9cb89c40fbe1b0f4c756f
Author: B Tasker <github@<Domain Hidden>>

Date: Fri Mar 29 11:09:59 2019 +0000
Commit Message: Fix variable name for HLS-33



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent 

GitHub-Hookshot/903858c


View Commit


2019-03-29 11:13:12
Have laid hands on a Mac running Mojave and have now set it up with homebrew and ffmpeg (bit of a rigmarole, why do people like these things so much?).

First run yielded some errors not visible in the video - I accidentally set the wrong variable for the ggrep detection, so we were setting SED='ggrep'. Which would be why the key wasn't getting inserted.

Test run following the commit to fix that resulted in the key being inserted correctly.

2019-03-29 11:39:32
The variable name was also incorrect in the original commit implementing the check - https://github.com/bentasker/HLS-Stream-Creator/commit/5807524b54017b429e77a357c757576d283baff5

2019-03-29 11:49:39
I've updated the title of this issue to no longer refer to updating the README, as work has been done within the script too. Readme is up to date.

2019-03-29 11:56:21

Repo: HLS-Stream-Creator
Commit: 74ed3d5e5a1ce8d43c9e7d68c436131f3509b78d
Author: B Tasker <github@<Domain Hidden>>

Date: Fri Mar 29 11:53:47 2019 +0000
Commit Message: Add setup steps on Mac OS X, based on having run through them in HLS-33



Modified (-)(+)
-------
README.md




Webhook User-Agent 

GitHub-Hookshot/903858c


View Commit


2019-03-29 13:09:13
btasker changed status from 'Open' to 'Resolved'

2019-03-29 13:09:13
btasker added 'Fixed' to resolution

2019-03-29 13:09:17
btasker changed status from 'Resolved' to 'Closed'