HLS-8: Implement ability to tell FFMPEG the input is a linear stream

Issue Information

Issue Type: New Feature
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: HLS Stream Creator (HLS)
Resolution: Done (2015-06-15 15:46:31)
Target version: 1.0,
Components: Linear Support ,

Created: 2015-06-11 12:11:17
Time Spent Working

Although it was a little ropey, older revisions could take a linear input (such as a named pipe) and use that in order to continuously output a live stream.

HLS-5 changed the way that the m3u8 is generated and so we now need to be more explicit about whether or not the stream is live (as it will affect the m3u8 options).

Need to add a simple command line flag to specify that it's linear video and act accordingly

Issue Links

Toggle State Changes


Marking as blocked by HLS-7 as I'd rather add as an option

Repo: HLS-Stream-Creator
Commit: 47311351daab4279a9b766f1d40b433757189645
Author: Ben Tasker <github@<Domain Hidden>>

Date: Thu Jun 11 13:53:17 2015 +0100
Commit Message: Added basic support for linear video. See HLS-8

Modified (-)(+)

Webhook User-Agent


View Commit

Basic support has been implemented via the addition of the following command line options
-l                        Input is a live stream
-c [count]           Number of segments to include in the playlist

The -c option is optional and defaults to 0 (infinite). Tested with various values, and ffmpeg correctly increments the EXT-X-MEDIA-SEQUENCE when sliding the playlist window.

Currently, you cannot tell the script to use an RTMP or RTSP stream (directly) as an input as the script checks for whether or not the input file exists. It's possible to do it by using a named pipe, but that's hacky and unpleasant.

So need to adjust the inputfile check so that it's more forgiving if the input type is a stream/remote resource.

I guess the best way to do that is probably to test whether the inputfilename is a path starting with some kind of scheme (i.e. rtmp://foo/bar or http://example.invalid/bar_)

Given that ffmpeg supports a fair few protocols - https://www.ffmpeg.org/ffmpeg-protocols.html - it's probably best to do a simple regex to see if :/ exists in the input filename.

Might need to make a special case for things like cache:URL (tell ffmpeg to cache the URL to a local file and then use it) as it excludes the slash. Similarly file: should be treated as if the user had just specified a path.

Alternatively, as ffmpeg will complain if the file doesn't exist anyway, it might be as simple to just remove the check for the input file (though we may then end up creating an output directory and erroring out just after).
btasker added 'Linear' to labels
btasker removed 'Linear' from labels
I think in the short-term, to allow testing on things like HLS-6 it's probably best to rely on ffmpeg checking the input stream is valid.
Once the file existance check is removed, will need to test against some live streams. For the time being, should test against the following and consider them explicitly supported

- RTMP stream
- Reading from named pipe
- Single bitrate HLS (outputting to adaptive)

For the HLS input, can test a few methods - reading from local file system, from remote URL and via a pipe.

Repo: HLS-Stream-Creator
Commit: b16020d65493c3c486a87ce73ddbdabff2cb8cf1
Author: Ben Tasker <github@<Domain Hidden>>

Date: Sun Jun 14 14:36:00 2015 +0100
Commit Message: Removed input file check for HLS-8

Modified (-)(+)

Webhook User-Agent


View Commit

As the input file check has been removed, have been able to run a number of tests.
The file being used is taken from JWPlayer's demo page - technically it's VoD but is at least long enough to run basic tests, will look at setting up a truly live stream later.

Simple RTMP
$ ./HLS-Stream-Creator.sh -i rtmp://fms.12E5.edgecastcdn.net/0012E5/mp4:videos/8Juv1MVa-485.mp4 -s 10 -o /tmp/rtmptest/ -l

Watching as it encodes

- Audio is in sync
- video plays without stutter

Single to Multiple Bitrates
$ ./HLS-Stream-Creator.sh -i rtmp://fms.12E5.edgecastcdn.net/0012E5/mp4:videos/8Juv1MVa-485.mp4 -s 10 -o /tmp/rtmptest/ -l -b 36,128,512
vlc appears to get stuck just after the video intro when using the master playlist - lower bitrates play fine, where the 512k stream falls off at the same point, so think it's actually just that the highest bitrate isn't currently keeping up. vlc is obviously parsing the manifest OK as it's playing back video, so think it's just falling off the end of the manifest (more a limitation of the hardware I'm testing on) - it's not switching down a bitrate because it's not experiencing delivery issues, just an EOF

Will drop the higher bitrate from the next set of tests to simplify testing, but worth coming back to

Sliding window, 50 seconds of video per manifest
./HLS-Stream-Creator.sh -i rtmp://fms.12E5.edgecastcdn.net/0012E5/mp4:videos/8Juv1MVa-485.mp4 -s 10 -o /tmp/rtmptest/ -l -b 36,128 -c 5

The two bitrates remain relatively in sync in terms of segment count, though the lower may be a few seconds ahead. The video looks terrible at 36K mind :)

So, all seems to work within the constraints of my test hardware.

Because the variant playlist is generated before the encode is started on the bitrates, a player arriving early may try to load the video before it's ready (for it's part ffmpeg doesn't generate the manifest until one segment has been encoded), so it may be worth looking at moving the variant playlist generation.

Using a non-local resource obviously works, though the streams may currently go out of sync. That's not an issue for VoD (as the encoding will likely be done before the media is available for delivery) but is a potential headache for live streams - you don't want there to be too big a difference when using a sliding Window.

If the 36k stream in test 3 where to get 50 seconds ahead (unlikely, but possible) a player switching down a bitrate might find it's current position simply isn't available in the manifest.
One of the requirements in HLSDVD-2 (which is intended to use this script) is that it should be possible to read from a named pipe (a FIFO). It'll also need short segment sizes so that there's not too great a delay between the source changing and the output changing (2 seconds seems an OK starting point).

Although technically it can be handled as a linear stream, there are a few complications (if you're wanting to publish an adaptive stream for example) so will raise a separate issue for that - HLS-13.

For a single bitrate output (i.e. a single ffmpeg output) the principle works OK though
$ mkfifo test.fifo.avi
$ mplayer -dumpstream -dumpfile test.fifo.avi test.avi

$ ./HLS-Stream-Creator.sh -i /tmp/test.fifo.avi -s 2 -o /tmp/fifotest/ -l -b 512 -c 20

The caveat being that ffmpeg needs to know the input stream type, so the FIFO needs to carry a suitable filename extension.

To get multiple output bitrates supported, we'll need some kind of wrapper which can read from the FIFO and make available for multiple processes to read.
The output stream from the FIFO also plays OK on in MX Player on an Android tablet.
Closing as implemented - the stream plays back OK on multiple devices and handling FIFO and ABR streams is better handled in the dedicated issue: HLS-13
btasker changed status from 'Open' to 'Resolved'
btasker added 'Done' to resolution
btasker changed status from 'Resolved' to 'Closed'