##########################################################################################

       HLS-8: Implement ability to tell FFMPEG the input is a linear stream

##########################################################################################


Issue Type: New Feature 
-----------------------------------------------------------------------------------------

Issue Information
====================

Priority: Major					Status:      Closed
Resolution:  Done (2015-06-15 15:46:31)
Project: HLS Stream Creator (HLS)


Reported By: btasker					
Assigned To: btasker

Components: 
	- Linear Support 


Targeted for fix in version: 
	- 1.0


Time Estimate: 0 minutes
Time Logged:   0 minutes


-----------------------------------------------------------------------------------------

Issue Description
==================

Although it was a little ropey, older revisions could take a linear input (such as a named
pipe) and use that in order to continuously output a live stream.

HLS-5 changed the way that the m3u8 is generated and so we now need to be more explicit
about whether or not the stream is live (as it will affect the m3u8 options).

Need to add a simple command line flag to specify that it's linear video and act
accordingly

-----------------------------------------------------------------------------------------

Issue Relations
================
	
	- is blocked by HLS-7: Implement use of getopt
	- relates to HLS-5: PR#6 FFMPEG will be called only once


-----------------------------------------------------------------------------------------

Activity
==========

	
-----------------------------------------------------------------------------------------
2015-06-11 12:11:46              btasker
-----------------------------------------------------------------------------------------

Marking as blocked by HLS-7 as I'd rather add as an option
	
-----------------------------------------------------------------------------------------
2015-06-11 13:57:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: HLS-Stream-Creator
Commit: 47311351daab4279a9b766f1d40b433757189645
Author: Ben Tasker <github@<Domain Hidden>>

Date: Thu Jun 11 13:53:17 2015 +0100
Commit Message: Added basic support for linear video. See HLS-8



Modified (-)(+)
-------
HLS-Stream-Creator.sh
README.md



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/99ae70d
 -- END SNIPPET --

https://github.com/bentasker/HLS-Stream-Creator/commit/47311351daab4279a9b766f1d40b433757189645


	
-----------------------------------------------------------------------------------------
2015-06-11 14:05:19              btasker
-----------------------------------------------------------------------------------------

Basic support has been implemented via the addition of the following command line options

-- BEGIN SNIPPET --

-l                        Input is a live stream
-c [count]           Number of segments to include in the playlist

 -- END SNIPPET --
The _-c_ option is optional and defaults to 0 (infinite). Tested with various values, and
_ffmpeg_ correctly increments the _EXT-X-MEDIA-SEQUENCE_ when sliding the playlist
window.

Currently, you cannot tell the script to use an RTMP or RTSP stream (directly) as an input
as the script checks for whether or not the input file exists. It's possible to do it by
using a named pipe, but that's hacky and unpleasant.

So need to adjust the inputfile check so that it's more forgiving if the input type is a
stream/remote resource.

I guess the best way to do that is probably to test whether the inputfilename is a path
starting with some kind of scheme (i.e. _rtmp://foo/bar_ or _http://example.invalid/bar_)

Given that ffmpeg supports a fair few protocols -
https://www.ffmpeg.org/ffmpeg-protocols.html - it's probably best to do a simple regex to
see if _:/_ exists in the input filename.

Might need to make a special case for things like _cache:URL_ (tell ffmpeg to cache the
URL to a local file and then use it) as it excludes the slash. Similarly _file:_ should be
treated as if the user had just specified a path.

Alternatively, as ffmpeg will complain if the file doesn't exist anyway, it might be as
simple to just remove the check for the input file (though we may then end up creating an
output directory and erroring out just after).
	
-----------------------------------------------------------------------------------------
2015-06-12 16:24:16              
-----------------------------------------------------------------------------------------

btasker added 'Linear' to labels
	
-----------------------------------------------------------------------------------------
2015-06-12 16:28:52              
-----------------------------------------------------------------------------------------

btasker removed 'Linear' from labels
	
-----------------------------------------------------------------------------------------
2015-06-12 16:39:12              btasker
-----------------------------------------------------------------------------------------

I think in the short-term, to allow testing on things like HLS-6 it's probably best to
rely on _ffmpeg_ checking the input stream is valid.

	
-----------------------------------------------------------------------------------------
2015-06-14 12:28:01              btasker
-----------------------------------------------------------------------------------------

Once the file existance check is removed, will need to test against some live streams. For
the time being, should test against the following and consider them explicitly supported

- RTMP stream
- Reading from named pipe
- Single bitrate HLS (outputting to adaptive)

For the HLS input, can test a few methods - reading from local file system, from remote
URL and via a pipe.
	
-----------------------------------------------------------------------------------------
2015-06-14 14:37:15              git
-----------------------------------------------------------------------------------------


-- BEGIN QUOTE --


Repo: HLS-Stream-Creator
Commit: b16020d65493c3c486a87ce73ddbdabff2cb8cf1
Author: Ben Tasker <github@<Domain Hidden>>

Date: Sun Jun 14 14:36:00 2015 +0100
Commit Message: Removed input file check for HLS-8



Modified (-)(+)
-------
HLS-Stream-Creator.sh



-- END QUOTE --

*Webhook User-Agent*


-- BEGIN SNIPPET --

GitHub-Hookshot/99ae70d
 -- END SNIPPET --

https://github.com/bentasker/HLS-Stream-Creator/commit/b16020d65493c3c486a87ce73ddbdabff2cb8cf1


	
-----------------------------------------------------------------------------------------
2015-06-15 14:51:21              btasker
-----------------------------------------------------------------------------------------

As the input file check has been removed, have been able to run a number of tests.
The file being used is taken from JWPlayer's demo page - technically it's VoD but is at
least long enough to run basic tests, will look at setting up a truly live stream later.

*Simple RTMP*

-- BEGIN SNIPPET --

$ ./HLS-Stream-Creator.sh -i
rtmp://fms.12E5.edgecastcdn.net/0012E5/mp4:videos/8Juv1MVa-485.mp4 -s 10 -o /tmp/rtmptest/
-l

 -- END SNIPPET --
Watching as it encodes

- Audio is in sync
- video plays without stutter

*Single to Multiple Bitrates*

-- BEGIN SNIPPET --

$ ./HLS-Stream-Creator.sh -i
rtmp://fms.12E5.edgecastcdn.net/0012E5/mp4:videos/8Juv1MVa-485.mp4 -s 10 -o /tmp/rtmptest/
-l -b 36,128,512

 -- END SNIPPET --
_vlc_ appears to get stuck just after the video intro when using the master playlist -
lower bitrates play fine, where the 512k stream falls off at the same point, so think it's
actually just that the highest bitrate isn't currently keeping up. _vlc_ is obviously
parsing the manifest OK as it's playing back video, so think it's just falling off the end
of the manifest (more a limitation of the hardware I'm testing on) - it's not switching
down a bitrate because it's not experiencing delivery issues, just an EOF

Will drop the higher bitrate from the next set of tests to simplify testing, but worth
coming back to

*Sliding window, 50 seconds of video per manifest*

-- BEGIN SNIPPET --

./HLS-Stream-Creator.sh -i
rtmp://fms.12E5.edgecastcdn.net/0012E5/mp4:videos/8Juv1MVa-485.mp4 -s 10 -o /tmp/rtmptest/
-l -b 36,128 -c 5

 -- END SNIPPET --
The two bitrates remain relatively in sync in terms of segment count, though the lower may
be a few seconds ahead. The video looks terrible at 36K mind :)

So, all seems to work within the constraints of my test hardware.

*General*
Because the variant playlist is generated before the encode is started on the bitrates, a
player arriving early may try to load the video before it's ready (for it's part _ffmpeg_
doesn't generate the manifest until one segment has been encoded), so it may be worth
looking at moving the variant playlist generation.

Using a non-local resource obviously works, though the streams may currently go out of
sync. That's not an issue for VoD (as the encoding will likely be done before the media is
available for delivery) but is a potential headache for live streams - you don't want
there to be too big a difference when using a sliding Window.

If the 36k stream in test 3 where to get 50 seconds ahead (unlikely, but possible) a
player switching down a bitrate might find it's current position simply isn't available in
the manifest.
	
-----------------------------------------------------------------------------------------
2015-06-15 15:32:42              btasker
-----------------------------------------------------------------------------------------

One of the requirements in HLSDVD-2 (which is intended to use this script) is that it
should be possible to read from a named pipe (a FIFO). It'll also need short segment sizes
so that there's not too great a delay between the source changing and the output changing
(2 seconds seems an OK starting point).

Although technically it can be handled as a linear stream, there are a few complications
(if you're wanting to publish an adaptive stream for example) so will raise a separate
issue for that - HLS-13.

For a single bitrate output (i.e. a single ffmpeg output) the principle works OK though

-- BEGIN SNIPPET --

$ mkfifo test.fifo.avi
$ mplayer -dumpstream -dumpfile test.fifo.avi test.avi

$ ./HLS-Stream-Creator.sh -i /tmp/test.fifo.avi -s 2 -o /tmp/fifotest/ -l -b 512 -c 20

 -- END SNIPPET --
The caveat being that _ffmpeg_ needs to know the input stream type, so the FIFO needs to
carry a suitable filename extension.

To get multiple output bitrates supported, we'll need some kind of wrapper which can read
from the FIFO and make available for multiple processes to read.
	
-----------------------------------------------------------------------------------------
2015-06-15 15:45:31              btasker
-----------------------------------------------------------------------------------------

The output stream from the FIFO also plays OK on in _MX Player_ on an Android tablet.
	
-----------------------------------------------------------------------------------------
2015-06-15 15:46:31              btasker
-----------------------------------------------------------------------------------------

Closing as implemented - the stream plays back OK on multiple devices and handling FIFO
and ABR streams is better handled in the dedicated issue: HLS-13
	
-----------------------------------------------------------------------------------------
2015-06-15 15:46:31              
-----------------------------------------------------------------------------------------

btasker changed status from 'Open' to 'Resolved'
	
-----------------------------------------------------------------------------------------
2015-06-15 15:46:31              
-----------------------------------------------------------------------------------------

btasker added 'Done' to resolution
	
-----------------------------------------------------------------------------------------
2015-06-15 15:46:35              
-----------------------------------------------------------------------------------------

btasker changed status from 'Resolved' to 'Closed'