HLS-12: Auto-delete expired segments

Issue Information

Issue Type: New Feature
Priority: Major
Status: Open

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: HLS Stream Creator (HLS)
Resolution: Unresolved
Components: Linear Support ,

Created: 2015-06-15 12:59:39
Time Spent Working

When the script has been told that an input stream is linear, it supports two options

- Generate full manifest
- Generate manifest with a sliding window

The latter of the two means that older segments will never be requested by the downstream client after a given period (as clients retrieving the manifest for the first time will see a newer segment as the new entry).

Those segments are therefore redundant, so should be removed.

Need to work out the best calculation to use when deciding whether a segment falls into that category though - (manifest length * segment length) * 2 may be slight overkill, but should be sufficient to ensure we don't accidentally remove something that's still live.

As this only applies to live streams, the encoding process(es) should have been forked into the background, the deletion watcher should also be forked so that the script can continue watching for encoding completion (and then kill the deletion watcher).

Toggle State Changes


One thing worth noting (and probably fixing) - if an output bitrate hasn't been specified using the -b option, the encoding process won't actually be forked into the background (as at the time of implementing HLS-6 it wasn't needed).

Without changing that, we can't run the cleanup process at the same time. So either we need to make sure it forks into the background, or auto-deletion is only supported on Adaptive streams (and to be fair, more bitrates == more files == more disk space wastage).
Where a manifest length has been specified (i.e. -c 5 ), a ffmpeg flag is now passed to wrap the segment numbering at double that number.

So, if the manifest has a sliding window of 5 segments, the highest numbered segment will be 10 - ffmpeg will then go back to 0 (overwriting the earlier file)

Repo: HLS-Stream-Creator
Commit: 1a1719a963c08cd63e9e8689f0aee3a7ec9de0f3
Author: B Tasker <github@<Domain Hidden>>

Date: Thu Jun 18 04:04:05 2015 +0100
Commit Message: Automatically overwrite expired segments in live streams. See HLS-12

Modified (-)(+)

Webhook User-Agent


View Commit

The current implementation will re-use segment names after 2 manifest lengths, so (assuming the manifest contains 3 segments at any given time), numbering will be

seg0, seg1, seg2, seg3, seg4, seg5, seg0, seg1.....etc

Which, for a directly served linear stream is fine.

However, if there's a downstream cache (for example a CDN), then the max-age header would need to be carefully set, otherwise the second request for seg0 would likely lead to a cached copy of the original being served.

There's also an odd, but real-world use-case, where a stream is linear whilst it's first airing, but later becomes VoD (for example, a football match being streamed live, and then made available as catchup afterwards).

The most efficient means of doing this is to use a linear manifest whilst streaming live, and then later republish the manifest with all segments listed to make the stream available as Vod. Because the segment names/paths are unchanged, downstream caches will likely have most or all of the segments in cache (assuming a suitable max-age) already, lowering the impact on the origin and improving delivery times to the end-user.

That use-case isn't possible with the current implementation though, because we're re-using segment filenames, so that needs to become an option (even if it's the default behaviour). Once that's done, it still won't be directly achievable (as the VoD manifest would need to be manually generated), but would at least be feasible.

So, there are 2 tasks left within this issue

- Make segment number wrapping a togglable option
- Fork single bitrate encodes (see earlier comment)
- When wrapping is disabled, there should be an option to delete old segments