HLS-27: Ability to define resolutions with bitrates



Issue Information

Issue Type: Sub-task
 
Priority: Major
Status: Closed

Reported By:
Ben Tasker
Assigned To:
Ben Tasker
Project: HLS Stream Creator (HLS)
Resolution: Done (2019-07-26 09:03:44)
Target version: 1.0,

Created: 2017-07-19 09:43:31
Time Spent Working
Estimated:
 
120 minutes
Remaining:
 
120 minutes
Logged:
 
0 minutes
Child of: HLS-25: Additional flags/features to aid Low Latency Streaming


Description
It's currently possible to create a multi-bitrate HTTP Adaptive Stream by using the -b flag.

However, if the video is high-resolution (say 720p or 1080i) then lower bitrates simply increase the compression, leading to diminished playback quality.

It'd be good to be able to instead (or if possible, as well) pass in a list of resolutions instead so that the player can step down to a lower resolution instead (rather than getting more blocky video at the same res)


Toggle State Changes

Activity


I don't know how trivial it'll be to be able to defined both bitrate and resolution (though it may be do-able).

For VoD streams, could potentially pass just the resolution in, then in post-processing calculate the average bitrate of the stream and update the master manifest, but that won't be an option for linear streams (as there's no post-processing stage).

Being able to drop resolution has some very useful applications to Ultra-low latency HLS though (which is why this is a child of HLS-25)
I think a good starting point would be to look at implementing the ability to specify a resolution along with the bitrate, something like -b 2000,1024-1280x1080,512-400x300 with the resolution being left untouched for any bitrate that doesn't have a resolution specified (though we'll still need to probe the video to find out what resolution to write into the manifest).

I do have some misgivings about this functionality though, based on player behaviour.

VideoJS for example, when used with the HLS plugin, tries to be a bit too smart for my liking. IMO it risks degrading the user's playback experience:

Say we have a video with the following bitrates/resolutions available:

- 2Mbps at 1080p
- 1Mbps at 720p
- 768k at 600x400
- 512k at 300x200

and you visit a webpage where, by default, the player is contained in a 300x200 div.

VideoJS will detect that the highest resolution it currently needs to fetch is 600x400, which is the 768k stream. It won't fetch a higher bitrate as the resolutions are higher than currently displayed (and video-js-hls will only go up to one size larger than the current container).

All well and good so far (as there's no point in fetching a 1080p stream for a 300x200 window).

However, the issue starts when you click the fullscreen button. Your window size is now (say) 1080p, so the player decides it should try and get the 2Mbps stream.

Unfortunately, you're on a 1.5Mbps connection, so can't fetch it fast enough. You've now got (fullscreen) slightly stuttery playback until the player realises it's going to have to fetch the 720p instead. Equally badly, for a short period before that stuttering, you've had some 600x400 video stretched to fill your screen - even if you then have the bandwidth to fetch the 1080p content. Not exactly pleasant.

It could all be avoided by the player not being so damned smart. If it had been basing decisions on bandwidth alone, then there's a reasonable chance you'd have been on the 720p stream by the time you clicked fullscreen, and the resulting change in displayed resolution would have had no impact.

It is, of course, an exercise in saving bandwidth (as it is fairly wasteful to deliver 1080p if you're only going to display it in a small container), but speaking personally I much prefer the user experience without this behaviour present. I'd prefer (if anything) that certain bitrates were potentially discarded on the basis that the display can't handle that resolution (so a tiny phone will never try the 4K rendition, for example) but even that's less than perfect.

There's more information on this player behaviour (including how to neuter it) here - https://github.com/videojs/videojs-contrib-hls/blob/master/docs/bitrate-switching.md - but I guess the point I'm trying to make, is that caution should be used when using this feature (once implemented) as it may potentially degrade the playback experience. Although I've used VideoJS as an example, it won't be the only player that does this.

So we'll need to find a (much) more concise way to note that in the README too.
Getting back to it, though. If we want to implement the ability to specify resolution as part of the bitrate (i.e. -b 2000-1280x720 then there are a few locations that'll need changing within the script.

- Function appendVariantPlaylistentry will need adjusting to check for the resolution being specified, and acting accordingly
- The iterative loop will need adjusting to check for presence of resolution and acting accordingly
- Function createStream will also need adjusting do that it can set the resolution

It's probably fine to let the manifest suffix follow the input values, so we'd end up with stream_2000-1280x720.m3u8 as a variant bitrate manifest

Repo: HLS-Stream-Creator
Commit: 2cac89e5184c89f4f4634d222bd99214feb86c5a
Author: B Tasker <github@<Domain Hidden>>

Date: Sat Feb 03 10:38:57 2018 +0000
Commit Message: Implement ability to specify resolution with bitrates. See HLS-27

This is just a first pass at this, and I fully expect it to break in a variety of weird and wonderful ways.

There's also some reliance on globals that I don't like (see comments in file), so that's still in need of tidying up. We should probably also think about testing that none of the given resolutions are higher than that of the source.

But, you can now pass a resolution in as part of the bitrate string: bitrate-widthxheight

As an example, ran the following test encode
$ ./HLS-Stream-Creator.sh -i testvid.mp4 -b 2744,1000-1280x720 -s 10

The original is 1080p at 2744k so that should give us a 1080p rendition as well as a 720p.

Master playlist is generated correctly
$ cat testvid_master.m3u8
testvid_2744.m3u8
testvid_1000-1280x720.m3u8


Segments report approximately the right bitrate (used vbr to speed up the test), and also the correct resolution
$ ffprobe -i testvid_1000_00001.ts
Input #0, mpegts, from 'testvid_1000_00001.ts':
  Duration: 00:00:13.70, start: 12.890956, bitrate: 989 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 100 kb/s

$ ffprobe -i testvid_2744_00001.ts
Input #0, mpegts, from 'testvid_2744_00001.ts':
  Duration: 00:00:13.70, start: 12.890956, bitrate: 2335 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 100 kb/s


Not been tested with a linear stream yet (though it should work)



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent

GitHub-Hookshot/2e08413


View Commit

Commit 2cac89e takes a first swipe at getting this implemented.

Specifying the resolution for any given bitrate is optional, and should be appended to the bitrate in the format -widthxheight giving something like 1000-1280x720.

Test encode ran well enough, though there are a few things I'd like to tidy up (see the commit message for more info).

We're also not currently writing the resolution into the master manifest for any variant where the res wasn't explicitly defined on the command line. Feels a bit incongruent so should consider whether that should be fixed. But, as we'd need to probe the input, that may be an issue for linear streams (or may not).
Testing creating a linear stream with multiple bitrates/resolutions from a VoD source

ben@milleniumfalcon:/tmp$ ~/Documents/src.old/HLS-Stream-Creator/HLS-Stream-Creator.sh -i http://streamingtest.bentasker.co.uk/Big_Buck_Bunny-Flash/bbb_sunflower_native_60fps_stereo_abl.mp4 -l -b 13495,12000-1920x1080 -s 10 -c 0 -C


Should give us a 4K stream at around 13Mbps and a 1080p at around 12Mbps, using a constant bitrate


FFMPEG has slightly undershot the bandwidth, but the resolutions are as they should be
ben@milleniumfalcon:/tmp/output$ ffprobe -hide_banner -i bbb_sunflower_native_60fps_stereo_abl.mp4_13495_00000.ts 
Input #0, mpegts, from 'bbb_sunflower_native_60fps_stereo_abl.mp4_13495_00000.ts':
  Duration: 00:00:10.70, start: 1.412000, bitrate: 11303 kb/s
  Program 1 
    Metadata:
      service_name    : Big Buck Bunny, Sunflower version
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 4000x4500 [SAR 1:1 DAR 8:9], 60 fps, 60 tbr, 90k tbn, 120 tbc
    Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 198 kb/s
    Stream #0:2[0x102](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, fltp, 130 kb/s



ben@milleniumfalcon:/tmp/output$ ffprobe -hide_banner -i bbb_sunflower_native_60fps_stereo_abl.mp4_12000_00000.ts 
Input #0, mpegts, from 'bbb_sunflower_native_60fps_stereo_abl.mp4_12000_00000.ts':
  Duration: 00:00:12.52, start: 1.412000, bitrate: 10610 kb/s
  Program 1 
    Metadata:
      service_name    : Big Buck Bunny, Sunflower version
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1920x1080 [SAR 1:2 DAR 8:9], 60 fps, 60 tbr, 90k tbn, 120 tbc
    Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, stereo, fltp, 198 kb/s
    Stream #0:2[0x102](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 48000 Hz, 5.1, fltp, 130 kb/s


So, it looks like linear handles the resolution support correctly.

Next thing to do, then, is to get the README updated to note this new feature
As resolutions are currently passed along with the bitrate, I've renamed this FR more appropriately.

Repo: HLS-Stream-Creator
Commit: e6038a4ca467a698f27bf775e59df5192d744bd0
Author: B Tasker <github@<Domain Hidden>>

Date: Sun Feb 04 12:35:06 2018 +0000
Commit Message: Linear support for HLS-27 has now been tested, so update README to note the resolution functionality.



Modified (-)(+)
-------
README.md




Webhook User-Agent

GitHub-Hookshot/2e08413


View Commit

I'm going to leave this issue open for now, as still need to make a decision on how bothered we are that the native resolution doesn't get written into the manifest.

Repo: HLS-Stream-Creator
Commit: 97b3add98604d39f63aa6a72c5b51b92b591d36b
Author: B Tasker <github@<Domain Hidden>>

Date: Sun Feb 04 12:42:45 2018 +0000
Commit Message: Yeah, I totally forgot to commit this fix for HLS-27 :(

Adjusts the path passed into appendVariantPlaylistentry so that it doesn't include the resolution (as the manifest filenames currently don't)



Modified (-)(+)
-------
HLS-Stream-Creator.sh




Webhook User-Agent

GitHub-Hookshot/2e08413


View Commit

I've created quite a few streams now, and not had any players balk at the native bitrate not including resolution. Rather than leave this issue hanging around, I'm going to close it as complete - sufficient time has passed anyway that should probably raise a new bug if there are issues.
btasker changed status from 'Open' to 'Resolved'
btasker added 'Done' to resolution
btasker changed status from 'Resolved' to 'Closed'