HLS-10: When parrallelised transcodes finish, the bitrate for the last isn't output

Looks like this is still an issue (and affected more than the last this time)

ffmpeg command found.... continuing
Generating HLS segments for bitrate 512k - this may take some time
Generating HLS segments for bitrate 1024k - this may take some time
Generating HLS segments for bitrate 2048k - this may take some time
All transcoding processes started, awaiting completion
Encoding for bitrate 512k completed
Encoding for bitrate k completed
Encoding for bitrate k completed

Came in as https://github.com/bentasker/HLS-Stream-Creator/issues/9 from user scotia70

Hi,
great script. I've found a cosmetic issue, here's the output from a run. Note the bitrate number is missing from some of the the "encoding completed" lines.

#HLS-Stream-Creator.sh -i camera-01-20160209T101155.mp4 -s 10 -b 28,96,512,1600
ffmpeg command found.... continuing
Generating HLS segments for bitrate 28k - this may take some time
Generating HLS segments for bitrate 96k - this may take some time
Generating HLS segments for bitrate 512k - this may take some time
Generating HLS segments for bitrate 1600k - this may take some time
All transcoding processes started, awaiting completion
Encoding for bitrate (0) 28k completed
Encoding for bitrate (0) k completed
Encoding for bitrate (0) k completed
Encoding for bitrate (0) k completed

I believe it's to do with the way the PIDS array is emptied. If you reverse the order of the bitrates (and hence they are likely to finish last to first, then the output is ok.

Thanks

Taking a quick scan over, I think the user is correct. Have asked them if they want to create a pull, otherwise will look at reversing the order myself

As the user hasn't created a pull. I've made the change.

It's a temporary workaround really, as it assumes bitrates will be provided in ascending order. Rather than sorting them it makes more sense to look at where the array is being emptied and fixing the issue there.

The root of the issue almost certainly exists here - https://github.com/bentasker/HLS-Stream-Creator/blob/2711edfdc43678aa134523050a76690260f7f550/HLS-Stream-Creator.sh#L178

    # Check each PID in the array
    for i in `seq 0 $pid_length`
    do
	  # Test whether the pid is still active
	  if ! kill -0 ${PIDS[$i]} 2> /dev/null
	  then
		echo "Encoding for bitrate ${BITRATE_PROCESSES[$i]}k completed"

		if [ "$LIVE_STREAM" == "1" ] && [ `grep 'EXT-X-ENDLIST' "$OUTPUT_DIRECTORY/${PLAYLIST_PREFIX}_${BITRATE_PROCESSES[$i]}.m3u8" | wc -l ` == "0" ]
		then
		    # Correctly terminate the manifest. See HLS-15 for info on why
		    echo "#EXT-X-ENDLIST" >> "$OUTPUT_DIRECTORY/${PLAYLIST_PREFIX}_${BITRATE_PROCESSES[$i]}.m3u8"
		fi

		unset BITRATE_PROCESSES[$i]
		unset PIDS[$i]
	  fi
    done

Dropped some debug text in to get a state of the array as we cycle through

DEBUG: i: 0, PIDS array:  539 540 542  BR_PROCESSES: 128 512 1024
Encoding for bitrate 128k completed
DEBUG: i: 0, PIDS array:  540 542  BR_PROCESSES: 512 1024
Encoding for bitrate k completed

So, in principle it should have been able to pull the Bitrate there...

But, breaking that down to specific array keys

DEBUG: i: 0, PIDS array:  2979 2981  BR_PROCESSES: 512 1024
PID 0: 2979
PID 1: 2981
PID 2: 

BR 0: 
BR 1: 512
BR 2: 1024

Encoding for bitrate k completed

When we unset the PID the first element is completely removed, whereas for BITRATE_PROCESSES it's simply set to a null value.

Which would be because Line 182 (https://github.com/bentasker/HLS-Stream-Creator/blob/2711edfdc43678aa134523050a76690260f7f550/HLS-Stream-Creator.sh#L182) strips any nulls from PIDS, but no such action is performed against the BR array

Inserting the following on line 183 resolves for most cases

    BITRATE_PROCESSES=("${BITRATE_PROCESSES[@]}") # remove any nulls

There should be no race caused by doing it this way - if two processes finish in the same second then their array keys won't change until all of the pids have been checked.