Anyone have a script I can tweak to reencode my h264 media to hevc

JaredBusch

Like many people that have been encoding things for themselves for years, most of my library is using the h264 codec. Now that hevc/h265 is supported on almost everything, I want to reencode things.

Also new stuff I record is often still in h264 because I do not own anything that natively records to hevc yet.

As far as I know, the best tool for the job is ffmpeg as it has been for years.

Goals:

Run a script against a directory
- Will be running on Fedora 32/33
have it recursively check all media files of type mkv, mp4, m4v, avi, or mov with ffprobe. It looks like this.

ffprobe -v error -select_streams v:0 -show_entries stream=codec_name -of default=noprint_wrappers=1:nokey=1 /home/jbusch/videos/Daerma/DaermaAdventures_2020-07-08.mkv
h264

If the result is not hevc then execute ffmpeg to convert the file.
- output as an mkv in a folder h265_converted while maintaining the original folder structure
If the result is already hevc move to the folder h265_converted
If the file name contains h264 or H264 change that to H265 or HEVC (haven't decided which is better)
- The prename command works well for this with regex
If the file name contains no identifier of the encoding type, append _H265 or _HEVC before the extension.

This is the ffmpeg command I'm using. I welcome suggestions, but it seems to work well for my needs.

fmpeg -hide_banner -i /home/jbusch/videos/Daerma/DaermaAdventures_2020-07-08.mkv -c:v hevc -preset medium -crf 28 -pix_fmt yuv420p10le -tag:v hvc1 -c:a copy /home/jbusch/videos/h265_converted/Daerma/DaermaAdventures_2020-07-08.mkv

JaredBusch

ok, here is what I have right now.

It is not handling the destination directory recursively yet. But I'm tired.

#!/bin/sh
destdir="/home/jbusch/hdd/plexmedia/H265/";
sourcedir="/home/jbusch/hdd/plexmedia/Movies/";
tempdir="/home/jbusch/hdd/plexmedia/";

IFS=$'\n'; # Google says this is hacky way to fix the find screwing up spaces in directories and filenames.
results=$(find ${sourcedir} -maxdepth 10 -regex ".*\.\(mkv\|mp4\|wmv\|flv\|webm\|mov\|avi\|m4v\)");

for fullpath in $results
do  
    filename="${fullpath##*/}";
    base="${filename%.[^.]*}";
    ext="${filename:${#base} + 1}";

# Troubleshooting 
#echo "##########";
#echo "Full Path: ${fullpath}";
#echo "Full File Name: ${filename}";
#echo "Source Directory: ${sourcedir}";
#echo "File Name: ${base}";
#echo "File Extension: ${ext}";
#echo "";

echo "##################";
echo "Getting info of $base";
    eval $(ffprobe -v quiet -show_format -of flat=s=_ -show_entries stream=duration,codec_name -sexagesimal "$fullpath");
    duration=${format_duration};
    current_codec=${streams_stream_0_codec_name};
echo "Duration = $duration , Current Codec: $current_codec";
# If the filename has x264/X264/h264/H264 replace it with HEVC
destfilename=$(echo $base | sed -e "s/[xXhH]264/HEVC/g");
# if the destination filename does not contain HEVC, append it.
destfilename=$(echo $destfilename | sed -e "/HEVC/!s/$/_HEVC/");
# Troubleshooting 
#echo "New File Name: $destfilename";

if [ "$current_codec" == "hevc" ]
then    
    echo "This video is already encoded as HEVC, skipping reencode.";
    echo "Moving original file to destination with updated name: $destfilename.mkv";
    mv $fullpath ${destdir}${destfilename}.mkv;
else
    echo "Reencoding video to HEVC";
    ffmpeg -hide_banner -loglevel quiet -stats -i "$fullpath" -c:v hevc -preset medium -crf 28 -pix_fmt yuv420p10le -tag:v hvc1 -c:a copy "${tempdir}current_encode.mkv"
    echo "Moving temp file to destination with updated name: $destfilename.mkv";
    mv ${tempdir}current_encode.mkv ${destdir}${destfilename}.mkv;
    echo "Renaming original file to .converted";
    mv $fullpath ${fullpath}.converted;
fi
done

This is what it looks like right now. I am not sure why the x265 info is not suppressed. I only want the frame line showing for progress visibility.

JaredBusch

Okay, updated to silence the noise and make the directories, but I did that the forceful way.

#!/bin/sh
sourcedir="/home/jbusch/hdd/plexmedia/Movies/";
tempdir="/home/jbusch/hdd/plexmedia/";

IFS=$'\n'; # Google says this is hacky way to fix the find screwing up spaces in directories and filenames.
results=$(find ${sourcedir} -maxdepth 10 -regex ".*\.\(mkv\|mp4\|wmv\|flv\|webm\|mov\|avi\|m4v\)");

for fullpath in $results
do
filename="${fullpath##*/}";
base="${filename%.[^.]*}";
ext="${filename:${#base} + 1}";
# Destination is the source directory but substitute H265 for Movies
destdir="${fullpath:0:${#fullpath} - ${#filename}}";
destdir=$(echo $destdir | sed -e 's/Movies/H265/');
# Recursively create destination directory
mkdir -p $destdir;

# Troubleshooting 
#echo "##########";
#echo "Full Path: ${fullpath}";
#echo "Full File Name: ${filename}";
#echo "Destination Directory: ${destdir}";
#echo "File Name: ${base}";
#echo "File Extension: ${ext}";
#echo "";

echo "##################";
echo "Getting info of $base";
    eval $(ffprobe -v quiet -show_format -of flat=s=_ -show_entries stream=duration,codec_name -sexagesimal "$fullpath");
    duration=${format_duration};
    current_codec=${streams_stream_0_codec_name};
echo "Duration = $duration , Current Codec: $current_codec";
# If the filename has x264/X264/h264/H264 replace it with HEVC
destfilename=$(echo $base | sed -e "s/[xXhH]264/HEVC/g");
# if the destination filename does not contain HEVC, append it.
destfilename=$(echo $destfilename | sed -e "/HEVC/!s/$/_HEVC/");
# Troubleshooting 
#echo "New File Name: $destfilename";

if [ "$current_codec" == "hevc" ]
then    
    echo "This video is already encoded as HEVC, skipping reencode.";
    echo "Moving original file to destination with updated name: $destfilename.mkv";
    mv $fullpath ${destdir}${destfilename}.mkv;
else
    echo "Reencoding video to HEVC";
    ffmpeg -y -hide_banner -loglevel quiet -stats -i "$fullpath" -c:v hevc -preset medium -crf 28 -pix_fmt yuv420p10le -x265-params log-level=error -tag:v hvc1 -c:a copy "${tempdir}current_encode.mkv"
    echo "Moving temp file to destination with updated name: $destfilename.mkv";
    mv ${tempdir}current_encode.mkv ${destdir}${destfilename}.mkv;
    echo "Renaming original file to .converted";
    mv $fullpath ${fullpath}.converted;
fi
done

JaredBusch

ok. script now running on a spare box... only getting 0.5x on the encode. so half speed.

Not exactly a power machine, but it is spare so it can just sit and run for days.

[jbusch@encoder ~]$ cat /proc/cpuinfo 
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 60
model name	: Intel(R) Core(TM) i5-4590T CPU @ 2.00GHz
stepping	: 3
microcode	: 0x28
cpu MHz		: 2594.217
cache size	: 6144 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts md_clear flush_l1d
vmx flags	: vnmi preemption_timer invvpid ept_x_only ept_ad ept_1gb flexpriority tsc_offset vtpr mtf vapic ept vpid unrestricted_guest ple shadow_vmcs
bugs		: cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit srbds
bogomips	: 3991.10
clflush size	: 64
cache_alignment	: 64
address sizes	: 39 bits physical, 48 bits virtual
power management:

gjacobse

It’s been quite a while since I worked with the different formats-
What is the benefit of going hevc?

1337

@gjacobse said in Anyone have a script I can tweak to reencode my h264 media to hevc:

It’s been quite a while since I worked with the different formats-
What is the benefit of going hevc?

25-50% better compression at the same video quality compared to h264.

But it's not worth it to go from h264 to h265 just to save space - unless you can re-encode really, really fast. If you have a large enough movie collection where the space saving is meaningful (many TBs) it will take years to re-encode it, unless you have a monster machine to do it on.

JaredBusch

@Pete-S said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@gjacobse said in Anyone have a script I can tweak to reencode my h264 media to hevc:

It’s been quite a while since I worked with the different formats-
What is the benefit of going hevc?

25-50% better compression at the same video quality compared to h264.

But it's not worth it to go from h264 to h265 just to save space - unless you can re-encode really, really fast. If you have a large enough movie collection where the space saving is meaningful (many TBs) it will take years to re-encode it, unless you have a monster machine to do it on.

I have spare compute just sitting there. And much of this is personal media encoded over the years.

1337

@JaredBusch said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@Pete-S said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@gjacobse said in Anyone have a script I can tweak to reencode my h264 media to hevc:

It’s been quite a while since I worked with the different formats-
What is the benefit of going hevc?

25-50% better compression at the same video quality compared to h264.

But it's not worth it to go from h264 to h265 just to save space - unless you can re-encode really, really fast. If you have a large enough movie collection where the space saving is meaningful (many TBs) it will take years to re-encode it, unless you have a monster machine to do it on.

I have spare compute just sitting there. And much of this is personal media encoded over the years.

Sure, I'm just saying it's usually not worth it if you consider how long it will take and what you'll save. But I mean you can do it for other reasons and have the space saved as a bonus.

For instance if the speed you have is 0.5x and the videos you have are 2GB per hour of recording, you will save 0.5GB for every hour of encoding.

If you want to save 1 TB for instance that means you need to have your machine encoding at 100% CPU for 2000 hours straight. That's a little more than 40 days. If your computer consumes 100W at full load you will consume $28 of electricity (14 cents per kWh). And a 1TB harddrive is something like $40.

If you have a larger movie collection say 10,000 movies @ 90 minutes runtime, you'll have 30,000 hours of encoding ahead of you before you've finished that job. That will take 3.5 years.

JaredBusch

@Pete-S said in Anyone have a script I can tweak to reencode my h264 media to hevc:

If you have a larger movie collection say 10,000 movies @ 90 minutes runtime, you'll have 30,000 hours of encoding ahead of you before you've finished that job. That will take 3.5 years.

And who would legally have something like that? Anyone who was in that scenario would just download as needed.

DustinB3403

@JaredBusch said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@Pete-S said in Anyone have a script I can tweak to reencode my h264 media to hevc:

If you have a larger movie collection say 10,000 movies @ 90 minutes runtime, you'll have 30,000 hours of encoding ahead of you before you've finished that job. That will take 3.5 years.

And who would legally have something like that? Anyone who was in that scenario would just download as needed.

That's only like .02% of all movies ever made in the world. . . don't you have enough capacity for all of those C and D rate movies @JaredBusch ....

marcinozga

https://github.com/HaveAGitGat/Tdarr - this will transcode your entire library.

JaredBusch

@marcinozga said in Anyone have a script I can tweak to reencode my h264 media to hevc:

https://github.com/HaveAGitGat/Tdarr - this will transcode your entire library.

I found that, but I don't need a web app to do a basic conversion. Why install MongoDB, etc...

DustinB3403

@JaredBusch said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@marcinozga said in Anyone have a script I can tweak to reencode my h264 media to hevc:

https://github.com/HaveAGitGat/Tdarr - this will transcode your entire library.

I found that, but I don't need a web app to do a basic conversion. Why install MongoDB, etc...

They link to a desktop driven solution as well.

JaredBusch

@DustinB3403 said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@JaredBusch said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@marcinozga said in Anyone have a script I can tweak to reencode my h264 media to hevc:

https://github.com/HaveAGitGat/Tdarr - this will transcode your entire library.

I found that, but I don't need a web app to do a basic conversion. Why install MongoDB, etc...

They link to a desktop driven solution as well.

That required HandBrake CLI.

DustinB3403

@JaredBusch said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@DustinB3403 said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@JaredBusch said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@marcinozga said in Anyone have a script I can tweak to reencode my h264 media to hevc:

https://github.com/HaveAGitGat/Tdarr - this will transcode your entire library.

I found that, but I don't need a web app to do a basic conversion. Why install MongoDB, etc...

They link to a desktop driven solution as well.

That required HandBrake CLI.

I assume that's an issue for you.

stacksofplates

@JaredBusch said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@marcinozga said in Anyone have a script I can tweak to reencode my h264 media to hevc:

https://github.com/HaveAGitGat/Tdarr - this will transcode your entire library.

I found that, but I don't need a web app to do a basic conversion. Why install MongoDB, etc...

Just use a container and you don't need to install any of that. They have a Dockerfile in the repo.

scottalanmiller

@JaredBusch said in Anyone have a script I can tweak to reencode my h264 media to hevc:

@Pete-S said in Anyone have a script I can tweak to reencode my h264 media to hevc:

If you have a larger movie collection say 10,000 movies @ 90 minutes runtime, you'll have 30,000 hours of encoding ahead of you before you've finished that job. That will take 3.5 years.

And who would legally have something like that? Anyone who was in that scenario would just download as needed.

I don't, but had I stayed collecting like I used to, I'd be pretty close at this point.

Unsolved Anyone have a script I can tweak to reencode my h264 media to hevc