Friday, April 8, 2011

TiVo to Mac to Apple TV — Part IV, Fine Tuning

In my previous post, TiVo to Mac to Apple TV — Part III, One-Stop Shopping, I told how I use kmttg to download onto my Mac movies from the Turner Classic Movies HD channel that have been recorded on my TiVo Premiere. During the download, kmttg simultaneously decrypts the file: removes the copy protection so that the resulting file on my Mac is an ordinary .mpg file. Then, since .mpg files can't be used by an Apple TV, I tell kmttg to invoke the popular video reformatter HandBrake to re-encode the file as an .m4v file. I furthermore arrange to have HandBrake insert into that .m4v file, as soft subtitles — "soft" subtitles are subtitles which may be viewed optionally, at the user's discretion — the closed captions that were embedded in the original .mpg file.

It also works with movies from other cable channels such as Encore HD.

To do it in my last post, I created, in a text file called hb_my_appletv.enc, a kmttg "encoding profile":

# Description (single line, keep short)
Handbrake AppleTV: m4v container, AppleTV preset (h.264 video, aac & ac3 audio)

# Encode command or script (single line)# Known keywords: FFMPEG, MENCODER, HANDBRAKE, PERL, INPUT, OUTPUT, PWD, CPU_CORES, SRTFILE
HANDBRAKE -i INPUT --cpu CPU_CORES --preset AppleTV --subtitle 1 -f m4v -D 1,1 -A "Stereo","AC3" --rate 29.97 -v -o OUTPUT

# Encoded output file extension
m4v

That hb_my_appletv encoding profile has drawbacks when kmttg uses it, though. Its main problem is a bit on the geeky side: it fails to "detelecine" the film-based video that it encodes in the .m4v output file.

Now for some extremely nerdy stuff shown in italics with a yellow background to explain what I've just said (feel free to skip over it):


When a movie is shown on 1080i HDTV, it undergoes a "telecine" process to compensate for the fact that motion picture film consists of only 24 frames per second. The 1080i format used by TCM HD and most other HD movie channels uses 30 video frames per second. (Actually, for highly technical reasons, that 30 fps is slightly lowered to 29.97 fps.)


The "telecine" process, also known as "3:2 pulldown" or "2:3 pulldown," shows some of the original film's frames to the video camera for a slightly longer time than it shows other of the film's frames.


Each second, as 24 film frames are being exposed to the video camera, the video camera is recording 29.97 video frames — not 24 video frames — in such a way that each video frame is divided into two video "fields." The video fields are captured by the video camera at a "field rate" of (29.97 x 2), or 59.94, fields per second.


The division of each video frame into two separate video fields is called "interlacing." The "i" in 1080i indicates that the video is interlaced.


Each video field is missing exactly half of the horizontal scan lines that make up a complete video frame. For example, in 1080i video each frame has 1080 scan lines. Lines 1, 3, 5, 7, ... , 1079 (the odd-numbered lines) make us the first field in each pair of video fields. Lines 0, 2, 4, 6, ... , 1078 (the even-numbered lines) make up the second field of each pair.


Ordinarily, each pair of fields — even though the two fields in the pair are time-offset by a small fraction of a second  — can be thought of as making up a single video frame. In telecined video, though, problems arise when you do that.


The root of the problem is that the .m4v file which HandBrake creates isn't permitted to use interlaced video. It has to use "progressively scanned" (or, more simply, "progressive") video, meaning that each video frame is kept intact and not divided into two successive, time-offset video fields.


When interlaced video is re-encoded as progressive video — for instance, when kmttg's use of HandBrake invokes the hb_my_appletv encoding profile shown above — some of the film's frames are presented for a duration long enough to show up, in the original interlaced video, in three successive video fields instead of just two. Accordingly, in the progressive video that kmttg's re-encoding process creates, there wind up being two video frames out of each group of five successive video frames that are composites of two successive film frames.


When that happens, objects that are in motion on the screen (or seem to be moving because the film camera is panning) show up in two slightly different places in the progressive video frame. This graphic shows an exaggerated version of the "jaggies" that result:






Obviously, if you use a computer-based video player such as VLC or QuickTime Player that can freeze video motion and then do frame-by-frame video advance, you can readily see the jaggies that occur in precisely two successive progressive video frames out of each group of five progressive frames, i.e., the two of the five that are derived from different film frames. But the important thing is that you can also see a lack of smoothness, clarity, and motion uniformity when the resulting progressive video is watched at normal speed.


All this is due to the fact that, when interlaced video that has been telecined from 24-fps film at 29.97 video frames per second (actually, 59.94 video fields per second) is re-encoded as 29.97-fps progressive video that does not employ video fields at all, jaggies result ... and the motion you see on a TV screen loses clarity, uniformity, and smoothness.


The problem goes away, though, if you "detelecine" the interlaced video as it is being re-encoded as progressive video. Luckily, HandBrake knows how to do that. If you specify its --detelecine parameter, it can detect which two frames (or, actually, which two field pairs) out of every five field pairs in the 1080i input video had the paired two fields come from different film frames. Knowing that, it can simply strip out the redundant information which the telecine process used to pad out the 24 fps of the movie to get 29.97-fps interlaced video.


The result is that each video frame of the .m4v output matches up exactly, one for one, with a single frame of the original film.


Clearly, however, this works properly only when HandBrake is permitted to reduce the 29.97-fps frame rate of the input .mpg video to 24 fps in the .m4v output. In other words, HandBrake's video output has to wind up with the same frame rate as the original film had: 24 frames per second. This requires not giving HandBrake a -r or --rate setting — such as, for instance, --rate 29.97.

OK. That's the end of the nerdy stuff. The takeaway is that "telecined" video is padded out to a frame rate higher than that of the original film. I need a kmttg encoding profile that employs HandBrake's --detelecine option ... but does not specify its -r or --rate setting at all. This will turn the 29.97-fps 1080i video in the input file into 24-fps video in the output. Each of the output video frames will represent exactly one film frame from the original movie.

Here is my hb_my_appletv_1280 kmttg encoding profile that does all that:

# Description (single line, keep short)
m4v container, soft subtitles, 1280 x ___ video, detelecine

# Encode command or script (single line)# Known keywords: FFMPEG, MENCODER, HANDBRAKE, PERL, INPUT, OUTPUT, PWD, CPU_CORES, SRTFILE
HANDBRAKE -i INPUT --cpu CPU_CORES -e x264 -q 20.0 -X 1280 --subtitle 1 --detelecine --loose-anamorphic -x cabac=0:ref=2:me=umh:b-adapt=2:weightb=0:trellis=0:weightp=0 -a 1,1 -E faac,copy:ac3 -B 160,160 -6 dpl2,auto -R Auto,Auto -D 1,1 -A "AAC","AC3" -f m4v -v -o OUTPUT

# Encoded output file extension
m4v

It is the --subtitle 1 specification that tells HandBrake to convert closed captions into "soft" subtitles, by the way.

(This page gives a full rundown on the various HandBrake settings such as, --subtitle--detelecine and --loose-anamorphic. You can learn what settings are specified in HandBrake's AppleTV and other built-in presets here.)

If you use the Finder's Duplicate command to make a copy of (say) the hb_appletv.enc file that already exists in the encode folder that comes with kmttg, you can then use TextEdit to edit the copy of the file. Copy the above text and paste it in place of the text already in the file, then save the file as hb_my_appletv_1280.enc. Launch kmttg (or if it is already running, choose the "Refresh Encoding Profiles" item in its File menu). Then you can select hb_my_appletv_1280 in its "Encoding Profile" popup menu.

Shortening the Encode

When I do that and then encode a video in kmttg using this hb_my_appletv_1280 encoding profile, the encode process takes up to, say, ten hours, the exact time depending on a number of factors. To shorten the encode time (at the expense of having lower video resolution in the output file) I created the kmttg encoding profile going by the filename hb_my_appletv_720.enc ...

# Description (single line, keep short)
m4v container, soft subtitles, 720 x ___ video, detelecine

# Encode command or script (single line)# Known keywords: FFMPEG, MENCODER, HANDBRAKE, PERL, INPUT, OUTPUT, PWD, CPU_CORES, SRTFILE
HANDBRAKE -i INPUT --cpu CPU_CORES -e x264 -q 20.0 -X 720 --subtitle 1 --detelecine --loose-anamorphic -x cabac=0:ref=2:me=umh:b-adapt=2:weightb=0:trellis=0:weightp=0 -a 1,1 -E faac,copy:ac3 -B 160,160 -6 dpl2,auto -R Auto,Auto -D 1,1 -A "AAC","AC3" -f m4v -v -o OUTPUT

# Encoded output file extension
m4v

... to give me output video with not 1280 but 720 pixels per scan line across the screen. When I tell kmttg to encode with that hb_my_appletv_720 profile ...



... I wind up with an .m4v output file that is suitably detelecined, but it only has video resolution comparable to that of an ordinary DVD. The encode process takes much less time, though, so I consider the loss of video resolution to be a very small price to pay.

Monday, April 4, 2011

TiVo to Mac to Apple TV — Part III, One-Stop Shopping

In TiVo to Mac to Apple TV — Part I, Moving Movies I showed how I make Apple TV videos out of movies that my TiVo Premiere has recorded from, say, the Turner Classic Movies HD channel, and that I've then downloaded to my Mac. Then in TiVo to Mac to Apple TV — Part II, Capturing Captions I showed how to turn the closed captions in the original video into "soft" subtitles in the Apple TV version: subtitles I can turn on and off at will when I watch the video on my Apple TV.

I've now discovered a better way. It dawned on me that I can use a single application, kmttg, to do it all.

kmttg is a so-called "TiVoToGo" app that knows how to download recordings from a TiVo to a Mac or a PC over a home computer network. It's also a Swiss army knife of useful tools for decrypting and re-encoding the downloaded recordings for other uses.



In the above screenshot, kmttg is set to download and decrypt "The Picture of Dorian Gray," which my TiVo Premiere has already recorded from TCM HD. After the decryption takes place, I'll have a The Picture of Dorian Gray (03_12_2011).mpg file on my Mac, in whose name the (03_12_2011) reflects the date the movie was recorded by the TiVo. The .mpg filename extension reflects the format of the video that results from kmttg's "decrypt" stage ... a format that will not work on an Apple TV.

The checkmark by kmttg's "metadata" stage — next to the "Start Jobs" button, near the top of the kmttg window — tells kmttg to make a copy of this information:

title : The Picture of Dorian Gray
seriesTitle : The Picture of Dorian Gray
description : Corrupted by a lord, Oscar Wilde's London aristocrat stays young, but his portrait begins to age. 
time : 2011-03-12T13:30:00Z
mpaaRating : N8
movieYear : 1945
isEpisode : true
isEpisodic : false
showingBits : 1
starRating : x5
displayMajorNumber : 890
callsign : TCMHD
seriesId : MV006454
vActor : Sanders|George
vActor : Hatfield|Hurd
vActor : Lansbury|Angela
vActor : Reed|Donna
vActor : Gilmore|Lowell
vActor : Lawford|Peter
vActor : Fraser|Richard
vActor : Owen|Reginald
vActor : Bilbrook|Lydia
vActor : Lowry|Morton
vActor : Walton|Douglas
vActor : Forbes|Mary
vActor : Greig|Robert
vActor : Carpenter|Lisa
vActor : MacGill|Moyna
vActor : Bevan|Billy
vActor : Mander|Miles
vDirector : Lewin|Albert
vProducer : Berman|Pandro S.
vProgramGenre : Horror
vSeriesGenre : Horror
vSeriesGenre : Movies

That metadata information accompanied the movie itself when the TiVo recorded it. In certain situations, it's nice to save the original metadata information somewhere.

The checkmark next to the "decrypt" stage in the screenshot tells kmttg to not only download the movie file but also decode the downloaded file, removing TiVo's copy protection from it. This is perfectly legal to do, as long as you don't share the decrypted recording with other people.

I put a checkmark by the "encode" stage in the screenshot to tell kmttg to convert the downloaded movie file so it will be usable by an Apple TV. It learns what kind of conversion to do from the encoding profile I've selected just below the "Start Jobs" button: hb_my_appletv.

hb_my_appletv is an encoding profile I defined myself. In the ~/Desktop/kmttg_v0p7t/encode folder which came with kmttg are several files with an .enc extension, among them hb_appletv.enc. These are actually text files. After quitting kmttg, I duplicated hb_appletv.enc in Finder, changed the duplicated file's name to hb_my_appletv.enc, and opened it in TextEdit, seeing:

# Description (single line, keep short)
Handbrake AppleTV: m4v container, AppleTV preset (h.264 video, aac & ac3 audio)

# Encode command or script (single line)
# Known keywords: FFMPEG, MENCODER, HANDBRAKE, PERL, INPUT, OUTPUT, PWD, CPU_CORES, SRTFILE
HANDBRAKE -i INPUT --cpu CPU_CORES --preset AppleTV -f m4v -D 1,1 -A "Stereo","AC3" --rate 29.97 -v -o OUTPUT

# Encoded output file extension
m4v

The line beginning with HANDBRAKE is the important one. It tells kmttg how to invoke the popular video re-formatter HandBrake to encode my "The Picture of Dorian Gray" file for Apple TV as a file using the .m4v filename extension.

This HANDBRAKE ... line in the original hb_appletv.enc file doesn't, however, tell HandBrake to capture the closed captions that are embedded in the input .mpg file.

To do that, I need (I'm still in TextEdit) to change the line that begins with HANDBRAKE to include:

--subtitle 1

That gives, as the full contents of the modified hb_my_appletv.enc file:

# Description (single line, keep short)
Handbrake AppleTV: m4v container, AppleTV preset (h.264 video, aac & ac3 audio)

# Encode command or script (single line)
# Known keywords: FFMPEG, MENCODER, HANDBRAKE, PERL, INPUT, OUTPUT, PWD, CPU_CORES, SRTFILE
HANDBRAKE -i INPUT --cpu CPU_CORES --preset AppleTV --subtitle 1 -f m4v -D 1,1 -A "Stereo","AC3" --rate 29.97 -v -o OUTPUT

# Encoded output file extension
m4v

This insertion of --subtitle 1 simply says that HandBrake should include as optionally displayed "soft" subtitles in the output .m4v file the closed captions, embedded in the input .mpg file, whose "track number" is 1.

The line beginning with HANDBRAKE and ending with OUTPUT is a single line of text. In the TextEdit window it wraps around. I need to be sure, when I'm editing the line, not to accidentally hit the Return key, which would divide the one line into two and make the encoding profile fail to work properly in kmttg later on.

I can then save the edited hb_my_appletv.enc file and quit TextEdit. The next time I launch kmttg, I will be able to choose the encoding profile hb_my_appletv, as seen in the screenshot above.

Tips and pointers:

1. In iTunes Preferences: Advanced ...



... "Copy files to iTunes Media folder when adding to library," if checked, will cause iTunes to make a brand new copy of your (big) .m4v file when it is added to your iTunes Movies library, so to prevent that from happening, make sure "Copy files ... " is unchecked.

2. kmttg wants to run AtomicParsely at the end of its "encode" stage. AtomicParsely inserts the metadata information I mentioned earlier in this post into the encoded .m4v file. I find, unfortunately, that using AtomicParsely causes problems later on, when the .m4v file has been added to the iTunes Movies library. So, prior to downloading and decrypting The Picture of Dorian Gray.mpg and then encoding The Picture of Dorian Gray.m4v, I disabled AtomicParsely in the kmttg configuration:



Under the Programs tab I cleared the box next to AtomicParsely, and clicked OK. (To bring up kmttg's configuration window, just choose "Configure ... " from kmttg's File menu.)

3. Instead of using AtomicParsely, after kmttg is done with its "encode" stage and the (in this example) The Picture of Dorian Gray (03_12_2011).m4v file has been generated, I use the freeware application MetaX to specify its metadata tags. In Finder, I delete the date information in the filename to obtain The Picture of Dorian Gray.m4v. I drag the file's icon to the MetaX icon on my Dock ... or else, I launch MetaX and use its Open icon to locate The Picture of Dorian Gray.m4v. When I do so, I'm asked whether the file is a Movie, a TV Show, or a Music Video, and I choose Movie. MetaX finds the movie in an online database and fills in its metadata tags automatically:



Notice that it has found poster art for the movie. I can change it or any of the other metadata tags it has filled in for me. Once I am satisfied, I click on the Write & Share icon. MetaX takes a few minutes to rewrite my The Picture of Dorian Gray.m4v file with the new metadata tags filled in, and then the file is automatically added by MetaX to my iTunes Movies library.

5. When you download kmttg and run it for the first time, it automatically downloads various pieces of software that it wants to have in its arsenal, HandBrake included. However, the "command line interface" (CLI) version of HandBrake that kmttg downloads is currently an obsolete one, version 0.9.4. Not only does 0.9.4 have some bugs in it that can impair its use for kmttg encodes, the latest official version (0.9.5) of HandBrake CLI can run a whole lot faster on newer Macs that support so-called "64-bit" code.

You are well advised to go to this page and click on the "Download (64 bit Intel)" link under Mac OS. If you have an older Mac, click on the "Download (32 bit Intel)" link instead. You'll receive a disk image file called HandBrake-0.9.5-MacOSX.5_CLI_x86_64.dmg. (The final "_64" will be missing for the 32-bit version.)

Open that .dmg file and you will obtain a virtual disk on your desktop with the same name as the .dmg file. Open the virtual disk and you'll see a window containing a HandBrakeCLI app and a doc folder. Create a HandBrake 0.9.5 CLI folder somewhere (say, on your Desktop or in your Applications folder) and drag both of those items to it, thereby making copies of the two items. (You can then Trash the .dmg file and eject the virtual disk.)

I actually put my HandBrake 0.9.5 CLI folder in a subfolder of my Applications folder. Then I launched kmttg and, under its File menu, selected Configure ... . A kmttg configuration window opened, and I clicked on the Programs tab. Then I double-clicked in the handbrake box and navigated to my HandBrake 0.9.5 CLI folder, and within that to my HandBrakeCLI app. I clicked on Choose File to see:



I clicked on OK and the kmttg configuration window went away. From that point on, my HandBrake encodes in kmttg used the 0.9.5 version.