Apple

Converting mkv, mpeg-2, anything from disc, to DnxHD or Prores using ffmpeg

Friday, March 25th, 2016
Let’s suppose, for the sake of argument, that you’ve been asked by a director to cut his directing reel. He hands you a DVD and a Blu-ray and asks you to grab clips from them.

There are three problems here. The first is that you really should have been handed Prores or something editable from the get-go. But let’s assume you can’t get the media any other way (because often, you just can’t), and let’s assume that this is all considered fair-use (though current law makes that a bit muddy). We’re left with two technical problems:

  1. Rip: Getting the media off the disc and into a file, which may require circumventing copy protection mechanisms and encryption.
  2. Transcode: Converting the resulting file into something editable. Avid Media Composer won’t cut a .mkv file, and works with a limited range of video codecs. FCPX and Adobe Premiere Pro, despite their focus on working efficiently with many codecs, will still have trouble with some file formats. So this step is required for them too.

Rip

Regarding getting the clips off the disc, there are several applications out there for DVD, and only one that I know of for Blu-ray. From DVD, many people use a pretty fancy open-source tool that lets you pull selected ranges of video, and saves to mp4 (the software is designed to make files that play back on media devices, not for editorial purposes). If I’ve ever had to do this hypothetical task, I’ve avoided using this tool because I want the video to transfer from disc to file with no transcoding or lossy compression in between. So I’d have used this one tool which works for both DVD and Blu-ray. The result is a .mkv file which contains all the video and audio from the disc in its original codec.

So you’ve ripped the video from the disc and you’ve got an mkv file or files. You can play these in VLC Player to have a look at their contents to be sure you’ve got what you need. But none of the NLEs with which you edit can work with the files directly. Here’s how to convert them to dnxhd in order to be able to bring them into Media Composer. We’re going to use an awesome command-line tool called ffmpeg.

Install ffmpeg

At this point, I’m going to be giving instructions that relate to computers running Apple’s OS X. I’m sure it’s possible to do the next steps on a machine running Windows or Linux, but I’m not sure of the exact process. And we’re going to be using the Terminal, which you can find in /Applications/Utilities. You might as well open a Terminal window now.

There are a number of precompiled versions of ffmpeg floating around for download, but the only way to be sure you’re getting one with support for all the necessary formats built-in is to build it from source on your machine. To compile it, you’ll need Apple’s free development tool, Xcode, or at least the Xcode command line utilities. Download and install that from Apple. Then you’ll need to install ffmpeg and its dependencies, a task that is made pretty easy by a packaging system called ‘homebrew‘. If you’ve never set a password for your user account on OS X, you’ll need to do that first (the unix ‘passwd’ command, entered at the Terminal prompt, is a fast way to do this). It’s very easy to install homebrew, you just copy and paste one long command from homebrew’s web site into your Terminal window.

Once you’ve got Xcode and homebrew installed, you’re ready to install ffmpeg. I do so by pasting the following into the terminal window:

brew install ffmpeg --with-fdk-aac --with-ffplay --with-fontconfig --with-freetype --with-frei0r --with-libass --with-libbluray --with-libcaca --with-libquvi --with-libsoxr --with-libvidstab --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-openssl --with-opus --with-rtmpdump --with-schroedinger --with-speex --with-theora --with-tools --with-x265

That should all be one long line. Hit return, and brew will start downloading tool after tool, occasionally prompting you, and eventually installing ffmpeg. This may take a while, depending on the speed of your computer and the speed of your net connection. A half-hour wouldn’t be out of the ordinary. But when it’s done you’ll have an amazing multimedia swiss army knife at your disposal. I use ffmpeg for nearly all of my transcodes, even to watermark and transcode for upload to video distribution platforms.

Transcode your files with ffmpeg

Now that ffmpeg is installed, you can transcode your mkv file into dnxhd with one line of commands entered at the Terminal prompt. Here it is:

ffmpeg -i inputfilename.mkv -vf "scale=1920:1080" -vcodec dnxhd -r 23.976 -b:v 36M -c:a pcm_s16le -ar 48000 -ac 2 outputfilename.mov

For command-line newbies, I should point out that spaces are used as a special character to separate commands and their arguments. So if there are any spaces in your filename, or in any directory along the way to your file, they need to be typed in a special way (“escaped” with a  backslash in front of it, like “\ “) so that the Unix shell knows to interpret them as a space and not as a separator. There are other special characters too. The safest way to type the filename is to let the Terminal do it for you — just type “ffmpeg -i ” (don’t type the quotes, but do type the space after the i) and then drag your input file right into the Terminal window. It’ll type the whole path to the file into the Terminal, with all spaces and special characters properly escaped. So for this command, I do:

ffmpeg -i drag_in_mkv_file -vf "scale=1920:1080" -vcodec dnxhd -r 23.976 -b:v 36M -c:a pcm_s16le -ar 48000 -ac 2 drag_in_mkv_file_and_then_edit_its_name_to_end_in_mov

Hit enter, and ffmpeg do its thing. It’ll take the video from that mkv file, uprez it to 1080p if necessary, and output it as a DNxHD36 23.98 quicktime file with stereo 16-bit 48Khz audio. If that matches your Media Composer project’s settings, the video will fast-import. Unfortunately I haven’t yet nailed down the exact audio settings, so that audio will just do a normal and not fast-import (which means some conversion is happening during the import).

If the file you’re converting isn’t 23.98p, change that part of the command line to the appropriate frame rate. You’ll likely also need to change the bitrate. There’s a list of bitrates to use for different frame rates of dnxhd here (scroll down to the dnxhd section).

You can also convert your mkv or other files to prores using ffmpeg. For our blu-ray mkv file, a line like this should work:

ffmpeg -i inputfilename.mkv -vcodec prores -profile 2 -pix_fmt yuv422p10le -acodec pcm_s16le outputfilename.mov

That’ll make a Prores422 file with 16-bit audio. You can also do other types of Prores by changing the number after “-profile” to 0 for proxy, 1 for LT, or 3 for 422HQ.

In summary

Yes you can pull clips from DVD or Blu-ray and convert them into editable media without an intermediate transcode to mp4 – by losslessly ripping to mkv and then converting to Avid DNxHD or Prores using ffmpeg.

And if you get heavier into the command-line, you’ll find that it makes complicated scripting and batch processing pretty easy to do. So having a good command-line conversion tool becomes very powerful.

An ffmpeg recipe to concatenate multiple quicktime files and convert to h264

Tuesday, April 29th, 2014
If you ever find yourself with a set of same-as-source quicktime files that have been exported from Media Composer, one per reel of a film…

–and you need to output a single h264 file of the film to upload to Dax or Pix or another of the various secure media platforms…

–and you know your way around the Unix command line and have ffmpeg installed on your machine…

–then you’re in luck. Here’s how to use these tools do what you need.

First, construct a text file that lists the files you mean to compress. My “mylist.txt” file contained the following lines:

file 'REEL 1V7 042914.mov'
file 'REEL 2V11 042914.mov'
file 'REEL 3V7 042914.mov'
file 'REEL 4V8 042914.mov'
file 'REEL 5V6 042914.mov'
file 'REEL 6V7 042914 fix.mov'
Put the mylist.txt file in the same directory as the source files, and also use the ‘cd’ command to switch your shell session into that directory so that long paths won’t be needed. Then, run this ffmpeg command (I wrote it as one line, but broke it up here with the ‘\’ character at the end of each line so that it’d be readable):
time ffmpeg -f concat -i mylist.txt -framerate 23.976 -s hd720 -c:v libx264 \
-preset slower -profile:v baseline -g 24 -deblock 0:0 -c:a libfaac -b:a 96k \
-movflags rtphint+faststart -vb 2500k -pix_fmt yuv420p -f mp4 -threads 80 outputfilename.mp4
45 minutes after I did this, I had a single mp4 file containing all the reels in order, as h264. I checked all the transition points between reels and there were no glitches.

This saved me from having to stay at work an extra 4 hours that night, waiting for Media Composer to export a full-length same-as-source movie and then converting it to h264.

I know what you’re thinking, “Why not just export to h264 from within Media Composer?”. Two reasons: Going to h264 directly from within Media Composer is slower than exporting same-as-source and feeding it into ffmpeg, and Media Composer’s h264 output doesn’t support as many options for hinting for streaming as were required for that upload.

Trailer for Spike Jonze film “Her” has been released

Sunday, August 18th, 2013
Poster-Xlarge
I spent the last year-and-a-half working on this film and, upon request, attempting to give spoiler-free explanations of its content to friends and relatives. Now I’m freed from that responsibility1 by the Warner Bros. Pictures marketing department. Thanks Brothers!

The trailer for the new Spike Jonze film, “Her”, has been released and can be found on both YouTube and Apple Trailers. I’d recommend watching the Apple Trailers version as even at the HD setting the YouTube version is a surprisingly poor quality encode, or maybe it was downgraded automatically for my poor connection at work while Apple’s site deigned to make me wait for the higher bitrate (Apple Trailers’ preference and mine for quality over convenience match in this instance).

I’m so happy that some of the film is now on display.

For fun, see if you can spot any VFX shots. It’s amazing how seamless and ubiquitous those are these days. I think there may be 17 or so in there, all in place to help tell a story rather than simply to wow the audience or call any attention to their presence.

  1. I’m referring to the need to explain the storyline, not the responsibility of working on the film –though I have moved on to another project []

1password apps are 50% off “for a limited time”

Thursday, May 30th, 2013
I love 1password, so I figure I’d pass this along. I don’t know how long it’ll last, but according to 9to5toys:
1Password is offering their first price drop since February.  For a limited time you can grab their iOS and Mac apps at 50% off, which brings the totals down to $8 and $25 respectively.
The deal is linked in their post here.

I use this application a ton every day. I’ve used it to create strong super-long garbled passwords for websites, to store those passwords, to enter them to login to websites, and to store and credit card numbers and addresses. It has browser extensions for all major browsers that allow its interface to be used from within a web browser to log in to websites with a single click. I keep its data files encrypted and backed-up to the cloud, so it’s secure and always accessible –and 1password’s data store includes an html document with fancy javascript that can decrypt and display the stored passwords and information in a web browser. So I don’t even have to install 1password on a computer to be able to access a password if I can find my way to the cloud backup.

Basically, it’s awesome, and it will make it possible for you to stop using the same password or variations for a bunch of different websites. I highly highly recommend this application. It’s the first thing I install when I sit down to a new computer.

confusion in os x Lion… where are my software updates?

Thursday, February 2nd, 2012
Software Update-2
No, my software is not up to date.
I want to try the multicam feature in the new FCP X 10.0.3 update, so I fire up Software Update. It checks Apple’s server and tells me that there are no available updates. How can this be? I’ve clearly got FCP X 10.0.2 installed, so the 10.0.3 update should appear. I run software update again, and again, with the same result. I ponder this a while, and then attain enlightenment.

Of course! FCP X is installed via the App Store application, and updates are likely also distributed via this route. –And so it is, I check for available updates in the App Store, and the 10.0.3 update is patiently waiting for me.

I’m happy to have found the update, but am a little surprised how unwieldy1 the process of finding updates has become. Apparently I need to keep it straight in my head that OS, Safari, iTunes, and some other updates are always delivered via the system’s “Software Update” function, and updates for apps installed via the App Store (should I ever manage to remember their provenance) are to be found only in the App Store app. And then there’s 3rd-party apps that have their own update mechanisms — if I’m lucky they use the Sparkle framework, which informs me of updates as soon as the apps are launched –nice.

If I’m confused about where to look for software updates, other users may be as well. An easy stopgap measure Apple could implement would be to add a line to Software Update when App store App updates are available such as “there are also 32 available updates waiting in the app store for you. Launch app store?”. Here’s an ugly mockup:

Software Update-3

I expect Apple will either do something like this, or will come up with some wonderfully elegant way to make this unnecessary, or will release a new Mac Pro2. Maybe all of the above. If they make this happen via a Software Update, I hope I can find it.

———-update 2/16/2012
It appears Apple addressed this issue with today’s release of OS X 10.8 “Mountain Lion”. All system and app software updates are now found and applied via the “app store” application.
  1. One might say “Windowsy” []
  2. unrelated, but please please please []

iCade’s available for 50% off

Thursday, December 29th, 2011
iCade
If you’ve got an iPad and enjoy classic arcade games or have ever thought of building a MAME cabinet, you might want to jump on this deal. Bed Bath and Beyond is clearing out their stock of iCades for half-off, at $49.99. I haven’t tried this, but it may be possible to even bring the price down further — Someone online says “Sign up for emails and get a printable 20% coupon bringing it to $39.99!”1

The iCade is a cabinet for the iPad that features an arcade-style joystick and eight buttons. It connects via bluetooth to the iPad as a wireless keyboard, and support for it is built into several gaming apps, including Atari’s Greatest Hits. Here’s one user’s review.

If you’ve got a jailbroken iPad2 you can use the iCade to control imame4all3 and other emulators. It’s great to play such games with a real joystick and buttons. I’ve been using an iCade for a while, and still plan to replace its joystick with a real sanwa, which apparently isn’t that difficult to do.

I don’t know how long this deal will last, but I think it’s a good one.

  1. Not my exclamation point, I should perhaps include a “sic” of disapproval. []
  2. jailbreaking the thing is very easy to do right now and is legal []
  3. just jailbreak and then install imame4all via cydia []

Little Snitch is half off for another day or so

Saturday, January 29th, 2011
Little Snitch
An example of the kind of warning dialog that Little Snitch pops up every time an application attempts to communicate with the outside world. The software allows you to set up rules and filters for the outgoing internet communications of your applications.
When applications running on my mac attempt to send information back to servers on the net, I like to be informed. I also like to have the opportunity to block such communication when I feel it’s of questionable utility or I don’t trust the motives of the software developer. If you live at or beyond my level of paranoia, you might be interested in buying yourself a copy of Little Snitch, it’s available on a site called MacUpdate for half price for another day or two, $14.99.

I get no kickbacks or referral fees from providing this link, I just think it’s a useful piece of software and it’s not normally available at this price.

A retro-gaming damask pattern, featuring Pac-Man, Pong, and Space Invaders

Wednesday, November 3rd, 2010
Damask Pattern
I threw together this little1 damask pattern, which I’m using as my desktop, iPhone, and iPad background. Figure I’d share in case anyone else would enjoy it.

Click the thumbnail at right to get hold of a png image at the exact resolution required for the iPhone 4G’s “retina” display. 960×540. Perhaps you can spot the game of Pong in it?

If you feel like using it on your phone, click through to the full-size image, then click and hold on the image to save it to your library, then navigate to and choose it in the wallpaper settings. Neato.

To make your own damask pattern, sit yourself down in front of a computer running Adobe Illustrator and check out this great tutorial over at tutsplus.com.

  1. Don’t you just love it when people are proud of an achievement, and they express their pride couched in diminutive adjectives and verbs that indicate the achievement took a minimal amount of effort. e.g.: “Oh, those little baubles I tossed off the other day? The full-scale model of the pyramids at Giza? It was nothing.” In this case, I think I’ve come up with a pattern I like (partly due to subject matter) but the pattern will probably please very few, (partly due to subject matter). I’m using the diminutive out of respect for those who hate retrogaming or hate damask patterns or just hate life in general. []

Where’s Chow? Aperture 3 facial recognition in action.

Thursday, June 3rd, 2010
Where's Chow
I thought this screenshot might amuse. Click the photo to see it at full-size. At the top of the image are photos that include Chow Yun Fat, cropped to just the automatically recognized face. Below a line are photos that Aperture 3 thinks may be the same person. To refine the computer’s idea of what his face looks like and tag more photos with the appropriate name I had to select all Chow Yun Fat photos from the bottom part of the screen and drag them to the top, above the line.

For more info on the facial recognition feature in Aperture 3, check here on Apple’s website.

I have about 16,000 photos in my archive, and though the process isn’t completely automatic, the facial recognition feature made it much more feasible to tag all the people in these photos. Even just the fact that the application can display an entire set of photos cropped and resized to only show faces alone would be a huge help, the fact that Aperture 3 makes decent guesses is a bonus.

Apple’s market cap now exceeds Microsoft’s

Wednesday, May 26th, 2010
If this trend holds, I may have to break out the Windows 7 party favors at my birthday party next week as per my previous blog post on the subject.

Live spreadsheet: