Geeky

Getting OS X Spotlight to index NTFS drives

Wednesday, June 7th, 2017

I had a need to select all quicktime files on an NTFS drive to open in a particular application (MetaCheater v2.0, if you must know), and the built-in file search feature in OS X called “Spotlight” seemed like the perfect solution as it was built into the file selection dialog that the application presented. But it turns out that OS X did not index the NTFS drive I’d connected, even though I’d mounted it using Paragon’s NTFS for Mac pane in System Preferences. But a google search turned up this tip from ‘Dracos’ on the macrumors forum.

Go to Applications – Utilities – Terminal

Type in:


/usr/bin/mdutil -i on /Volumes/NTFS_DRIVE
(Enter)
/usr/bin/mdimport /Volumes/NTFS_DRIVE
(Enter)

That worked!

Just filing this away in case I ever need it again.

Converting mkv, mpeg-2, anything from disc, to DnxHD or Prores using ffmpeg

Friday, March 25th, 2016

Let’s suppose, for the sake of argument, that you’ve been asked by a director to cut his directing reel. He hands you a DVD and a Blu-ray and asks you to grab clips from them.

There are three problems here. The first is that you really should have been handed Prores or something editable from the get-go. But let’s assume you can’t get the media any other way (because often, you just can’t), and let’s assume that this is all considered fair-use (though current law makes that a bit muddy). We’re left with two technical problems:

  1. Rip: Getting the media off the disc and into a file, which may require circumventing copy protection mechanisms and encryption.
  2. Transcode: Converting the resulting file into something editable. Avid Media Composer won’t cut a .mkv file, and works with a limited range of video codecs. FCPX and Adobe Premiere Pro, despite their focus on working efficiently with many codecs, will still have trouble with some file formats. So this step is required for them too.

Rip

Regarding getting the clips off the disc, there are several applications out there for DVD, and only one that I know of for Blu-ray. From DVD, many people use a pretty fancy open-source tool that lets you pull selected ranges of video, and saves to mp4 (the software is designed to make files that play back on media devices, not for editorial purposes). If I’ve ever had to do this hypothetical task, I’ve avoided using this tool because I want the video to transfer from disc to file with no transcoding or lossy compression in between. So I’d have used this one tool which works for both DVD and Blu-ray. The result is a .mkv file which contains all the video and audio from the disc in its original codec.

So you’ve ripped the video from the disc and you’ve got an mkv file or files. You can play these in VLC Player to have a look at their contents to be sure you’ve got what you need. But none of the NLEs with which you edit can work with the files directly. Here’s how to convert them to dnxhd in order to be able to bring them into Media Composer. We’re going to use an awesome command-line tool called ffmpeg.

Install ffmpeg

At this point, I’m going to be giving instructions that relate to computers running Apple’s OS X. I’m sure it’s possible to do the next steps on a machine running Windows or Linux, but I’m not sure of the exact process. And we’re going to be using the Terminal, which you can find in /Applications/Utilities. You might as well open a Terminal window now.

There are a number of precompiled versions of ffmpeg floating around for download, but the only way to be sure you’re getting one with support for all the necessary formats built-in is to build it from source on your machine. To compile it, you’ll need Apple’s free development tool, Xcode, or at least the Xcode command line utilities. Download and install that from Apple. Then you’ll need to install ffmpeg and its dependencies, a task that is made pretty easy by a packaging system called ‘homebrew‘. If you’ve never set a password for your user account on OS X, you’ll need to do that first (the unix ‘passwd’ command, entered at the Terminal prompt, is a fast way to do this). It’s very easy to install homebrew, you just copy and paste one long command from homebrew’s web site into your Terminal window.

Once you’ve got Xcode and homebrew installed, you’re ready to install ffmpeg. I do so by pasting the following into the terminal window:

brew install ffmpeg --with-fdk-aac --with-ffplay --with-fontconfig --with-freetype --with-frei0r --with-libass --with-libbluray --with-libcaca --with-libquvi --with-libsoxr --with-libvidstab --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-openssl --with-opus --with-rtmpdump --with-schroedinger --with-speex --with-theora --with-tools --with-x265

That should all be one long line. Hit return, and brew will start downloading tool after tool, occasionally prompting you, and eventually installing ffmpeg. This may take a while, depending on the speed of your computer and the speed of your net connection. A half-hour wouldn’t be out of the ordinary. But when it’s done you’ll have an amazing multimedia swiss army knife at your disposal. I use ffmpeg for nearly all of my transcodes, even to watermark and transcode for upload to video distribution platforms.

Transcode your files with ffmpeg

Now that ffmpeg is installed, you can transcode your mkv file into dnxhd with one line of commands entered at the Terminal prompt. Here it is:

ffmpeg -i inputfilename.mkv -vf "scale=1920:1080" -vcodec dnxhd -r 23.976 -b:v 36M -c:a pcm_s16le -ar 48000 -ac 2 outputfilename.mov

For command-line newbies, I should point out that spaces are used as a special character to separate commands and their arguments. So if there are any spaces in your filename, or in any directory along the way to your file, they need to be typed in a special way (“escaped” with a  backslash in front of it, like “\ “) so that the Unix shell knows to interpret them as a space and not as a separator. There are other special characters too. The safest way to type the filename is to let the Terminal do it for you — just type “ffmpeg -i ” (don’t type the quotes, but do type the space after the i) and then drag your input file right into the Terminal window. It’ll type the whole path to the file into the Terminal, with all spaces and special characters properly escaped. So for this command, I do:

ffmpeg -i drag_in_mkv_file -vf "scale=1920:1080" -vcodec dnxhd -r 23.976 -b:v 36M -c:a pcm_s16le -ar 48000 -ac 2 drag_in_mkv_file_and_then_edit_its_name_to_end_in_mov

Hit enter, and ffmpeg do its thing. It’ll take the video from that mkv file, uprez it to 1080p if necessary, and output it as a DNxHD36 23.98 quicktime file with stereo 16-bit 48Khz audio. If that matches your Media Composer project’s settings, the video will fast-import. Unfortunately I haven’t yet nailed down the exact audio settings, so that audio will just do a normal and not fast-import (which means some conversion is happening during the import).

If the file you’re converting isn’t 23.98p, change that part of the command line to the appropriate frame rate. You’ll likely also need to change the bitrate. There’s a list of bitrates to use for different frame rates of dnxhd here (scroll down to the dnxhd section).

You can also convert your mkv or other files to prores using ffmpeg. For our blu-ray mkv file, a line like this should work:

ffmpeg -i inputfilename.mkv -vcodec prores -profile 2 -pix_fmt yuv422p10le -acodec pcm_s16le outputfilename.mov

That’ll make a Prores422 file with 16-bit audio. You can also do other types of Prores by changing the number after “-profile” to 0 for proxy, 1 for LT, or 3 for 422HQ.

In summary

Yes you can pull clips from DVD or Blu-ray and convert them into editable media without an intermediate transcode to mp4 – by losslessly ripping to mkv and then converting to Avid DNxHD or Prores using ffmpeg.

And if you get heavier into the command-line, you’ll find that it makes complicated scripting and batch processing pretty easy to do. So having a good command-line conversion tool becomes very powerful.

Simple workaround for Media Composer’s ADM_COREAUDIO_UNDERRUN error

Wednesday, December 9th, 2015

Every once in a while I’d sit down at my editing system, would open a clip in the source window, would hit play, and would be presented instantly with a little popup modal dialogue window displaying the error:

• Exception: ADM_COREAUDIO_UNDERRUN, Detected:2147483648

I’d hit the “OK” button, hit play again in the source window, and the error would happen again. And again. At this point Media Composer just wouldn’t let me play anything as it kept popping up audio errors each time I hit play.

It seemed likely to me that the audio output engine had gotten itself into an unstable state.

I like trying to guess what’s going on under the hood, as sometimes it helps point the way to a solution. Looking at the error, “ADM” probably means something like “Avid Device Manager”, “COREAUDIO” is an OS X framework for audio, and “UNDERRUN” might be the type of error coming from the Coreaudio framework, complaining that Media Composer isn’t feeding it enough audio sample data. A “buffer underrun”.

It’s as if the operating system is presenting the application with a spoon (AKA the “buffer”), the size of which is directly related to the sampling rate and bit depth of the audio output setting, and the application needs to exactly fill that spoon when presented in order for any sound to be output. Media Composer for whatever reason isn’t completely filling the spoon. Maybe it’s forgotten the size of the spoon, or needs its eyes checked.

Restarting the application would probably fix the problem, and barring that, restarting the computer might fix all. A bit of shallow net searching1 comes up with those suggested solutions. I’ve found another way that requires no relaunches or reboots:

  1. Open /Applications/Utilities/Audio MIDI Setup.app
  2. If necessary, hit command-1 to open the “Audio Devices” window.
  3. Click to select the audio output device you’re using for Media Composer, in my case at this station it’s “Built-in Line Output”.
  4. In the pane at right, switch the output format from its current channel number and bit depth setting to one of the other options in that menu. Then change it back. In my case, I switch it from “2ch-24bit Integer” to “2ch-16bit Integer” and then back to “2ch-24bit Integer” (I want to leave it at 24bit, as that’s the bit depth of my project’s audio and I’d like to avoid having either Media Composer or Coreaudio have to waste effort doing any conversions).
  5. That’s it, you’re done. Now go back to Media Composer, hit play, and bask in the glory of functional audio output.

2015-12-09 at 10.34If I were to hazard a guess as to what’s going on inside Media Composer that makes this work it is that step 4, changing the audio output device’s bit depth, pulls the rug out from under Media Composer a bit. OS X likely informs the application that the bit depth for the Coreaudio output device has changed. Media Composer probably then closes its own internal output device object (managed by the aforementioned “ADM”) that had been connected to Coreaudio and then creates a new internal output device with the new settings and connects it to Coreaudio.

So it’s kind of a reset of the application’s audio output system, and now it’s got a nice new empty set of spoons coming in, and once again knows their exact size. –Apologies to all at Avid if this isn’t correct, I’m just guessing.

A bit of google searching comes up with a similar error message afflicting those running Media Composer on Windows. I can’t offer up a solution for those unfortunate souls2 , but I’d bet there’s a similar bit of OS-level audio setting rejiggering that will have a similar effect.

  1. Admittedly not a very deep search. It’s possible this same solution is out there in one of the forums. []
  2. Unfortunate since you’re afflicted with this error — not because you’re running that other OS that’s also not NetBSD. []

Fantastic new Star Wars trailer

Thursday, May 21st, 2015

Kudos to the VFX artist behind this new Star Wars Trailer.

nice price on a Dual-extrusion 3d printer

Sunday, March 22nd, 2015
116141

If you have been thinking about getting into the time-consuming but educational 3D printing hobby, and have a budget that won’t be blown by $700, Monoprice is ready to help you to depart from your money and time.

They’ve been selling a metal-cased dual-extrusion 3d printer with heated print bed for a while now, for $1200. The printer appears to be a Monoprice-branded Flashforge Creator X, which is a well-regarded dual-extrusion printer, similar in design to a Makerbot.

Last week Monoprice dropped the price of their printer to $699, which makes it a very good deal. Here’s a link to the item. It looks like they’ve run out (?) but that there’s an ETA of 5/25/2015, so maybe more are coming in. Are they clearing stock for a Monoprice branded version of the newer (but mostly the same) Flashforge Creator Pro? If I had no 3d printer and were thinking of jumping in, I’d likely buy this discounted model. $699 is great for a dual-extrusion 3d printer with heated bed and metal case/structure.

The fact that this printer is not locked down and can be used with a variety of filaments and open-source software packages is a great thing. This makes it a more advanced option than something like the similarly-priced XYZ DaVinci 2.0 or other printers that require the use of expensive proprietary filaments and less customizable applications.

A few useful links:

An ffmpeg recipe to concatenate multiple quicktime files and convert to h264

Tuesday, April 29th, 2014

If you ever find yourself with a set of same-as-source quicktime files that have been exported from Media Composer, one per reel of a film…

–and you need to output a single h264 file of the film to upload to Dax or Pix or another of the various secure media platforms…

–and you know your way around the Unix command line and have ffmpeg installed on your machine…

–then you’re in luck. Here’s how to use these tools do what you need.

First, construct a text file that lists the files you mean to compress. My “mylist.txt” file contained the following lines:

file 'REEL 1V7 042914.mov'
file 'REEL 2V11 042914.mov'
file 'REEL 3V7 042914.mov'
file 'REEL 4V8 042914.mov'
file 'REEL 5V6 042914.mov'
file 'REEL 6V7 042914 fix.mov'

Put the mylist.txt file in the same directory as the source files, and also use the ‘cd’ command to switch your shell session into that directory so that long paths won’t be needed. Then, run this ffmpeg command (I wrote it as one line, but broke it up here with the ‘\’ character at the end of each line so that it’d be readable):

time ffmpeg -f concat -i mylist.txt -framerate 23.976 -s hd720 -c:v libx264 \
-preset slower -profile:v baseline -g 24 -deblock 0:0 -c:a libfaac -b:a 96k \
-movflags rtphint+faststart -vb 2500k -pix_fmt yuv420p -f mp4 -threads 80 outputfilename.mp4

45 minutes after I did this, I had a single mp4 file containing all the reels in order, as h264. I checked all the transition points between reels and there were no glitches.

This saved me from having to stay at work an extra 4 hours that night, waiting for Media Composer to export a full-length same-as-source movie and then converting it to h264.

I know what you’re thinking, “Why not just export to h264 from within Media Composer?”. Two reasons: Going to h264 directly from within Media Composer is slower than exporting same-as-source and feeding it into ffmpeg, and Media Composer’s h264 output doesn’t support as many options for hinting for streaming as were required for that upload.

The Bluetooth Car Audio Streaming Conundrum

Tuesday, March 25th, 2014

This is the story of the journey I’ve taken to my current car audio streaming solution, and the devices I’ve tried and abandoned along the way. Maybe some of this information will be of use to you, as it describes solutions and pitfalls related to the bluetooth streaming of audio from a phone or tablet to a car stereo. A followup post will describe how to do it better.

Sometimes one just needs to take a break from endless projects and geekery and just go for a drive and listen to some music. It’s the American way to hit the road and then hit play. That phone you carry around is just waiting to play tons of high quality music from its gargantuan internal storage, or stream music from the net as if you’re living in the post-cloud era.

We Want Wireless


But what’s that you say? Your car doesn’t have built-in bluetooth music streaming, and it’s annoying to have to connect a cable from your phone jack to your car stereo’s aux input every time you get in the car? Maybe you’re worried about wearing-out the phone’s headphone jack, or you’d rather be able to run a true line-level output to the car stereo’s aux input1 The simple answer is to just buy a Bluetooth car audio receiver such as the Soundfly BT, which plugs right into your car’s cigarette lighter outlet or whatever they call that port these days2. The Soundfly BT will power up with your car, your paired phone will automatically redirect its audio output to it via the Bluetooth A2DP service, and music will play from the Soundfly BT’s line out port to your car stereo’s aux input or it’ll broadcast to your car stereo’s radio receiver using the device’s built-in FM transmitter.

Apart from the fact that bluetooth audio transmission uses lossy compression3 and theoretically could degrade the audio a bit, this thing sounds great and is a pretty ideal solution. Unless, that is, your car is like mine, and features an excellent built-in bluetooth speakerphone system that you’d like to continue to use.

Trouble in Paradise: The Bluetooth Speakerphone War

Because the Soundfly BT also features a (horrible) mic and can be used as a speakerphone, it has registered itself with your paired phone as both a Hands-Free and an A2DP audio streaming device. The. Horror.

Your new bluetooth music receiver and your car speakerphone are now at war.

Here’s how this situation regularly played out for me: I’d be in my car in bumper to bumper traffic on the 101 and I’d get a call about a job. I’d hit the answer button on my steering wheel, and my car’s bluetooth speakerphone would answer the call. I’d hear the caller’s perfectly-clear voice on the car’s dedicated hands-free speaker for about 2 seconds, and then 6 times out of 10 everything would go eerily silent in the middle of a word. My phone had just switched from using the car’s hands free system to the Soundfly BT’s hands free system. If my car stereo was set to play from its aux input, there would be a second of silence as the connection switched and then I’d hear (coming from the stereo) the concerned caller repeating my name adorned with question marks. I’d talk back and they’d barely able to make out what I’m saying because of the aforementioned awful mic.

Driving in Songjiang, a suburb of Shanghai.
Keep your eyes on the road! (Especially if you’re driving in Songjiang, a suburb of Shanghai)

That was the best-case-scenario with the Soundfly BT in my car. It didn’t go so well if my stereo was not already set to the aux input, because this meant that in order to hear the caller at all I’d have to manage to switch the stereo to that input while driving, a mean feat since I don’t like to take my eyes from the road while the car is in motion to avoid running over babies or dying. Usually the end result was that people would call, I’d pick up, and they’d hear silence until they hung up.

It’s too bad the Soundfly BT doesn’t feature an option to have it not present itself to your phone as a hands-free system, because its A2DP streaming works a treat. For music streaming it’s almost perfect. It even comes with a remote that could conceivably control your connected device but will instead sit in the glove compartment unused, forever. It’s not that the Soundfly is a bad device, quite the opposite. I highly recommend it if your car doesn’t have a built in hands-free system that you like. Unfortunately I don’t think any similar device includes a feature to disable its hands-free system –this problem affects the entire product category. Maybe they’re all built around the same chipset and the chip is just incapable of advertising itself to the phone as anything but hands-free + A2DP, or maybe no manufacturer strays from using the reference design and bothers to let the user choose to disable one of its features?

A turnkey solution that I skipped due to cost and possibly-dumb philosophical reasons

At this point a normal person might consider buying a bluetooth car audio device that lacks a hands-free system and is known to turn on with the car and stream just fine. The TuneLink Auto (iOS version
, Android version
4 ) looks to have been designed to fit this niche. But I have simple needs, and the TuneLink Auto seems to be positioned as a premium device with features and price to match. I don’t need a device controlled via a custom iOS or Android app5, all I want is a cheap bluetooth device that turns on when power is supplied to it and supports the bluetooth A2DP profile and can stream music from my phone without issue, auto-reconnect would be a plus.

An unfortunately imperfect solution

I came across the EDUP EP-B3502 Wireless Car Bluetooth Music Receiver and bit the (very low-cost) bullet, for science. This device is almost great. Positives:

  • It doesn’t only charge its internal battery via its micro-USB port, but can run while so connected. That was the deal breaker for the similar BCK-08, a device with which I experimented a few years ago.
  • No hands-free profile – no more war with the car speakerphone!
  • It sounds pretty good, good enough at least. I never did a comparative test, but it seemed like it was maybe a small step down in audio quality from the Soundfly BT. It only costs about $13, and all the bluetooth audio devices are using lossy compression, so my expectations were met.

I used this device for a few weeks and at first it seemed all right. One notable negative is that the device does not power itself on when the car starts and power hits its micro-usb port. I’d have to hold down its large button for a couple of seconds at the start of every journey. The fact that this device didn’t clobber my car’s built-in hands-free system made this nit feel minor and worth tolerating, and I figured at some point I’d add some electronics to complete that circuit automatically. A bigger problem is that the thing would occasionally start overlaying beeps on the music. This always felt random, though my guess is that the device would decide that its battery (which I was not using) was low, or maybe it was trying to communicate some other unnecessary message to me. Whatever the reason, it’d interrupt my music with beeps seemingly at random, and this became a maddening thing.6

Conclusion

Obviously this tale has to end with the abandonment of bluetooth streaming, a ton more research, the buying of parts, the setting up of development environments in a virtual machine, the compilation of custom firmwares, and a switch to the use of uncompressed7 audio transmission over wifi to a hacked USB-powered router connected to a decent USB DAC. But that deserves its own post.

  1. If you’re lucky-enough to have such an aux input. []
  2. The “absurdly-deep friction-dependent smartphone and vaping device power outlet”? []
  3. Bluetooth A2DP, used for audio transmission, “supports optionally: MPEG-1, MPEG-2, MPEG-4, AAC, and ATRAC (shudder), and is extensible to support manufacturer-defined codecs, such as apt-X.” Wikipedia: Bluetooth profile, Advanced Audio Distribution Profile []
  4. A2DP is a cross-platform standard, and I can’t think of a good reason why whatever custom interface is used to communicate with the thing’s iOS and Android apps (bluetooth serial?) couldn’t be used for both iOS and Android. Why are there different versions of this device? []
  5. In fact, I see such a device as less valuable than a simpler device –it’s not unlikely that the device’s custom iOS app will not work years from now when our phones are running iOS v15 on 256-core CPUs. I’d rather the device have a non-proprietary control method not tied to custom software, such as a web interface, or some hardware buttons and an LCD. Why unnecessarily tie the future of a spend piece of hardware to software that could quickly become obsolete? []
  6. Honestly guys, it’s the 21st century, we shouldn’t have to decipher beeps or sequences of flashing LEDs any more than we should have to program computers with punch-cards. Instead of beeping, say what the problem is, in a language that is spoken in your target market. []
  7. well, non-lossy compression []

What’s likely coming in the next GoPro (Hero 4) – 4k 30fps, 2k 120fps!

Tuesday, March 18th, 2014

If the specs for the next version of the system-on-a-chip system used in the GoPro Hero 3 are any indication, the next GoPro is going to have some ridiculous specs. 4k-resolution at 30 frames-per-second, 120fps 2k, 240fps 720p, better dynamic range, quality, etc etc etc. The Ambarella A9 sounds like a monster.

I’m looking forward to the cameras the next couple of years will bring. Now if I could just focus on writing something worth shooting…

How to rank the latests slate of Amazon pilots?

Friday, February 28th, 2014
2/27 stars, votes, and imdb rating of Amazon's 2014 pilots
2/27 stars, votes, and imdb rating of Amazon’s 2014 pilots

It’s nearing the end of the first month of Amazon’s 2014 pilot season. Amazon, being a company of the digital age, posts all the pilot episodes of their new slate of shows online (here’s a link to view all of them for free) and then tracks how well they’re received by the public. Presumably, the shows that get the most interest and the highest ratings are the ones that Amazon will decide to run for a full season, though I wonder how much or whether the expected production budget of each show figures into the calculation. I worked as assistant editor on The Rebels, so I’m likely a bit more curious than the average person as to how the various pilots are doing. I’ve checked in on the pilots roughly once a week to record their metrics.

There are a few ratings for each pilot that are publicly available. These include the star rating (out of five, rounded to the nearest half-star), number of votes, and imdb rating, listed at the top of each pilot’s page. Further down the page, in the ‘Customer Reviews’ section, is a more precise star rating rounded to the nearest decimal point (i.e. 4.7 rather than 4&1/2).

At right is a table of the current standings of Amazon’s non-kid-oriented pilots as of yesterday, and below are a few graphs showing the trend in those metrics over the past few weeks:

Amazon pilots 2014 trend graphs
Amazon pilots 2014 trend graphs

It looks to me as though all the pilots have been well received by the very non-random-sampling of Amazon members who choose to both watch and review them, with the minimum star rating hitting 4 out of 5. Only Amazon knows how many viewers each episode has accrued. Bosch appears to be the runaway hit, though I can’t say for certain that this means it’ll be picked up as there are a lot of factors to which I’m not privy –such as the budget for the show and its viewer breakdown. It interests me that The After and Transparent started strong but have seen an fall in their star (and for Transparent, also its imdb) ratings over time. Until I get around to watching all the pilots I keep in mind that it’s possible that this just means the people who are most likely to enjoy those shows are the ones who were first in line to watch them, and the downward trend has nothing to do with how much I will like them. The Rebels seems to have had a pretty consistent reception, with a center of the pack star rating and a steadily rising IMDB rating. The number of votes for The Rebels and Mozart In the Jungle have been almost identical at every moment I’ve sampled, so that’s why they obscure each other in the “number of votes” chart below.

Strength of Coffee vs. strength of wasabi/soy mixture

Thursday, October 17th, 2013

I realized that the difference in color between Rachel’s coffee and mine is generally a mirror image of the difference in color between the wasabi/soy mixtures we each stir up when we order sushi. It felt deserving of a quick infographic:

Zach-And-Rachel-Coffee-And-Wasabi-Preferences