A workaround to keep Apple Compressor from transforming DnxHD quicktimes into dark-red freeze frames

Leave a reply

If an DNxHD quicktime file (reference or self contained) contains only a single video track, it will turn into a dark-red freeze-frame when run through Apple Compressor. If it doesn’t, then bully for you! You can stop reading.

But if you’re afflicted with this issue, there’s an easy workaround. If you add any video track, even a text track, to the quicktime file using QT7 Pro, it will pass through Compressor like a dream (with a slight gamma shift). I’ve made a quicktime text track of a tiny period character in the upper-left corner that I paste into Quicktime files for this purpose.

To use:

  1. Download this zip file (local link, dropbox link) and uncompress it.
  2. Open the quicktime “dot-text-track-invisible.mov” in QuickTime 7 Pro (assuming you can still run QuickTime 7 Pro — at some point a macOS update is going to break that ancient piece of very usable software, and I don’t think QuickTime X lets you add tracks to a QuickTime file).
  3. copy to clipboard
  4. open your dnxhd quicktime in QuickTime 7 Pro
  5. From the menubar, choose “Edit > Add to Selection & Scale”
  6. Save the dnxhd quicktime, which now has a new but invisible text track composited atop its video track

The resulting quicktime file should now be fully usable in Compressor. Having more than a single track in the file somehow knocks the compression job into a different code path or portion of the compositing engine that avoids the problem, or maybe it just flips a magic bit somewhere, I know nothing beyond speculation –but it works.

In case you’re reading this and thinking,”Why would I use Apple Compressor for this when I have Adobe Media Encoder which is not afflicted by this DnxHD issue?”, the answer to that could be any of the following:

  • A properly-configured Apple Compressor on a multiple-core machine (or network of machines) still beats Adobe Media Encoder on the same machine at making H264 in both speed and quality, at least last time I tested it.
  • You may already have created some nice presets for making files with certain watermarks in Apple Compressor and want to use them.
  • One click to compress and author/burn a simple DVD or Blu-ray disc or image. Nice.
  • You don’t have the Adobe Creative Suite on hand and are intimidated by ffmpeg

As an aside — ffmpeg can do watermarking, scaling, timecode burns, and a billion other things (as well as encode to Prores on Windows machines), and I highly recommend it. But it’s got a steep learning curve once you get into complicated tasks.

Me on Mastodon.

Sign of the times – firmware updates for cars and pacemakers

Leave a reply

Yesterday I ran into 3 signs that we live in the future.

  1. I had to report my wife’s car’s odometer reading to our car insurance provider to help estimate mileage for policy renewal. She was busy and I had no physical access to the car —but I realized during the call that I could launch her car’s dedicated iPhone app, log in to her car, and fetch the desired information. Crisis averted.
  2. A friend’s car offered up the option to upgrade its firmware while we were off running an errand.
  3. And this is kind of a cheat since it’s not a personal observation – but last night I came across this blog post by JWZ about a “Firmware Update to Address Cybersecurity Vulnerabilities Identified in Abbott’s (formerly St. Jude Medical’s) Implantable Cardiac Pacemakers”. That would surely make me nervous if my life depended on one of those devices, both the prospect of having a mechatronic body part hacked and also the prospect of surviving the firmware update.
The firmware update message that appeared on my friend's car's display last night as we headed out on an errand.
The firmware update message that appeared on my friend’s car’s display last night as we headed out on an errand.
Retrieving the odometer reading using a car iphone app.
Retrieving the odometer reading using a car iphone app.

Getting OS X Spotlight to index NTFS drives

Leave a reply

I had a need to select all quicktime files on an NTFS drive to open in a particular application (MetaCheater v2.0, if you must know), and the built-in file search feature in OS X called “Spotlight” seemed like the perfect solution as it was built into the file selection dialog that the application presented. But it turns out that OS X did not index the NTFS drive I’d connected, even though I’d mounted it using Paragon’s NTFS for Mac pane in System Preferences. But a google search turned up this tip from ‘Dracos’ on the macrumors forum.

Go to Applications – Utilities – Terminal

Type in:


/usr/bin/mdutil -i on /Volumes/NTFS_DRIVE
(Enter)
/usr/bin/mdimport /Volumes/NTFS_DRIVE
(Enter)

That worked!

Just filing this away in case I ever need it again.

Converting mkv, mpeg-2, anything from disc, to DnxHD or Prores using ffmpeg

3 Replies

Let’s suppose, for the sake of argument, that you’ve been asked by a director to cut his directing reel. He hands you a DVD and a Blu-ray and asks you to grab clips from them.

There are three problems here. The first is that you really should have been handed Prores or something editable from the get-go. But let’s assume you can’t get the media any other way (because often, you just can’t), and let’s assume that this is all considered fair-use (though current law makes that a bit muddy). We’re left with two technical problems:

  1. Rip: Getting the media off the disc and into a file, which may require circumventing copy protection mechanisms and encryption.
  2. Transcode: Converting the resulting file into something editable. Avid Media Composer won’t cut a .mkv file, and works with a limited range of video codecs. FCPX and Adobe Premiere Pro, despite their focus on working efficiently with many codecs, will still have trouble with some file formats. So this step is required for them too.

Rip

Regarding getting the clips off the disc, there are several applications out there for DVD, and only one that I know of for Blu-ray. From DVD, many people use a pretty fancy open-source tool that lets you pull selected ranges of video, and saves to mp4 (the software is designed to make files that play back on media devices, not for editorial purposes). If I’ve ever had to do this hypothetical task, I’ve avoided using this tool because I want the video to transfer from disc to file with no transcoding or lossy compression in between. So I’d have used this one tool which works for both DVD and Blu-ray. The result is a .mkv file which contains all the video and audio from the disc in its original codec.

So you’ve ripped the video from the disc and you’ve got an mkv file or files. You can play these in VLC Player to have a look at their contents to be sure you’ve got what you need. But none of the NLEs with which you edit can work with the files directly. Here’s how to convert them to dnxhd in order to be able to bring them into Media Composer. We’re going to use an awesome command-line tool called ffmpeg.

Install ffmpeg

At this point, I’m going to be giving instructions that relate to computers running Apple’s OS X. I’m sure it’s possible to do the next steps on a machine running Windows or Linux, but I’m not sure of the exact process. And we’re going to be using the Terminal, which you can find in /Applications/Utilities. You might as well open a Terminal window now.

There are a number of precompiled versions of ffmpeg floating around for download, but the only way to be sure you’re getting one with support for all the necessary formats built-in is to build it from source on your machine. To compile it, you’ll  need Apple’s free development tool, Xcode, or at least the Xcode command line utilities. Download and install that from Apple. Then you’ll need to install ffmpeg and its dependencies, a task that  is made pretty easy by a packaging system called ‘homebrew‘. If you’ve never set a password for your user account on OS X, you’ll need to do that first (the unix ‘passwd’ command, entered at the Terminal prompt, is a fast way to do this). It’s very easy to install homebrew, you just copy and paste one long command from homebrew’s web site into your Terminal window.

Once you’ve got Xcode  and homebrew installed, you’re ready to install ffmpeg. I do so by pasting the following into the terminal window:

brew install ffmpeg  --with-fdk-aac --with-ffplay --with-fontconfig --with-freetype --with-frei0r --with-libass --with-libbluray --with-libcaca --with-libquvi --with-libsoxr --with-libvidstab --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-openssl --with-opus --with-rtmpdump --with-schroedinger --with-speex --with-theora --with-tools --with-x265

That should all be one long line. Hit return, and brew will start downloading tool after tool, occasionally prompting you, and eventually installing ffmpeg. This may take a while, depending on the speed of your computer and the speed of your net connection. A half-hour wouldn’t be out of the ordinary. But when it’s done you’ll have an amazing multimedia swiss army knife at your disposal. I use ffmpeg for nearly all of my transcodes, even to watermark and transcode for upload to video distribution platforms.

Transcode your files with ffmpeg

Now that ffmpeg is installed, you can transcode your mkv file into dnxhd with one line of commands entered at the Terminal prompt. Here it is:

ffmpeg -i inputfilename.mkv -vf "scale=1920:1080" -vcodec dnxhd -r 23.976 -b:v 36M -c:a pcm_s16le -ar 48000 -ac 2 outputfilename.mov

For command-line newbies, I should point out that spaces are used as a special character to separate  commands and their arguments. So if there are any spaces in your filename, or in any directory along the way to your file, they need to be typed in a special way (“escaped” with a   backslash in front of it, like “\ “) so that the Unix shell knows to interpret them as a space and not as a separator. There are other special characters too. The safest way to type the filename is to let the Terminal do it for you — just type “ffmpeg -i ” (don’t type the quotes, but do type the space after the i) and then drag your input file right into the Terminal window. It’ll type the whole path to the file into the Terminal, with all spaces and special characters properly escaped. So for this command, I do:

ffmpeg -i drag_in_mkv_file -vf "scale=1920:1080" -vcodec dnxhd -r 23.976 -b:v 36M -c:a pcm_s16le -ar 48000 -ac 2 drag_in_mkv_file_and_then_edit_its_name_to_end_in_mov

Hit enter, and ffmpeg  do its thing. It’ll take the video from that mkv file, uprez it to 1080p if necessary, and output it as a DNxHD36 23.98 quicktime file with stereo 16-bit 48Khz audio. If that matches your Media Composer project’s settings, the video will fast-import. Unfortunately I haven’t yet nailed down the exact audio settings, so that audio will just do a normal and not fast-import (which means some conversion is happening during the import).

If the file you’re converting isn’t 23.98p, change that part of the command line to the appropriate frame rate. You’ll likely also need to change the bitrate. There’s a list of bitrates to use for different frame rates of dnxhd here (scroll down to the dnxhd section).

You can also convert your mkv or other files to prores using ffmpeg. For our blu-ray mkv file, a line like this should work:

ffmpeg -i inputfilename.mkv -vcodec  prores  -profile 2 -pix_fmt yuv422p10le -acodec pcm_s16le outputfilename.mov

That’ll make a Prores422 file with 16-bit audio. You can also do other types of Prores by changing the number after “-profile” to 0 for proxy, 1 for LT, or 3 for 422HQ.

In summary

Yes you can pull clips from DVD or Blu-ray and convert them into editable media without an intermediate transcode to mp4 — by losslessly ripping to mkv and then converting to Avid DNxHD or Prores using ffmpeg.

And if you get heavier into the command-line, you’ll find that it makes complicated scripting and batch processing  pretty easy to do. So having a good command-line conversion tool becomes very powerful.

Simple workaround for Media Composer’s ADM_COREAUDIO_UNDERRUN error

8 Replies

Every once in a while I’d sit down at my editing system, would open a clip in the source window, would hit play, and would be presented instantly with a little popup modal dialogue window displaying the error:

- Exception: ADM_COREAUDIO_UNDERRUN, Detected:2147483648

I’d hit the “OK” button, hit play again in the source window, and the error would happen again. And again. At this point Media Composer just wouldn’t let me play anything as it  kept  popping up audio errors each time I hit play.

It seemed likely to me that the audio output engine had gotten itself into an unstable state.

I like trying to guess what’s going on under the hood, as sometimes it helps point the way to a solution. Looking at the error, “ADM” probably means something like “Avid Device Manager”, “COREAUDIO” is an OS X framework for audio, and “UNDERRUN” might be the type of error coming from the Coreaudio framework, complaining that Media Composer isn’t feeding it enough audio sample data. A “buffer underrun”.

It’s as if the operating system is presenting the application with a spoon (AKA the “buffer”), the size of which is directly related to the sampling rate and bit depth of the audio output setting, and the application needs to exactly fill that spoon when presented in order for any sound to be output. Media Composer for whatever reason isn’t completely filling the spoon. Maybe it’s forgotten the size of the spoon, or needs its eyes checked.

Restarting the application would probably fix the problem, and barring that, restarting the computer might fix all. A bit of shallow net searching ((Admittedly not a very deep search. It’s possible this same solution is out there in one of the forums.)) comes up with those suggested solutions. I’ve found another way that requires no relaunches or reboots:

  1. Open /Applications/Utilities/Audio MIDI Setup.app
  2. If necessary, hit command-1 to open the “Audio Devices” window.
  3. Click to select the audio output device you’re using for Media Composer, in my case at this station it’s “Built-in Line Output”.
  4. In the pane at right, switch the output format from its current channel number and bit depth setting to one of the other options in that menu. Then change it back. In my case, I switch it from “2ch-24bit Integer” to “2ch-16bit Integer” and then back to “2ch-24bit Integer” (I want to leave it at 24bit, as that’s the bit depth of my project’s audio and I’d like to avoid having either Media Composer or Coreaudio have to waste effort  doing any conversions).
  5. That’s it, you’re done. Now go back to Media Composer, hit play, and bask in the glory of functional audio output.

2015-12-09 at 10.34If I were to hazard a guess as to what’s going on inside Media Composer that makes this work it is that step 4, changing the audio output device’s bit depth, pulls the rug out from under Media Composer a bit. OS X likely informs the application  that the bit depth for the Coreaudio output device has changed. Media Composer probably then closes its own internal output device object (managed by the aforementioned “ADM”) that had been connected to Coreaudio and then creates a new internal output device with the new settings and connects it to Coreaudio.

So it’s kind of a reset of the application’s audio output system, and now it’s got a nice new empty set of spoons coming in, and once again knows their exact size. —Apologies to all at Avid if this isn’t correct, I’m just guessing.

A bit of google searching comes up with a similar error message afflicting those running Media Composer on Windows. I can’t offer up a solution for those unfortunate souls ((Unfortunate since you’re afflicted with this error — not because you’re running that other OS that’s also not NetBSD.)) , but I’d bet  there’s a similar bit of OS-level audio setting rejiggering that will have a similar effect.

nice price on a Dual-extrusion 3d printer

Leave a reply
116141

If you have been thinking about getting into the time-consuming but educational 3D printing hobby, and have a budget that won’t be blown by $700, Monoprice is ready to help you to depart from your money and time.

They’ve been selling a metal-cased dual-extrusion 3d printer with heated print bed for a while now, for $1200. The printer appears to be a Monoprice-branded Flashforge Creator X, which is a well-regarded dual-extrusion printer, similar in design to a Makerbot.

Last week Monoprice dropped the price of their printer to $699, which makes it a very good deal. Here’s a link to the item. It looks like they’ve run out (?) but that there’s an ETA of 5/25/2015, so maybe more are coming in. Are they clearing stock for a Monoprice branded version of the newer (but mostly the same) Flashforge Creator Pro? If I had no 3d printer and were thinking of jumping in, I’d likely buy this discounted model. $699 is great for a dual-extrusion 3d printer with heated bed and metal case/structure.

The fact that this printer is not locked down and can be used with a variety of filaments and open-source software packages is a great thing. This makes it a more advanced option than something like the similarly-priced XYZ DaVinci 2.0 or other printers that require the use of expensive proprietary filaments and less customizable applications.

A few useful links:

An ffmpeg recipe to concatenate multiple quicktime files and convert to h264

6 Replies

If you ever find yourself with a set of same-as-source quicktime files that have been exported from Media Composer, one per reel of a film…

—and you need to output a single h264 file of the film to upload to Dax or Pix or another of the various secure media platforms…

—and you know your way around the Unix command line and have ffmpeg installed on your machine…

—then you’re in luck. Here’s how to use these tools do what you need.

First, construct a text file that lists the files you mean to compress. My “mylist.txt” file contained the following lines:

file 'REEL 1V7 042914.mov'
file 'REEL 2V11 042914.mov'
file 'REEL 3V7 042914.mov'
file 'REEL 4V8 042914.mov'
file 'REEL 5V6 042914.mov'
file 'REEL 6V7 042914 fix.mov'

Put the mylist.txt file in the same directory as the source files, and also use the ‘cd’ command to switch your shell session into that directory so that long paths won’t be needed. Then, run this ffmpeg command (I wrote it as one line, but broke it up here with the ‘\’ character at the end of each line so that it’d be readable):

time ffmpeg -f concat -i mylist.txt -framerate 23.976 -s hd720 -c:v libx264 \
-preset slower -profile:v baseline -g 24 -deblock 0:0 -c:a libfaac -b:a 96k \
-movflags rtphint+faststart -vb 2500k -pix_fmt yuv420p -f mp4 -threads 80 outputfilename.mp4

45 minutes after I did this, I had a single mp4 file containing all the reels in order, as h264. I checked all the transition points between reels and there were no glitches.

This saved me from having to stay at work an extra 4 hours that night, waiting for Media Composer to export a full-length same-as-source movie and then converting it to h264.

I know what you’re thinking, “Why not just export to h264 from within Media Composer?”. Two reasons: Going to h264 directly from within Media Composer is slower than exporting same-as-source and feeding it into ffmpeg, and Media Composer’s h264 output doesn’t support as many options for hinting for streaming as were required for that upload.

The Bluetooth Car Audio Streaming Conundrum

3 Replies

This is the story of the journey I’ve taken to my current car audio streaming solution, and the devices I’ve tried and abandoned along the way. Maybe some of this information will be of use to you, as it describes solutions and pitfalls related to the bluetooth streaming of audio from a phone or tablet to a car stereo. A followup post will describe how to do it better.

Sometimes one just needs to take a break from endless projects and geekery and just go for a drive and listen to some music. It’s the American way to hit the road and then hit play. That phone you carry around is just waiting to play tons of high quality music from its gargantuan internal storage, or stream music from the net as if you’re living in the post-cloud era.

We Want Wireless


But what’s that you say? Your car doesn’t have built-in bluetooth music streaming, and it’s annoying to have to connect a cable from your phone jack to your car stereo’s aux input every time you get in the car? Maybe you’re worried about wearing-out the phone’s headphone jack, or you’d rather be able to run a true line-level output to the car stereo’s aux input ((If you’re lucky-enough to have such an aux input.)) The simple answer is to just buy a Bluetooth car audio receiver such as the Soundfly BT, which plugs right into your car’s cigarette lighter outlet or whatever they call that port these days ((The “absurdly-deep friction-dependent smartphone and vaping device power outlet”?)). The Soundfly BT will power up with your car, your paired phone will automatically redirect its audio output to it via the Bluetooth A2DP service, and music will play from the Soundfly BT’s line out port to your car stereo’s aux input or it’ll broadcast to your car stereo’s radio receiver using the device’s built-in FM transmitter.

Apart from the fact that bluetooth audio transmission uses lossy compression ((Bluetooth A2DP, used for audio transmission, “supports optionally: MPEG-1, MPEG-2, MPEG-4, AAC, and ATRAC (shudder), and is extensible to support manufacturer-defined codecs, such as apt-X.” Wikipedia: Bluetooth profile, Advanced Audio Distribution Profile)) and theoretically could degrade the audio a bit, this thing sounds great and is a pretty ideal solution. Unless, that is, your car is like mine, and features an excellent built-in bluetooth speakerphone system that you’d like to continue to use.

Trouble in Paradise: The Bluetooth Speakerphone War

Because the Soundfly BT also features a (horrible) mic and can be used as a speakerphone, it has registered itself with your paired phone as both a Hands-Free and an A2DP audio streaming device. The. Horror.

Your new bluetooth music receiver and your car speakerphone are now at war.

Here’s how this situation regularly played out for me: I’d be in my car in bumper to bumper traffic on the 101 and I’d get a call about a job. I’d hit the answer button on my steering wheel, and my car’s bluetooth speakerphone would answer the call. I’d hear the caller’s perfectly-clear voice on the car’s dedicated hands-free speaker for about 2 seconds, and then 6 times out of 10 everything would go eerily silent in the middle of a word. My phone had just switched from using the car’s hands free system to the Soundfly BT’s hands free system. If my car stereo was set to play from its aux input, there would be a second of silence as the connection switched and then I’d hear (coming from the stereo) the concerned caller repeating my name adorned with question marks. I’d talk back and they’d barely able to make out what I’m saying because of the aforementioned awful mic.

Driving in Songjiang, a suburb of Shanghai.
Keep your eyes on the road! (Especially if you’re driving in Songjiang, a suburb of Shanghai)

That was the best-case-scenario with the Soundfly BT in my car. It didn’t go so well if my stereo was not already set to the aux input, because this meant that in order to hear the caller at all I’d have to manage to switch the stereo to that input while driving, a mean feat since I don’t like to take my eyes from the road while the car is in motion to avoid running over babies or dying. Usually the end result was that people would call, I’d pick up, and they’d hear silence until they hung up.

It’s too bad the Soundfly BT doesn’t feature an option to have it not present itself to your phone as a hands-free system, because its A2DP streaming works a treat. For music streaming it’s almost perfect. It even comes with a remote that could conceivably control your connected device but will instead sit in the glove compartment unused, forever. It’s not that the Soundfly is a bad device, quite the opposite. I highly recommend it if your car doesn’t have a built in hands-free system that you like. Unfortunately I don’t think any similar device includes a feature to disable its hands-free system —this problem affects the entire product category. Maybe they’re all built around the same chipset and the chip is just incapable of advertising itself to the phone as anything but hands-free + A2DP, or maybe no manufacturer strays from using the reference design and bothers to let the user choose to disable one of its features?

A turnkey solution that I skipped due to cost and possibly-dumb philosophical reasons

At this point a normal person might consider buying a bluetooth car audio device that lacks a hands-free system and is known to turn on with the car and stream just fine. The TuneLink Auto (iOS version
, Android version
((A2DP is a cross-platform standard, and I can’t think of a good reason why whatever custom interface is used to communicate with the thing’s iOS and Android apps (bluetooth serial?) couldn’t be used for both iOS and Android. Why are there different versions of this device?)) ) looks to have been designed to fit this niche. But I have simple needs, and the TuneLink Auto seems to be positioned as a premium device with features and price to match. I don’t need a device controlled via a custom iOS or Android app ((In fact, I see such a device as less valuable than a simpler device —it’s not unlikely that the device’s custom iOS app will not work years from now when our phones are running iOS v15 on 256-core CPUs. I’d rather the device have a non-proprietary control method not tied to custom software, such as a web interface, or some hardware buttons and an LCD. Why unnecessarily tie the future of a spend piece of hardware to software that could quickly become obsolete?)), all I want is a cheap bluetooth device that turns on when power is supplied to it and supports the bluetooth A2DP profile and can stream music from my phone without issue, auto-reconnect would be a plus.

An unfortunately imperfect solution

I came across the EDUP EP-B3502 Wireless Car Bluetooth Music Receiver and bit the (very low-cost) bullet, for science. This device is almost great. Positives:

  • It doesn’t only charge its internal battery via its micro-USB port, but can run while so connected. That was the deal breaker for the similar BCK-08, a device with which I experimented a few years ago.
  • No hands-free profile — no more war with the car speakerphone!
  • It sounds pretty good, good enough at least. I never did a comparative test, but it seemed like it was maybe a small step down in audio quality from the Soundfly BT. It only costs about $13, and all the bluetooth audio devices are using lossy compression, so my expectations were met.

I used this device for a few weeks and at first it seemed all right. One notable negative is that the device does not power itself on when the car starts and power hits its micro-usb port. I’d have to hold down its large button for a couple of seconds at the start of every journey. The fact that this device didn’t clobber my car’s built-in hands-free system made this nit feel minor and worth tolerating, and I figured at some point I’d add some electronics to complete that circuit automatically. A bigger problem is that the thing would occasionally start overlaying beeps on the music. This always felt random, though my guess is that the device would decide that its battery (which I was not using) was low, or maybe it was trying to communicate some other unnecessary message to me. Whatever the reason, it’d interrupt my music with beeps seemingly at random, and this became a maddening thing. ((Honestly guys, it’s the 21st century, we shouldn’t have to decipher beeps or sequences of flashing LEDs any more than we should have to program computers with punch-cards. Instead of beeping, say what the problem is, in a language that is spoken in your target market.))

Conclusion

Obviously this tale has to end with the abandonment of bluetooth streaming, a ton more research, the buying of parts, the setting up of development environments in a virtual machine, the compilation of custom firmwares, and a switch to the use of uncompressed ((well, non-lossy compression)) audio transmission over wifi to a hacked USB-powered router connected to a decent USB DAC. But that deserves its own post.