Most of you know that we're running an ethernet based SAN here at BCM. It's the Maxx Digital Final Share SAN which runs a combination of their drive arrays with Small Tree ethernet wizardry and some stout Atto Host Bus Adapters. We generally get around 100 - 120MB/s to the 7 workstations that are connected to the SAN. More than enough speed to cut using Apple's ProRes codec all day long.
At the moment we're cutting two feature documentaries (over 300 hours of material), 3 PBS series and a multitude of other projects all on the SAN simultaneously. It's been a very stout performer and when we need to do Uncompressed or 2k work, we have two local 8TB RAIDs directly connected to two workstations giving us 500 to 650MB/s. So for our needs, we've got everything set up to handle whatever comes in the door and need to allocate the uncompressed workstations as necessary.
Well at NAB 2010, Bob Zelin brought me over to the Maxx Digital Booth to show me 350MB/s coming off a single 8TB RAID connected via ethernet.
Now we're getting into Uncompressed HD territory. Albeit a single stream of Uncompressed HD, but that's perfectly fine for color grading or sound mixing and even editing. Not to mention serious multi-stream ProRes capabilities. Via Ethernet!
I knew this technology was going to get better as we moved along but I didn't expect a 250MB/s jump in just one year. In fact, the system could go faster but we need to wait for the drive manufacturers to catch up! Can you say Multi-stream Uncompressed HD and 2k via Ethernet on the horizon? All I can say is Wow.
Really looking forward to working with these new speeds once everything is ready for shipping. I'll update you all as details become available.
SEE UPDATES BELOW, I've updated this story twice since the original entry.
We took delivery of the brand spankin' new 27" i7 iMac last week and connected it immediately to our Final Share SAN in about 5 minutes for video editing. Quick tests showed the SAN was connected and working fine.
Then today I started really editing on it and I'm dropping frames every 10 to 30 seconds. Now it appears the ethernet controller that is in the new Mac cannot support the speeds necessary to edit video via the SAN. Our 2 year old iMacs can, but the brand spanking new, most powerful iMac cannot.
I'm at a loss as to how Apple can improve every aspect of this machine, including the absolutely stunning 27" LED backlit display, but then cut back on something as simple as an Ethernet Controller that should be designed to work with today's equipment running high speed internet instead of stepping backwards to the speed of older model PowerMac machines.
We're working with a few folks to see if this can be addressed at all with a driver update or if it's just all that the card can do. If this is all the speed we get, this machine is going back and we'll move on with another machine. My original plan was to install up to 4 of these 27" iMacs in our new facility, but that may have to change now if Apple is going to stay with these crippled controllers instead of giving us the speed we're paying for.
I'll update you guys as more information becomes available. To say I'm disappointed right now is an understatement. Wonder what I need to do to get on a beta test team because I seem to the person pushing all the systems further than any of their beta testers are. Everyone always tells me "you're the first person to find this......"
UPDATE #1 - 2/25/2010
I've been told the problem is limited to the 27" i5 and i7 Quad Core iMacs.
The Core Duo machines appear to support full Jumbo Frames across Ethernet. We're going to do definitive testing on both my i7 and a Core Duo machine this afternoon. Will update with more later.
UPDATE #2 - 2/25/2010
We took the iMac to one of the local Apple Stores where the technician at the Genius Bar confirmed that the Broadcom 5764 Ethernet Controller in the i7 iMac 27" does not support speeds over 1500. This same controller is in the i5 iMac 27" machine too.
From what he could gather, the Broadcom website is very vague as to whether the controller itself cannot support higher speeds or if it's just a driver issue. I've been told by outside sources that the documentation on the 5764 states it does not support Jumbo Frames so that leads me to believe 1500 is the max.
Now the Intel Core Duo 2 machines, which is pretty much the rest of Apple's lineup, all support Jumbo Frames. This includes the Core Duo 2 iMac 27" machines. I've returned the i7 and have ordered the 3.33Ghz 27" iMac.
I'm dumbfounded as to what happened with the design of the i7 and i5 machines. How was something as simple as an ethernet controller allowed to become a bottleneck on the flagship machine of a company? I don't know, but be aware that if you plan to use this in a professional environment and will require true high speed ethernet data transfer, as of right now, the i5 and i7 iMacs will not support that.
Again, to say I'm disappointed with Apple right now is an understatement.
In December of '08 we installed a Maxx Digital Final Share SAN system consisting of a 16TB array that is shared to 6 workstations via high speed ethernet connectivity. You can read a full article I wrote on the installations at the Creative Cow website.
The primary purpose for this installation was to allow a shared editing environment for three feature length documentaries. We have in the neighborhood of 450 hours of footage (and growing) for all three and our first doc, Foul Water, Fiery Serpent is using around 100 or so hours. All footage is digitized at Apple ProRes 720p/60 via Apple's Final Cut Pro so we're using a very low bandwidth format for the edit.
We started really cutting on the project in March of '09 and as I have reported both on the Creative Cow website and here on my blog, it has for the most part, been a thing of beauty. I'm cutting on a Mac Pro / AJA Kona 3 workstation while my edit assist is cutting on a 21" iMac. We broke the doc into 9 segments to make for easier project management and to allow each of us to work on different segments simultaneously.
To give you an idea of the size of this project, we have between 2,800 and 3,600 raw video clips, over 100 music cuts, animations, graphics and voice tracks. So we're in the neighborhood of 4,000 to 4,500 media files for this project. That's as big a project as I've worked on yet. As I said, the SAN has worked great during the editing process.
However, in the past few months I believe we have found the limits of ethernet based SANs; playback of a large project timeline. In November we finally had a full 98 minute timeline cut of the entire documentary. I could not play the entire timeline without dropping frames. And not just once, it would drop frames multiple times, every time during playback of the timeline. Plenty of speed on the RAID (about 600MB/s or more), plenty of speed on the network (about 100MB/s) but for whatever reason, I was dropping frames throughout the timeline.
Now I don't believe the length of the timeline is an issue. In my testing I played a 30 minute episode of "Good Eats" in a continuous loop for 3 hours on multiple systems simultaneously. So the system can easily sustain a long playback cycle across multiple systems, let alone on a single Mac Pro workstation. No there has to be something else other than pure speed.
If you've read my blog entries you also know that we've been dealing for months with an ethernet port issue introduced by Apple with the latest Mac Pros that caused the network disconnect from my Mac. So this problem of dropped frames was thought to be part of the same issue. Over the Christmas Holidays, with help from Small Tree Electronics, we dealt with that issue finally by moving the SAN to the Snow Leopard operating system because Apple finally created a fix to the disconnect issue in the latest updates.
But our dropped frames remain. The SAN is running as fast as ever, but we're still dropping frames during playback of our 90 minute timelines. From what I can gather, we are the only facility running this large of a project off this type of ethernet SAN. All the other facilities are doing 30 minute or shorter programming and a lot of :30 to :60 spots. In our own shop, we have multiple workstations doing projects of 20 minutes or less with no problems.
How we're dealing with these dropped frames right now is to export a self contained movie to a local 8TB array that is connected directly to my Mac Pro. This is how I screen the film for the client or folks who come in for reviews. It's the only way I can play the film without it stopping.
So for whatever reason, it appears to be the sheer project size and the amount of files that the project has to access during playback of the timeline that seems to be the issue. It really shouldn't matter, but it does appear in our real world application, the system simply does not support playback of an extended timeline from a project with this many files.
It's a real shame because the system is performing incredibly well overall, we have two series being cut on it and I've been able to work with a 2nd editor simultaneously on editing the documentary. But if you can't play back your main timeline without dropping frames on a large project, well then the system is not made for all editing applications as I originally thought and was led to believe when I made the purchase.
So if you're working on shorter projects, episodic television, 1 hour projects and need to share media across multiple workstations, this is still a killer deal. Fibre Channel is still the only alternative and you can't come close to what this system does for the money. We will continue to use this system moving forward on most of our projects.
But for the long form stuff like these documentaries, I'm going to invest in a few more local 8 and 16TB arrays. The primary workstation for each documentary will have its own dedicated local storage and anything that needs to move across to other workstations, we'll push to the SAN. It'll make things a little less efficient for the really REALLY big projects, but I'll have the best of both worlds. Low cost SAN for 90% of our projects. High speed local RAID for the documentaries.
So would I still recommend an Ethernet SAN for you? Absolutely, but go in understanding the limits and make sure it's right for your application before you buy. I won't say that this system was a $20,000 mistake, but I would have spent my money a little differently 12 months ago had I known this system would be limited by the documentary. And we all know technology improves almost daily so with any luck, future improvements will allow this type of SAN to even support the really REALLY big projects in the future.
After some suggestions from colleagues, I mixed down the audio tracks (we had a total of 24) in the timeline and attempted a full timeline playback. We got 38 minutes through the timeline before it dropped frames, but it did not drop frames again. It was a 1 hour 18 minute timeline. So that's progress. Not exactly efficient since it took a while for the computer to do the mix, but it's an improvement.
Today's just been one of those days where the Final Share SAN isn't working right, the AJA Kona 3 isn't working right and FCP isn't working right either. Just as we're about to finish our first feature documentary.
It's one of those days where you wonder what the hell you invested all of this money for and if you should just switch over to something else....
Well, as an update the previous posts about our Final Share SAN, yesterday we installed a new Ethernet card to bypass the new Mac Pro ports altogether, but unfortunately ran into another snag. Final Cut Pro won't even launch with the card in the machine. No clue why at the moment. My guess is some sort of a conflict with one of the other cards that are in there and for whatever reason, FCP is just throwing up when it tries to launch.
So more head scratching and sleuthing for the guys to see what's causing the conflict. Their suggestion was to re-install FCP, but I'm not doing that right now in the middle of all this production and especially since FCP runs perfectly fine with the card removed.
So for the moment, we're dealing with the SAN disconnects and moving on with the edit.