BLOGS: My COW Blog Adobe Blog Editing Technology After Effects Final Cut Entertainment

Ethernet SAN has limits - UPDATED

COW Blogs : walter biscardi's Blog : Ethernet SAN has limits - UPDATED
In December of '08 we installed a Maxx Digital Final Share SAN system consisting of a 16TB array that is shared to 6 workstations via high speed ethernet connectivity. You can read a full article I wrote on the installations at the Creative Cow website.

The primary purpose for this installation was to allow a shared editing environment for three feature length documentaries. We have in the neighborhood of 450 hours of footage (and growing) for all three and our first doc, Foul Water, Fiery Serpent is using around 100 or so hours. All footage is digitized at Apple ProRes 720p/60 via Apple's Final Cut Pro so we're using a very low bandwidth format for the edit.

We started really cutting on the project in March of '09 and as I have reported both on the Creative Cow website and here on my blog, it has for the most part, been a thing of beauty. I'm cutting on a Mac Pro / AJA Kona 3 workstation while my edit assist is cutting on a 21" iMac. We broke the doc into 9 segments to make for easier project management and to allow each of us to work on different segments simultaneously.

To give you an idea of the size of this project, we have between 2,800 and 3,600 raw video clips, over 100 music cuts, animations, graphics and voice tracks. So we're in the neighborhood of 4,000 to 4,500 media files for this project. That's as big a project as I've worked on yet. As I said, the SAN has worked great during the editing process.

However, in the past few months I believe we have found the limits of ethernet based SANs; playback of a large project timeline. In November we finally had a full 98 minute timeline cut of the entire documentary. I could not play the entire timeline without dropping frames. And not just once, it would drop frames multiple times, every time during playback of the timeline. Plenty of speed on the RAID (about 600MB/s or more), plenty of speed on the network (about 100MB/s) but for whatever reason, I was dropping frames throughout the timeline.

Now I don't believe the length of the timeline is an issue. In my testing I played a 30 minute episode of "Good Eats" in a continuous loop for 3 hours on multiple systems simultaneously. So the system can easily sustain a long playback cycle across multiple systems, let alone on a single Mac Pro workstation. No there has to be something else other than pure speed.

If you've read my blog entries you also know that we've been dealing for months with an ethernet port issue introduced by Apple with the latest Mac Pros that caused the network disconnect from my Mac. So this problem of dropped frames was thought to be part of the same issue. Over the Christmas Holidays, with help from Small Tree Electronics, we dealt with that issue finally by moving the SAN to the Snow Leopard operating system because Apple finally created a fix to the disconnect issue in the latest updates.

But our dropped frames remain. The SAN is running as fast as ever, but we're still dropping frames during playback of our 90 minute timelines. From what I can gather, we are the only facility running this large of a project off this type of ethernet SAN. All the other facilities are doing 30 minute or shorter programming and a lot of :30 to :60 spots. In our own shop, we have multiple workstations doing projects of 20 minutes or less with no problems.

How we're dealing with these dropped frames right now is to export a self contained movie to a local 8TB array that is connected directly to my Mac Pro. This is how I screen the film for the client or folks who come in for reviews. It's the only way I can play the film without it stopping.

So for whatever reason, it appears to be the sheer project size and the amount of files that the project has to access during playback of the timeline that seems to be the issue. It really shouldn't matter, but it does appear in our real world application, the system simply does not support playback of an extended timeline from a project with this many files.

It's a real shame because the system is performing incredibly well overall, we have two series being cut on it and I've been able to work with a 2nd editor simultaneously on editing the documentary. But if you can't play back your main timeline without dropping frames on a large project, well then the system is not made for all editing applications as I originally thought and was led to believe when I made the purchase.

So if you're working on shorter projects, episodic television, 1 hour projects and need to share media across multiple workstations, this is still a killer deal. Fibre Channel is still the only alternative and you can't come close to what this system does for the money. We will continue to use this system moving forward on most of our projects.

But for the long form stuff like these documentaries, I'm going to invest in a few more local 8 and 16TB arrays. The primary workstation for each documentary will have its own dedicated local storage and anything that needs to move across to other workstations, we'll push to the SAN. It'll make things a little less efficient for the really REALLY big projects, but I'll have the best of both worlds. Low cost SAN for 90% of our projects. High speed local RAID for the documentaries.

So would I still recommend an Ethernet SAN for you? Absolutely, but go in understanding the limits and make sure it's right for your application before you buy. I won't say that this system was a $20,000 mistake, but I would have spent my money a little differently 12 months ago had I known this system would be limited by the documentary. And we all know technology improves almost daily so with any luck, future improvements will allow this type of SAN to even support the really REALLY big projects in the future.


UPDATE 1-21-2010

After some suggestions from colleagues, I mixed down the audio tracks (we had a total of 24) in the timeline and attempted a full timeline playback. We got 38 minutes through the timeline before it dropped frames, but it did not drop frames again. It was a 1 hour 18 minute timeline. So that's progress. Not exactly efficient since it took a while for the computer to do the mix, but it's an improvement.



Comments

Re: Ethernet SAN has limits - UPDATED
by Ryan Stiles
Any updates 3 years later?

Also just to clarify the dropped frames are only in the over an hour timeline? I'm looking at setting this kind of system up for a three day festival that will acumulate over 16TB of assets by the end. The timelines will only be 5minutes max (for streaming broadcast) but there will be a huge amount of footage in the NLE library. Should be good, right?
Re: Ethernet SAN has limits - UPDATED
by Patricia Tamowski
You said you found a solution with local 8TB and 16TB arrays. Can you suggest a low cost reliable one for our full length documentary? We prefer local and preferably eSATA for editing speed. We edit in FCP from MacBook Pro. Thank you.
Audio really might be the culprit
by Don Walker
Walter

I agree with Oliver, Try your timeline with either no audio or the audio mixdowned. I have problems from time to time with my Octocore dropping frames on very simple large timelines because of music tracks coming off an internal Sata drive (not system). and there does not seem to be any rhyme or reason for it.
my situation continued
by Tom Meegan


5. I have two storage devices on the network
my situation
by Tom Meegan
Walter,

My ethernet SAN system just completed two two hours shows for SPEED. The sequences were over 100 minutes. 90 minutes of content + filler for re-cues during breaks. I did not have issues with output to tape.

The differences between our set-up and yours are significant however.

1. We posted in ProRes LT, standard definition NTSC.

2. The connection between the server and the switch is 10GbE.

3. The total footage load and complexity of the sequences was likely MUCH simpler than yours, as the shows were largely live to tape.

4. The server is an xServe.

5. I have two storage devices on the network
my situation
by Tom Meegan
Walter,

My ethernet SAN system just completed two two hours shows for SPEED. The sequences were over 100 minutes. 90 minutes of content + filler for re-cues during breaks. I did not have issues with output to tape.

The differences between our set-up and yours are significant however.

1. We posted in ProRes LT, standard definition NTSC.

2. The connection between the server and the switch is 10GbE.

3. The total footage load and complexity of the sequences was likely MUCH simpler than yours, as the shows were largely live to tape.

4. The server is an xServe.

5. I have two storage devices on the network
SAN MP
by walter biscardi
We're only running Snow Leopard on the SAN at this time. All other machines are still running solid with Leopard. As I said in my original post, Snow Leopard actually fixes some of the underlying issues that created other problems in the past.

I'm starting to wonder if it's all the audio tracks that might be of an issue.
SANmp
by Victor Perez
We are also having problems with our SAN although we are connected thru Fiber. Our tech and SANmp believe the problem lies with Snow Leopard compatibility. We plan to revert one of our new MacPro's back to Leopard to see if the problem continues with that machine. The problem with the SAN crashing has been increasing to almost once a day now. We are running the new 2.26 MacPros with 16 GB of memory. Our previous 3.0 MacPros with 8GB ram were real workhorses.
interesting article
by adam taylor
just a thought , Walter, based on Olivers comment - I remember years ago on an old Avid, that i used to have to do an audio mixdown before playing back a longer cut....have you tried doing anything like that to your doc?

good luck

Adam
Local system
by walter biscardi
Oliver,

That's precisely what we're doing. Pushing a self contained QT to our local SAS/SATA array (8TB, 550MB/s) and playing that for the client.

We're hoping that eventually the ethernet SAN will be able to keep up with even the large projects but at the moment, this is the main hiccup with it.

Walter
Ethernet SAN has limits
by Oliver Peters
Since playout is the issue, my suggestion is to export a self-contained QT to a locally-connected eSATA array. Playout from the self-contained file.

I have seen similar issues with large shows on some FCP systems, even on local arrays. It seems that for whatever reason, the next files in the timeline aren't called up fast enough to "get to speed" and play correctly (i.e. "in sync").

Frequently audio files are the issue. Audio files are actually more of a problem than video files, because even though they are small, they hit the system more frequently with data than larger HQ video files.

- Oliver
More media, more problems
by David Braswell
Your headaches are bigger than mine! But I'm thankful people like you are out there to push the envelope and force developer innovation. I'm neither a Mac nor FCP user, but your posts about your system have been very informative.


Professional Video Editor, Producer, Creative Director, Director since 1990.

Credits include multiple Emmys, Tellys, Aurora and CableAce Awards.

Creative Director for Georgia-Pacific and GP Studios, Atlanta. Former Owner / Operator of Biscardi Creative Media. The show you knew us best for was "Good Eats" on the Food Network. I developed the HD Post workflow and we also created all the animations for the series.

Favorite pastime is cooking with pizza on the grill one of my specialties. Each Christmas Eve we serve the Feast of the Seven Fishes, a traditional Italian seafood meal with approx. 30 items on the menu.

If I wasn't in video production I would either own a restaurant or a movie theater.

 




Archives:

July 2018 (1)
July 2017 (1)
November 2016 (1)
May 2016 (1)
April 2016 (3)
April 2015 (1)
March 2015 (2)
February 2015 (1)
January 2015 (1)
July 2014 (2)
December 2013 (5)
November 2013 (1)
October 2013 (3)
September 2013 (4)
August 2013 (9)
July 2013 (1)
May 2013 (2)
April 2013 (3)
March 2013 (2)
January 2013 (1)
December 2012 (1)
August 2012 (1)
July 2012 (3)
June 2012 (1)
April 2012 (4)
March 2012 (5)
February 2012 (1)
December 2011 (2)
October 2011 (1)
September 2011 (3)
August 2011 (3)
July 2011 (3)
June 2011 (10)
May 2011 (2)
April 2011 (2)
March 2011 (5)
February 2011 (3)
January 2011 (2)
December 2010 (1)
November 2010 (1)
October 2010 (1)


Tags:

Southeast Creative Summit (13)
Post Production (13)
Adobe Premiere Pro (11)
Final Cut Pro (10)
Training (5)
Apple (5)
Editing (5)
Avid (4)
Production (4)
Adobe (3)
Dell (3)
video editing (3)
Production Planning (3)
NAB (2)
Color Enhancement (2)
Color Grade (2)
4k (2)
(2)
Business (2)
Monitors (2)
AJA Kona (2)
Davinci Resolve (2)
final cut pro (2)
show more
© 2019 CreativeCOW.net All Rights Reserved
[TOP]