As I reported a few weeks ago, Small Tree replaced our original ethernet SAN we've been running since December 2008 with a new system including an all new 48TB Granite Stor RAID II. I also reported that we discovered some things about the Macs that Steve and Chris wanted to go back and test even further.
During the original install, the new Edge Core Switch turned out to have an issue which caused one of the 10Gig ports to fail. But we all expected that the system would work well for us while a solution was found to that issue. Unfortunately, though we tested the system for a full day with all the systems running and doing some editing, after the first full week, we knew we had to improve the performance of the system in a hurry. There just seemed to be a fine tolerance to what could be happening at any given time to ensure that all the systems worked properly, particularly the ability to master shows without the edit to tape aborting for dropped frames. It really came to a head one afternoon when we were trying to get two network shows out at the same time and of course, dropped frames turned everything into a race against the clock to make overnight shipping. We made it, but it was way too close for comfort.
We remained in almost daily contact with Steve, Chris and the rest of the Small Tree engineering team and they repeatedly would log into our various computer systems, make some tweaks, take some notes and keep working at it. Then last week Steve, Chris and Corky Seeber made a return visit to our facility, but this time they brought a brand new Small Tree 10gig switch along with a quad port 10 Gb card (installed in the server to take the best advantage of the new Small Tree 10 GB switch), and 4 Single port Ethernet cards. Small Tree had noticed during testing in their offices that they were able to get better performance from the Small Tree 1 GbE cards than the internal Ethernet ports of the Apple systems when using the Small Tree 10 Gb Switch.
Basically what Small Tree has been able to do in the past is make high speed, off-the-shelf network switches work for high speed video editing. But from what I understand we've hit the breaking point where if ethernet shared storage is going to continue to evolve and move forward, we need switches designed more for that task than for regular network traffic. Enter the new Small Tree 24 Port 10Gigabit Ethernet switch.
This thing was built with complete 10GigE infrastructure within to ensure that the maximum data throughput is achieved at all times. Designed from the ground up by Small Tree, this unit is something that should be able to withstand the constant beating that a shop like ours throws at it. So that was step one, install the new switch to give the entire system a speed boost right off the bat.
The next step was to go through each individual computer in our facility and individually tune them further. The first time they were here they did the first round, but since discovered some more "secret sauce" to make things work more smoothly. They go into the Terminal of each machine and make some internal tweaks to the setup of each system. There's no "one setting fits all" as I found out. Each systems has to be configured based on all the particulars of how the machine is set up including the software it runs and the third party hardware installed. In the end, I believe Small Tree only installed one of the new 10GigE cards into the machines because quite frankly, the others didn't need them once they were tuned correctly.
In short order we had the SAN up and running and every edit suite playing down video timelines along with all of our iMacs. So that's 5 Mac Pro workstations and 7 iMacs all playing 720p or 1080i ProRes video. The Mac Pros were were all playing FCP 7 timelines in a loop and the iMacs were all playing 20 to 30 minute clips in a loop. The iMacs don't have editing software on them, they're used by Producers to review footage as necessary.
Once again, the system in Edit 1 was the most vexing because it's one of the fastest systems in the entire facility and the one we use to cut the feature documentaries, yet it would drop frames playing the same timeline that the slowest system in the shop could play with no problems. Normally you would say "Add more Ram!" "Add a faster GigE Card!" and those might fix the problem temporarily. But there was something fundamental with the way this machine was configured that needed to be addressed. This was a super fast 8 core machine being outperformed by a much slower four core machine. I would guess the pair of them spent about 6 hours just on that one Mac Pro but they finally hit on a combination of settings that made a major change in the behavior of the machine. We did not touch the RAM, we did not change the Ethernet Card (A Small Tree Peg1 card that's been in there all along), Steve and Chris simply kept going into the Terminal and making adjustments to the way the Mac Pro operates. I honestly don't know all the particulars but by the end of the day, the Edit 1 system was behaving better than it has in a long time.
So when we started this whole process of moving from the older SAN configuration to the new SAN configuration we could lay off a half hour show to tape, but we would have to carefully manage what all the clients on the SAN were doing. Even then, we could get those aborts due to dropped frames.
Now, we can literally lay off two shows simultaneously and not pay any attention to what any of the other clients are doing. We've never been able to do that. In fact we did it 4 times in a row as a test. Two Mac Pros laying off 30 minute 720p HD shows being converted to 1080i via the AJA Kona 3, Three other Mac Pros playing 5 to 30 minute timelines in a loop or editing / scrubbing video (I was jumping from edit suite to edit suite to take over) and the 7 iMacs all playing long video clips in a loop. In fact I even laid off 3 shows simultaneously in one test. We've never been able to do any of this in the past. Oh we could edit in all the suites at the same time no problem, but mastering two shows at the same time to tape, that's never happened, but we always knew that and planned for it.
Then on Monday the editors came in and without any prompting, one of them said, "Everything is snappier today." All three of my editors noticed immediate improved performance from the system. More realtime playback, dropped frames non-existant and absolutely no concern for mastering off shows anymore, even two at a time. After a full week of hammering on the system, I'm glad to say that the system is proving itself on a day to day basis.
And it's not just that we have some faster products now with the new 48TB RAID and the 10GigE switch from Small Tree. It's the technical experience by Steve, Chris and all the engineers at Small Tree to completely understand the inner workings of all the machines that are connected to the system. Not accepting that we can "just throw more RAM or another card" at the problem and try to make it go away. It's getting to the heart of the problem, understanding it and then taking the correct course of action to solve it. Sometimes that means adding more hardware / RAM and other times it simply means tweaking the inner workings of the system.
It's very exciting to see what started out really as a cheaper alternative to a fibre channel SAN now evolving into a much more robust and fast system. Oh and don't think this is a Mac based solution, this concept can run on Windows as well, these guys have expertise in both platforms and of course Linux too. That's VERY important to me right now as we ponder the future course of our company and the NLE solution(s) we go with. We have to be prepared for the possibility that a Windows workstations (or two, or three) could start appearing in our shop. Thankfully, the guys at Small Tree will be ready to take our SAN in whatever direction we need to go.
Ok, about that * in the title - Technically what we are using is a NAS, not a SAN. But when Bob Zelin and I first started talking about it publicly we both referred to this configuration as a SAN because that's what we call shared storage in video production. Steve Modica got tired of correcting us and just went along with our (incorrect) terminology. So if you're fussy about the correct terminology, what we are using to edit video is a NAS. Happy now? Good!