Welcome! » Log In » Create A New Profile

FC tape over distance - 30%less perf 45km vs 10km

Posted by grogpolymer 
FC tape over distance - 30%less perf 45km vs 10km
October 12, 2016 05:10PM
Hi,

We are trying to read and write from/to Oracle T10KD drives over 8Gb FC links using Cisco ONS and Brocade 16Gb switches.

We can't get the links to trunk as ONS is using time division multiplexing on the links for the ports on the line card.

We have turned on compression at the brocade transport level and we have upped the buffer credits to where we see no buffer credit starvation.

Our data paths are 45km and 65km from the site where the host writing/reading tape is to to the 2 main campuses. The application is SGI DMF so data represents part of HSM file systems.

The tapes are being read/written as compressed. Doing a bit of homework shows that 8Gb FC should be capable of somewhere between 650MB and 800 MB/sec and that should be plenty to stream the drives.

We are seeing roughly 30% lower throughput than when we had the host at one campus and writing to the same drives but only using brocade 8Gb switches and hence no compression at that layer as it is only supported on the 16Gb switches.

I have a suspicion that the tape driver is where the bottleneck is as I would expect that the tape driver has a limit to how much data it can have in flight for writes as there needs to be adequate space to fit all data in flight on the tape after the drive has signaled EOT but I cannot find any documentation/definitions of what that limit is. Obviously an LTO4 and a T10KD may well be able to tolerate markedly different amounts of data written to that region of the tape.

I cannot think of anywhere else to look. The ONS is alleged to be getting 16Gb capable line cards in the future but if the limitation is where I think it is then that won't help.

I'm also curious about tape driver behavior for reads with respect to how many outstanding reads it can have. Read and write behaviour may well be limited higher up the stack with how much DMF tries to write ahead/read ahead. I will approach SGI on this.

I will also see what metrics I can get from our Unix team on read and write performance.

Regards,
Greg
FC tape over distance - 30%less perf 45km vs 10km
October 13, 2016 08:27AM
My first reaction was, "What? He's backing up directly to tape over a 45Km connection? And it's working at all?"

This is a complicated thing to solve, unfortunately. It's hard enough when it's a direct FC connection sitting 10M away. But if I read your post correctly, you are asking why your performance is 30% slower over a longer link. I'm going to go on a limb and say that the problem is latency and block ACKs. You prove/disprove this theory by using the same boxes and connecting them back to back. If that's faster, then there's your answer.

Most people wanting to do what you're doing do it with replicated virtual tape. You back up to a local virtual tape, dedupe that, then replicate it.
FC tape over distance - 30%less perf 45km vs 10km
December 05, 2016 03:26PM
Yep tape direct.

We have been looking at what we can do. We were forced to relocate HPC to a colo datacentre due to an upgrade on our datacentre being politically a step too far when the answer is cloud (apparently). We are examining Brocade SAN extension switches as these do provide tape acceleration by early acks for the blocks written. It does some funky things with error handling but it is far from cheap. That said, relocating a tape library to the colo would incur greater than 10K a month based on the way the contract works(footprint). It is called a rock and a hard place. SAMQFS would have given more transport options.

If it was backup then dedupe VTL would be great but it isn't. It is HSM (DMF) data for the SGI HPC system. Huge data sets and not going to dedupe. DMF is short of transport mechanisms - FTP or FC tape represent the 'broad' range of options. Indeed SGI suggested at one point to stand up another DMF instance to be the NFS target for the one at the colo. Then it gets down to having 2 points of presence for DMF reads and writes ergo two different DMF NFS targets with some disk and , guess what, two tape targets each. So we would end up with 4 copies of data where we have 2 now. This would of course be more systems, more tape carts, more drives, more slots required.

We are examining all options at this time. Can we get a more proximate location for tape? Can we leverage another Colo inhabitant's tape library (that is looking a possibility right now)?

SGI pointed out the obvious which is their CXFS but that would be a pull of all data back in to rewrite it into CXFS file systems and push it out again to tape with DMF. This is a major catch 22 given the Petabytes of research data.

We are also looking at what FC options we can rake up as there is a desire at management levels to move more stuff to the colo. No, apparently it does not have to make sense. Right now we are using a managed service that uses Cisco ONS 15454 which uses time division multiplexing and this does 2 things to us. 8Gb is as fast as we get and we can't trunk. We do have multiple links to each campus. We are looking at what we may be able to do with ROADMs and forming a ring between the various sites the university has connections to or sites of our own. Whether we use ROADMs or another vendor's amplified mux family for some links is being examined. At the time we were to move there was a managed service provider and a dark fibre provider only. It appears there may be more dark fibre providers now and we will look into that.

Regards and Thanks,
Greg
Sorry, only registered users may post in this forum.

Click here to login