| Posted by VirtualRain, 12-17-2010, 12:49 AM | |
I'm just doing research on a project I am working on.
I'm looking into the best options for building a SAN, the SAN would have to have high I/O, so no SATA.
One option was NetApp, has anyone had any experience with them?
|
| Posted by NoSupportLinuxHostin, 12-17-2010, 05:12 AM | |
You could read http://www.zfsbuild.com/ or http://www.anandtech.com/show/3963/z...d-benchmarking
|
| Posted by eming, 12-17-2010, 05:57 AM | |
We've (OnApp) been through something like 300-400 client SAN's the last 5-6 months and it seems there is 3-4 different approaches out there.
One - lowcost - route is a whitebox with a bunch of 2TB SATA drives in it + something like OpenFiler. We've seen that setup more than once, but it still scares me. Not a good idea for redundancy, performance or reliability.
One step up would be to do Supermicro (something like SC836E1-R800B) with 14 SAS 500gb-1TB drives (more spindles/gb->better). Adding something like Open-E to it will make it easy for you to carve out lun's etc.
Finally going up another step (before going NetAPP/etc). You could take two of the SAN's above. Chuck in something like MaxIQ and you've got a high performing hardware setup. For redundancy I would suggest you go with Starwind, their active-active setup is sweet (if you can get over the fact it runs on windows). We've got a lot of OnApp clients with that exact setup, and I can vouch for it's performance and reliability.
Obviously the last step up would be go to with something like NetAPP, EMC or HP etc. Actually I've seen some new things coming out from VSI that looks really nice as well, and at a very good price.
Finally, in the last 6 months I've been in ongoing talks with the guys from http://acunu.com/ - you should keep an eye on them. They have some VERY interesting technology on the way.
D
|
| Posted by DeanoC, 12-17-2010, 10:48 AM | |
There is some interesting work going into hierarchical multiple node systems that look at the problem not from a disk point of view, most at the moment are custom builds, but the right ones can perform and be reliable far more than would suggest from the outside.
The work derives oddly not from the traditional file system work but from throughput computing areas.
The entirety of the SAN can be considered a throughput computing device. The large scale but slow spindles are considered the end of the memory hierarchy (treated like memory or virtual memory is treated in HPC). DDR and NV drives are different cache pools with CPU (and its memory and caches) and drive controllers as unreliable compute devices, each server node is then seen as a large compute node.
You can see some of the basic approaches in L2ARC, but that is just the tip of the ice-burg and its gets alot more interesting once you take the leap from disk thinking to throughput computing imho.
|
| Posted by WebGuyz, 12-17-2010, 10:51 AM | |
What do most SAN builders usually choose for their disk RAID level? RAID 10 or RAID 6?
Thanks!
|
| Posted by eming, 12-17-2010, 10:58 AM | |
I'm not sure I've seen anything but RAID10.
But it depends on what you are trying to do.
Raid 10 would give you faster reads and writes of the two, BUT it is possible to lose everything if you lose the wrong two drives. But on larger disk arrays you could lose exactly half the drives and retain full operations. If you loose two drives from the same side at the same time...you'r in trouble...
However, with Raid 6, your writes could be a bit slower because of the extra checksum. But you could lose any two drives and not lose any data.
|
| Posted by CRego3D, 12-17-2010, 11:16 AM | |
Raid-10
I would not advise anything other, not even Raid-60
|
| Posted by jameshsi, 12-20-2010, 02:04 PM | |
We are study how to make a workable and low cost iSCSI SAN to be used in onapp's solution, it seems there is no cheap way to do this.
If you only build one SAN, you feel if something happen on that SAN, you loss everything, all your VM clients will start calling you, even you have backup on a NAS, you still need time to get it back on what ever SAN you re-build.
So that means you need 2 SAN, 2 SAN means double your cost, one SAN is already cost you much, 2 SAN makes you .... painful ?
If you spend too much money on the storge, that means you might lost the competeition, or you can give large disk space as other hosting company do.
Is there anybody using onapp and happy to share how you build your SAN and how much you invest on it ?
|
| Posted by eming, 12-20-2010, 02:10 PM | |
This setup:
Isn't that bad really. Like $6k or less.
You should also consider going with a hosted infrastructure with a provider that already has a strong shared SAN setup that you can buy a 1TB from to get started. That way you wouldn't have to worry about redundancy, capex etc.
D
|
| Posted by JordanJ, 12-20-2010, 02:21 PM | |
I think alot of people will be shocked at the IOPS limitations on large sans.
When you factor in De-Dupe, IOPs, Redundancy, management tools - NetAPP is one of the cheapest ways to go.
|
| Posted by Uncorrupted-Michael, 12-20-2010, 03:10 PM | |
Yea the IOPS limitations are pretty rough. Even with SAS drives.
Last edited by Uncorrupted-Michael; 12-20-2010 at 03:13 PM.
|
| Posted by jameshsi, 12-20-2010, 03:22 PM | |
What u guys mean ? You mean it will cost much than you may imagine if you need a big storage ?
If you build a cloud for 500 clients, each of them need a 50GB space, you need 25000 GB = 25 TB space, is that big to you ?
|
| Posted by jameshsi, 12-20-2010, 03:26 PM | |
I don't think the above config can be less than USD$6k, also, it seems you didn't count the open-e or software you need to spend.
Also, the above's performance ?
|
| Posted by eming, 12-20-2010, 03:40 PM | |
That san performs very well because of the MaxIQ SSD controller dealing with the majority of the iops.
I ignored the price of san software (starwind/open-e) as you said this was for OnApp.
D
|
| Posted by jameshsi, 12-20-2010, 09:05 PM | |
Don't follow you, why ? you mean if go with onapp you don't have to use/pay for the (starwind/open-e) software ?
|
| Posted by Cristi4n, 12-21-2010, 04:47 AM | |
@eming the setup posted above has DDR2 fully buffer dimms. Are you sure E5620 works on a motherboard that supports DDR2 FB dimms. Those dimms are pretty expensive now compared to DDR3 that you usually add when you have an E5620. You can also get a MB that supports 4 x 1Gbit nics at a very similar price if you really need more speed
Anyway, I am curious why there is a need for 2 CPUs in a SAN ? The op needs a cheap SAN and I recommend for example an X3450 since the difference in price between 5x and 3x is pretty big.
If you have a choice than I would suggest looking at what OnAPP or whatever software you plan on using really knows about your setup. For example, you can buy a NetApp (pretty expensive and doesn't worth it, anyway that's just my opinion) but if your software doesn't know how to take full advantage of NetApp for example than you shouldn't actually buy one or at least look for an other that actually can do things like fast disk clone/snapshot and others.
I am curious about open-e, I've seen VPS.net wishing for Open-E to go out of business, there must have been a reason there.
I am not sure if I personally would even think of going with openfiler or open-e but if you have the hardware you can test every solution mentioned here (except for netapp) and draw your conclusions after that.
|
| Posted by eming, 12-21-2010, 05:35 AM | |
yes, Starwind/Open-E comes free with OnApp. Contact us for more details - let's keep that talk off WHT.
good points - will doublecheck the configs.
agree - 2 CPU's are not needed, and it would be a way for you to save a bit of $$. Obviously those crucial seconds startup's and reboots might take may warrent the extra CPU, but if you are on a budget, thats an obvious place to save! Good input!
yeah, they had a pretty rough ride with Open-E, and they are putting their money on Starwind now.
Starwind and Open-E are two very different beasts. I'd be happy to post more about my experiences with each of them if there is any interest. We've deployed many SAN's on both platforms and I'd be glad to share.
Have you looked into Nexenta?
Other alternative could be AOE based Coraid appliances - they've got a great pricing setup (lease options etc) and good technology.
Keep an eye on Acunu.com as well - they've got greatness on the way, could be the holy grail of storage...
D
|
| Posted by Cristi4n, 12-21-2010, 05:40 AM | |
@eming yes, I built a SR for nexenta and xenserver that I am testing now with everything supported like cloning and snapshots a.s.o.
However, Nexenta has some bugs so I adapted my SR to actually work with Solaris since I would trust that more than Nexenta for zfs.
I would be glad if we can have a quick talk about Open-E and Starwind or other things related to storage
Last edited by Cristi4n; 12-21-2010 at 05:46 AM.
|
| Posted by RavC, 12-21-2010, 05:58 AM | |
Please elaborate
|
| Posted by jameshsi, 12-21-2010, 07:02 AM | |
eming, You must have a lot of experience about iSCSI SAN, since onapp build a lot of clients' SAN, so, if there are some examples of fail ? Can you share to us ?
Currently I want to try onapp but I don't have a deep pocket full with money, so we might decide to test run onapp to build a small cloud, using the cheapest iSCSI SAN that you can build of, what will that be ?
Can it just use RAID5 or RAID6 instead of RAID 10 ?
If use RAID5 (DELL R610,1CPU,4GB RAM,6X 2.5 SATA 1TB)
+ Open-E (but without expensive Adaptec MAXIQ), what will happen ?
|
| Posted by eming, 12-21-2010, 07:06 AM | |
That would work, however, depending on your client profile you may hit a IOPS bottleneck fairly fast.
Raid5/6 would not be recommended though.
D
|
| Posted by jameshsi, 12-21-2010, 07:38 AM | |
When we hit the IOPS bottleneck, what will it looks like ? I mean what will happen ? High loading ? Client's server act very slowly ?
|
| Posted by DeanoC, 12-21-2010, 08:19 AM | |
Your best bet for a cheaper SAN is start with OpenIndiana/Nextenta and use ZFS and COMSTAR/iSCSI support (which are both built in). IOPS can be helped with a few SSD for ZIL and L2ARC.
Whilst not necessarily easy (tho nextenta do a free easy version upto 18TB) it will likely outperform in performance and price any other budget solution.
|
| Posted by CRego3D, 12-21-2010, 09:20 AM | |
Correct, the clients servers will experience high load due to IOwait being too high, slowing the cloud server functions.
|
| Posted by NodePlex, 12-22-2010, 09:13 AM | |
Yes, you can do this config (less software) for $6K or probably a little less. The MaxIQ Raid card is around $1K alone though. So while you could cheapen it up your performance would suffer.
Cutting corners on a cloud setup due to budget issues could be disaster as you could easily have thousands of dollars in a setup that could turn out useless for it's intended purpose due to low performance.
|
| Posted by DeanoC, 12-22-2010, 10:51 AM | |
Budget SAN-a-likes are very doable, but you will need someone who understands the soft/hard storage stack from top to bottom.
HW RAID really doesn't belong in a SAN (doesn't stop it being in many), but they are easy/simple to set-up I guess...
|
| Posted by AdmoNet, 12-26-2010, 10:48 PM | |
I second this quote.
NetApp has tons of extra value with block-level deduplication, easy SAN replication, flexible volumes and RAID-DP.
You simply can't find a more flexible _true_ multiprotocol solution. If anyone has any NetApp related questions, please let me know. Keep in mind an IOP is not always an IOP it really depends on the workload and desired latency.
NetApp wins hands down to any other solution.
|
| Posted by FranciscoV, 12-27-2010, 02:16 AM | |
I know some people that have been trying to start to offer their own cloud offerings. It's one of the first few types of hosting that I'm trying to learn. Most complain not because it's a pain to build, but because of the price that's involved in building a SAN. The most efficient and cost effective SAN would be a 24 bay chassis with a 5620 proc. Yes, that's 24 1TB or 500GB hard drives depending on how big you want your SAN.
Also, check out colocation. You might save yourself a few thousand dollars.
EDIT: I forgot to mention that RAID is a must as well.
|
| Posted by jameshsi, 12-27-2010, 04:11 AM | |
The RAID you mention must be a RAID 10 ?
|
| Posted by DeanoC, 12-27-2010, 07:05 AM | |
Which RAID is entirely dependent on how many disk failures you want to absorb. The issue for any SAN (which usually have at least >10TB of disks) is that when a failure occurs, RAID10 absorbs it but you can't take another failure without catastrophe. As the drives are working overtime due to working 1 disk down, the odds of another failure actually go up.
Hence why 2+ disk resilience (RAID6 or better) is becoming more popular.
RAID10 is probably okay if you have daily backups and your or your clients can handle losing a day of data.
|
| Posted by kris1351, 12-27-2010, 09:55 AM | |
Raid 6 is a large performance hit is why most support Raid 10. One thing you failed to mention is that when building a Raid 10 you can lose 1 drive on each side, but the immediate solution to that is having at least 2 hot spares in a system and a system that has monitoring to allow you to know when the hot spares have been utilized.
|
| Posted by AdmoNet, 12-27-2010, 12:09 PM | |
Unfortunately it's not 50 disks in one RAID-6 set. You would make two to three RAID-groups inside an aggregate with NetApp solutions.
This will limit your exposure to double disk failure as you have less disks inside one RAID group. NetApp's RAID-DP is as fast as RAID-10 hands down.
|
| Posted by DeanoC, 12-27-2010, 12:27 PM | |
RAID10 can survive 2 failures BUT not in all cases. The industry hasn't really settled on a better form of numbering when we get to probability based resiliency. In RAID10 it is 100% for 1 disk failure and a 2 disk resilience as a factor of the size of the array, iirc its (100-200/N)% where N is the number of drives in total array), the probability continues in linear progression decreases upto N/2 failures (beyond which is 0%).
The rate with which hot spares are useful, is related to the rate at which a hot spare can be converted to a replacement. For mirrors its time to copy the failed drive (or active data on the drive if usage tracking). For parity systems its the time to rebuild the disk to the hot spare which is generally much slower as touches more drives to get the data.
|
| Posted by DeanoC, 12-27-2010, 12:40 PM | |
Which is why most SAN vendors 'forget' to mention the probability of catastrophic failures. Normal aggregates of RAID (5/6/DP etc.) don't alter the resilience (worst case failure it can absorb) only the probability that the worst case won't happen. No matter how many RAID-DP groups, 3 disk failure within the same group is total failure.
Methods that can always guarantee more than that, are currently in the specialist areas.
|
| Posted by FranciscoV, 12-27-2010, 01:54 PM | |
Yeah I didn't mention which RAID because I wasn't too sure. I know that RAID10 is known to cause issues with arrays over 10TB but it seems to work pretty well when everything is in working order. On a side note though and it may help the OP or anyone else for that matter, I was discussing with my possible business partner yesterday that the only bad thing about the cloud is that if the SAN goes down, all of our clients go down with it and we came to the resolution that we can install r1soft onto each cloud node. This will help if the SAN ever does go down *knocks on wood* to keep the clients data safe and secure within the hypervisor.
|
| Posted by murphy13, 12-30-2010, 10:23 AM | |
Personally I love EMC equipment. I think that the CX4 is by far the most reliable, scalable, and has the highest amount of VMware integration among all the mid-range storage vendor. The biggest selling point for me was the Fully Automated Storage Tierring.
The EMC unified platform, known as Celerra, is also a nice option. It is basically the CX4 with a NAS filer on it.
|
| Posted by AdmoNet, 12-30-2010, 12:03 PM | |
Honestly between EMC and NetApp... NetApp wins price every time. Also the last CX-4 I installed had two Windows Storage Server licenses on the top. If you trust your data to Gates, by all means go with a glorified server chassis with some batteries slapped below them running Windows.
I don't believe in the block-level model where you simply keep handing out LUNs. I believe that the storage system should be able to pool storage (pooling RAID groups) to take advantage of all spindles. The upgrade path to add more storage shouldn't involve handing out more LUNs, it should involve growing the underlying storage for I/O requirements or space requirements.
To me, EMC seems very old school, block storage, very static. They say multiprotocol but are they talking about NFS too? Usually EMC requires you buy more "gateway" products to hand out their storage.
NetApp is a true unified device servicing FC,iSCSI,NFS,CIFS and allows you to create virtual filers for multi-tenant environments. It's simply more flexible.
Here is some SPEC numbers on NetApp vs. EMC: http://bit.ly/bJZpRD
Thanks!
|
| Posted by Visbits, 12-30-2010, 05:29 PM | |
If your looking at iSCSI the dell MD3220i + 3x MD1220 drive shelfs will give you 96 2.5" drive bays. We use these with 300gb 10k sas, we split them up 6 drives in raid 5. So far the performance has been great. And for the price point you can't beat it.
|
|
 Add to Favourites
 Print this Article
|
