How to back up 40TB of data...

[PewterTA]

Build another server!!!!



Second server case is in the house (RPC-4224) and I've already moved the original server into this case giving it 4 more drive bays (filled with 2TB drives that need added to the array) to allow for even more storage. Only mod I had to do was drill holes into the side of the case to allow for the 2 OS 2.5" HDs that I had in the original RPC-4020 case. I was worried that moving the drives from the SATA connection backplane to the SAS backplanes might cause an issue with the controller, but it didn't even blink and I'm getting better speeds (unexpected and cool!) and it's a lot nicer having just 6 SAS cables instead of all the SATA connections. I get much better air flow with this case as well, HDs stay right at 37 degrees celcius under heavy use...nice and cool!

I lost the ability of the DVD burner (but can easily put that in an external case), but I'm not that worried about it.

Now the old RPC-4020 will be the backup storage server for complete redundancy...though I'm waiting for HDs to come back down from their outrageous prices....

The backup server is an Asus systemboard with an AMD A4 (3.6GHz) and 8GB of RAM. Only 2 80GB drives for the OS, but not much is needed for that. Of course running Win2k8r2 with some HyperVs on it (like the original server). So eventually this will be a completely redundant backup of the original server.

Yes I got bored and wanted something to do and this is what happens...

[wkhanna]

Dude, that is just SICK!

The best part is Dan set me up so I have access to all his music files on his server directly from my personal laptop. :T

[wettou]

whoa doing Genomic Sequencing?

[PewterTA]

Inside the New Server case (old server actually):

[PewterTA]

Inside the Backup Server:

[PewterTA]

Wettou....

No, though I could or Seti... ha ha.

Actually it's all for media serving, BluRay, DVD, Music, Pictures, Videos, Application Storage, FTP hosting, Music and Video Streaming to my Android phone and whatever else I think up of doing!

[impala454]

Think I'll just stick with getting my ass off the couch to put a bluray in the player

Friggin awesome setup though :T

[Chris D]

Holy crap... at first I thought you were posting a funny pic of somebody that made something way beyond reality.

I bow to you, sir...

[aud19]

Holy Jebus! 8O

Kind of puts the potential 12TB NAS I want to build to shame ops:

[wkhanna]

Don't feel bad, Jason.
Dan just has a 'sickness' :W
I'm already investigating possible treatments/therapy.
An' intervention' is not being rule out :roflmao:

[Hdale85]

Well during the intervention I'll take possession of his servers to keep him from his addictive habits.

[PewterTA]

This is what happens when my GF dumps me and I have nothing better to do with my time... I think it is a sickness...it's just so easy to hit "buy" on websites! LOL!

Actually this all came about as I figured that even with Raid6 on the HDs... there's still a possibility of failure and it was either get an LTO 5 or better (really LTO 8 is what would be needed to make it all manageable)...or for the same price...well really cheaper...just duplicate the server. Again, once the HD prices come back down... So that's what I've started with.

I have 4 1.5TB drives and 4 2TB in the old server, well I moved the 4 2TB drives into the main server and will eventually expand the array to incoporate the 4 more drives giving me a full 40TB of useable space.

I just got XMBC configured so now I can at the touch of a finger play anything through it through the iPad 2! I don't have an iPad 2, but my sisters husband let me borrow his for a little bit to play with when he wasn't using it. So now I can flip through all the movies on the iPad, tap on the movie and it starts playing on the home theater!!! Pretty freaking cool.

I might invest in the iPad 2 to do all the control for the home theater equipment as well. They have a cool app and you buy the remote control IR modules to put on your equipment and gives you exactly the same as the Remotes you have and can easily input your remote into it. Supposedly has like a million remotes built into the app already or something like that. I'm thinking about this, but it's like a $100 for 2 or 3 IR devices... There's some money involvement there where I have money slated for other things (*cough* B&W 802Ds *cough*)... so that remote thingy will probably have to wait.

Only thing I need too is for HD prices to come out of the stratosphere... because I refuse to pay almost 3x the cost on same HDs as what I had just about a year ago.

So if anyone wants to donate I'm gladly accepting all 1.5TB, 2TB or 3TB drives I can get.

[Hdale85]

So you never sold the 1.5TB drives to the other guy huh? I wish you were still selling them lol! I actually have the money now Was looking to buy some but prices just too high.

[PewterTA]

Actually the OCD in me could say that the 2 new backplanes that Norco sent me were driving me nuts enough to buy the new server. They changed the power and activity LED colors and made them brighter (no complaints there)... So the old server case looked stupid!

[PewterTA]

New server lights look a lot better with the new backplane design that Norco is using.

[PewterTA]

Dougie, no, the guy never got back to me on the 1.5TBs. I wish you would've bought them at the time too as I would've had 4 2TB drives that I could really be using right now! ha ha. Oh well.

I'm sure I'll get rid of the 1.5TBs at some point by replacing them with either 2 or 3TB drives if the prices come down at some point.

[Hdale85]

Yeah my storage situation is painful lol. I'm going to have to clean up a bit here soon to clear up some space but I can't justify spending the money they want right now. They make poor business decisions and we have to pay......seem familiar?

[impala454]

I'm still just dumbfounded by the amount of space and what you would actually do with it. So you actually have 40TB of space? What approximately is the breakdown of files? I mean if you ripped 500 bluray movies at 50GB a piece (which most are nowhere near) that's still only 25TB. Throw in 10,000 CDs ripped completely lossless and you've got another what, 7TB? Maybe 100,000 full RAW images for another ~2TB? I mean, I am all about and love overkill, I just simply don't see the point. Unless you're running some kind of business we don't want to know about :P

[PewterTA]

Most of it is Movies, then Music, then software, if I had to guess.

My movies take up about 20TB currently, another 5TB of lossless Music Files (a lot of that is hi-res that are roughly 1.5GB a piece), and probably about 4TB of other software and application backups. Then probably another 2 or 3TB in backups of all the computers in the house, sisters, parents, etc. so things are lost for them...

There's some other things that it does as well that takes up space (FTP server).

The 2nd "backup" server will be purely for redudancy so it's not adding another ~40TB of space for 80TB.

[wkhanna]

Some people stockpile ammo, guns & food.
Some put their faith in a higher power.
Some rely on the hope the human race will some day get its act together.
One thing that has not changed is the fact that knowledge is power.
Today, that translates to DATA.
I think Dan is just preparing for the Big day :W

[PewterTA]

Yeah only got about another 10 months right??? Then the world ends...

Though I don't know what I'm going to do without electricity... Maybe I need some panels on the roof?!!?

Actually I'm working on just making my own Cloud.

[wkhanna]

Actually I'm working on just making my own Cloud.

:roflmao:

[impala454]

4 TB of software??!?

[Chris D]

Yeah, that's the problem... you can make electronics as great as you want, to prepare for Armageddon. But unless you've got your own generator and self-sufficient fuel source, you're going to be SOL when power goes out. (which could be pretty quick)

[madmac]

Just back it all up onto a netbook....DONE!!! Hehehehe!!!

[Chris D]

... and then the netbook lasts for 6 hours of battery power, and it's done too...

(you see the problem here)

[PewterTA]

Hence the solar panels... as long as they last all I need is sun.

But I'm in Pittsburgh...so we'd only get like 50 days of use a year! Or is even 50 days a stretch???! LOL

Yes 4TB of software. I have all my company's software that I use for repairing things on this server as well so if I'm out at a clients site and they don't have the discs we give them, I can get what I need off the server from their location. It's sort of become my store for everything I tend to possibly need to use on any given day. I basically got irritated having to go to a clients site twice because I wouldn't have the software I need to fix everything and to carry close to a 300 DVDs of software is just insane. Though I do have a lot of it on a 2TB drive that I keep with me most times as well...

Anything to make my day easier at work...well it's worth it! Plus it's nice to stream all my music right to my phone or computer from wherever I am, keeps me sort of sane...

[impala454]

300 DVDs of software??? And that's only 1.4TB I don't mean to keep questioning you but it just sounds nuts. I have a full MSDN subscription through work and I doubt I could download the entire thing and fill up 4TB... and that is every version of every piece of software MS has ever made, minus games. Just curious is all.

[PewterTA]

I have the full MSDN, I have entire Adobe collection, all the AutoCad, 10 different backup software sets, Audio Editing software, Video Editing software, POS (Point of Sale and Piece of Sh#t) software, etc... Cisco updates, WatchGuard updates, Barracuda updates, HP, IBM, Dell (firmware updates and patches). All the updated drives for all of the computers we have out from any manufacturer we've dealt with. We support 500 companies and beyond about MS & Adobe, we never know what we'll need and it all comes in handy to have on hand.

In total, work alone I probably have close to 2000 software packages and I have all updates for all of it organized and everything, it's amazing well organized if I say so.

Then included in that is my own personal software library collection of about 300 programs (probably of which, I only use about 50 now)...but it's never bad to "not" have them.

Oh... and we have images of all the types of machines we deploy for people, which is close to 50 I think at about 15GB per image, so it's close to if not about 1TB in that alone. It's nice to be able to just quickly (about an hour) completely rebuild a computer/server for a client when needed.

Again with it all... I'd say I don't use and probable don't need a good bit of it. But as soon as something is gone...that's when I'll be like, "I wish I had that piece of software still."

[impala454]

That is pretty crazy. Though yeah, you definitely don't need the full MSDN lol. Unless you're out there installing Office 95 or something :P

I'm guessing that storing all of that on your home setup isn't exactly kosher (though I'll never tell)

[madmac]

... and then the netbook lasts for 6 hours of battery power, and it's done too...

(you see the problem here)

Oh I see more problems than that!!. If you tried to put 40TB of data onto a little netbook I think it would explode!!! :rofl:

[PewterTA]

It might just be a little tougher to carry around... I don't think it'd explode though. Probably within 5 years those netbook/small laptop/tablet type of a thing will be standard with 40TB or 100TB!

[madmac]

It might just be a little tougher to carry around... I don't think it'd explode though. Probably within 5 years those netbook/small laptop/tablet type of a thing will be standard with 40TB or 100TB!

I love my little MSI Netbook but it only has a 160GB hard drive!!. That's a far cry from the amount of data we're talking about here!! :E

[Hdale85]

If you're just file serving it doesn't take a lot of computing power so a netbook could likely do it. Anything more then that you'd have issues though.

[PewterTA]

Dougie... that's like saying you'd take a vette with a Geo Metro engine in it.... :P HA HA HA.

You are right, there's not much computing power needed for just fileserving...but it also depends on how much file serving you're doing also. If you're pushing 4 or 5 HD streams a netbook might have a slight issue with doing that (providing you have the connections for all the HD data storage built into it).

[numberoneoppa]

Just came in to say that you have good taste.

[impala454]

Netbook power is ridiculous. My two year old phone ousts most of those things. You definitely need more than that to host a 40TB file server. I didn't catch exactly what SAS card or whatever Pewter is using but it definitely requires more beef than any netbook is going to have.

[PewterTA]

Both SAS cards are Areca ARC-1880ix-24 4GB cards. Main server has the battery backup for the card (not the server), backup server does not *yet*.

Both servers are on APC 1500 Battery Backups incase of power loss (well the whole network is completely on battery backup to keep it up should need be during a power failure).

[knguyen429]

;x( ;x( ;x( and i thought my 26tb was huge. i know what you mean about hdd prices. i've been at 95% capacity since the beginning of the year, but i can't justify paying these current prices on hdds.

[Hdale85]

Yeah now that I have money to buy hard drives I can't lol. I've been floating between 95 and nearly 100% for quite a while

[Kevin P]

What gets me is the price of computers and laptops haven't gone up when HDD prices did. Which means either comp manufacturers stockpile drives to put in their machines, or the HDD price gouging is only for end-user purchased drives, and not OEMs.

[PewterTA]

It's a little bit of both Kevin. The big manufacturers have contracts that keep the stockpiled drives coming in and drives that are being slowly made are given to them first before any go out for consumers.

I'd say about a year and they might be back down to almost where they were. I've heard production is about 70% of what it was already...so that's getting there and the prices are slowly coming down...so...... who knows when it gets back to normal.

Dougie... I have 2 1.5TB drives if you need them... I could be willing to let them go at a decent price. Unfortunately, the other 2 1.5TBs are in use at my parents place and I don't know when I'll be getting them back (they are doing some video editing of my sister's wedding).