How to back up 40TB of data...

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • PewterTA
    Moderator
    • Nov 2004
    • 2901

    How to back up 40TB of data...

    Build another server!!!!



    Second server case is in the house (RPC-4224) and I've already moved the original server into this case giving it 4 more drive bays (filled with 2TB drives that need added to the array) to allow for even more storage. Only mod I had to do was drill holes into the side of the case to allow for the 2 OS 2.5" HDs that I had in the original RPC-4020 case. I was worried that moving the drives from the SATA connection backplane to the SAS backplanes might cause an issue with the controller, but it didn't even blink and I'm getting better speeds (unexpected and cool!) and it's a lot nicer having just 6 SAS cables instead of all the SATA connections. I get much better air flow with this case as well, HDs stay right at 37 degrees celcius under heavy use...nice and cool!

    I lost the ability of the DVD burner (but can easily put that in an external case), but I'm not that worried about it.

    Now the old RPC-4020 will be the backup storage server for complete redundancy...though I'm waiting for HDs to come back down from their outrageous prices....

    The backup server is an Asus systemboard with an AMD A4 (3.6GHz) and 8GB of RAM. Only 2 80GB drives for the OS, but not much is needed for that. Of course running Win2k8r2 with some HyperVs on it (like the original server). So eventually this will be a completely redundant backup of the original server.

    Yes I got bored and wanted something to do and this is what happens...
    Digital Audio makes me Happy.
    -Dan
  • wkhanna
    Grumpy Old Super Moderator Emeritus
    • Jan 2006
    • 5673

    #2
    Dude, that is just SICK!

    The best part is Dan set me up so I have access to all his music files on his server directly from my personal laptop. :T
    _


    Bill

    Practicing Curmudgeon & Audio Snob
    ....just an "ON" switch, Please!

    FinleyAudio

    Comment

    • wettou
      Ultra Senior Member
      • May 2006
      • 3389

      #3
      whoa doing Genomic Sequencing?
      Farming looks mighty easy when your plow is a pencil, and you're a thousand miles from the corn field."Dwight D. Eisenhower

      Comment

      • PewterTA
        Moderator
        • Nov 2004
        • 2901

        #4
        Inside the New Server case (old server actually):





        Digital Audio makes me Happy.
        -Dan

        Comment

        • PewterTA
          Moderator
          • Nov 2004
          • 2901

          #5
          Inside the Backup Server:





          Digital Audio makes me Happy.
          -Dan

          Comment

          • PewterTA
            Moderator
            • Nov 2004
            • 2901

            #6
            Wettou....

            No, though I could or Seti... ha ha.

            Actually it's all for media serving, BluRay, DVD, Music, Pictures, Videos, Application Storage, FTP hosting, Music and Video Streaming to my Android phone and whatever else I think up of doing!
            Digital Audio makes me Happy.
            -Dan

            Comment

            • impala454
              Ultra Senior Member
              • Oct 2007
              • 3814

              #7
              Think I'll just stick with getting my ass off the couch to put a bluray in the player

              Friggin awesome setup though :T
              -Chuck

              Comment

              • Chris D
                Moderator Emeritus
                • Dec 2000
                • 16877

                #8
                Holy crap... at first I thought you were posting a funny pic of somebody that made something way beyond reality.

                I bow to you, sir...
                CHRIS

                Well, we're safe for now. Thank goodness we're in a bowling alley.
                - Pleasantville

                Comment

                • aud19
                  Twin Moderator Emeritus
                  • Aug 2003
                  • 16706

                  #9
                  Holy Jebus! 8O

                  Kind of puts the potential 12TB NAS I want to build to shame ops:
                  Jason

                  Comment

                  • wkhanna
                    Grumpy Old Super Moderator Emeritus
                    • Jan 2006
                    • 5673

                    #10
                    Don't feel bad, Jason.
                    Dan just has a 'sickness' :W
                    I'm already investigating possible treatments/therapy.
                    An' intervention' is not being rule out :roflmao:
                    _


                    Bill

                    Practicing Curmudgeon & Audio Snob
                    ....just an "ON" switch, Please!

                    FinleyAudio

                    Comment

                    • Hdale85
                      Moderator Emeritus
                      • Jan 2006
                      • 16075

                      #11
                      Well during the intervention I'll take possession of his servers to keep him from his addictive habits.

                      Comment

                      • PewterTA
                        Moderator
                        • Nov 2004
                        • 2901

                        #12
                        This is what happens when my GF dumps me and I have nothing better to do with my time... I think it is a sickness...it's just so easy to hit "buy" on websites! LOL!

                        Actually this all came about as I figured that even with Raid6 on the HDs... there's still a possibility of failure and it was either get an LTO 5 or better (really LTO 8 is what would be needed to make it all manageable)...or for the same price...well really cheaper...just duplicate the server. Again, once the HD prices come back down... So that's what I've started with.

                        I have 4 1.5TB drives and 4 2TB in the old server, well I moved the 4 2TB drives into the main server and will eventually expand the array to incoporate the 4 more drives giving me a full 40TB of useable space.

                        I just got XMBC configured so now I can at the touch of a finger play anything through it through the iPad 2! I don't have an iPad 2, but my sisters husband let me borrow his for a little bit to play with when he wasn't using it. So now I can flip through all the movies on the iPad, tap on the movie and it starts playing on the home theater!!! Pretty freaking cool.

                        I might invest in the iPad 2 to do all the control for the home theater equipment as well. They have a cool app and you buy the remote control IR modules to put on your equipment and gives you exactly the same as the Remotes you have and can easily input your remote into it. Supposedly has like a million remotes built into the app already or something like that. I'm thinking about this, but it's like a $100 for 2 or 3 IR devices... There's some money involvement there where I have money slated for other things (*cough* B&W 802Ds *cough*)... so that remote thingy will probably have to wait.

                        Only thing I need too is for HD prices to come out of the stratosphere... because I refuse to pay almost 3x the cost on same HDs as what I had just about a year ago.

                        So if anyone wants to donate I'm gladly accepting all 1.5TB, 2TB or 3TB drives I can get.
                        Digital Audio makes me Happy.
                        -Dan

                        Comment

                        • Hdale85
                          Moderator Emeritus
                          • Jan 2006
                          • 16075

                          #13
                          So you never sold the 1.5TB drives to the other guy huh? I wish you were still selling them lol! I actually have the money now Was looking to buy some but prices just too high.

                          Comment

                          • PewterTA
                            Moderator
                            • Nov 2004
                            • 2901

                            #14
                            Actually the OCD in me could say that the 2 new backplanes that Norco sent me were driving me nuts enough to buy the new server. They changed the power and activity LED colors and made them brighter (no complaints there)... So the old server case looked stupid!

                            Digital Audio makes me Happy.
                            -Dan

                            Comment

                            • PewterTA
                              Moderator
                              • Nov 2004
                              • 2901

                              #15
                              New server lights look a lot better with the new backplane design that Norco is using.

                              Digital Audio makes me Happy.
                              -Dan

                              Comment

                              • PewterTA
                                Moderator
                                • Nov 2004
                                • 2901

                                #16
                                Dougie, no, the guy never got back to me on the 1.5TBs. I wish you would've bought them at the time too as I would've had 4 2TB drives that I could really be using right now! ha ha. Oh well.

                                I'm sure I'll get rid of the 1.5TBs at some point by replacing them with either 2 or 3TB drives if the prices come down at some point.
                                Digital Audio makes me Happy.
                                -Dan

                                Comment

                                • Hdale85
                                  Moderator Emeritus
                                  • Jan 2006
                                  • 16075

                                  #17
                                  Yeah my storage situation is painful lol. I'm going to have to clean up a bit here soon to clear up some space but I can't justify spending the money they want right now. They make poor business decisions and we have to pay......seem familiar?

                                  Comment

                                  • impala454
                                    Ultra Senior Member
                                    • Oct 2007
                                    • 3814

                                    #18
                                    I'm still just dumbfounded by the amount of space and what you would actually do with it. So you actually have 40TB of space? What approximately is the breakdown of files? I mean if you ripped 500 bluray movies at 50GB a piece (which most are nowhere near) that's still only 25TB. Throw in 10,000 CDs ripped completely lossless and you've got another what, 7TB? Maybe 100,000 full RAW images for another ~2TB? I mean, I am all about and love overkill, I just simply don't see the point. Unless you're running some kind of business we don't want to know about :P
                                    -Chuck

                                    Comment

                                    • PewterTA
                                      Moderator
                                      • Nov 2004
                                      • 2901

                                      #19
                                      Most of it is Movies, then Music, then software, if I had to guess.

                                      My movies take up about 20TB currently, another 5TB of lossless Music Files (a lot of that is hi-res that are roughly 1.5GB a piece), and probably about 4TB of other software and application backups. Then probably another 2 or 3TB in backups of all the computers in the house, sisters, parents, etc. so things are lost for them...

                                      There's some other things that it does as well that takes up space (FTP server).

                                      The 2nd "backup" server will be purely for redudancy so it's not adding another ~40TB of space for 80TB.
                                      Digital Audio makes me Happy.
                                      -Dan

                                      Comment

                                      • wkhanna
                                        Grumpy Old Super Moderator Emeritus
                                        • Jan 2006
                                        • 5673

                                        #20
                                        Some people stockpile ammo, guns & food.
                                        Some put their faith in a higher power.
                                        Some rely on the hope the human race will some day get its act together.
                                        One thing that has not changed is the fact that knowledge is power.
                                        Today, that translates to DATA.
                                        I think Dan is just preparing for the Big day :W
                                        _


                                        Bill

                                        Practicing Curmudgeon & Audio Snob
                                        ....just an "ON" switch, Please!

                                        FinleyAudio

                                        Comment

                                        • PewterTA
                                          Moderator
                                          • Nov 2004
                                          • 2901

                                          #21
                                          Yeah only got about another 10 months right??? Then the world ends...

                                          Though I don't know what I'm going to do without electricity... Maybe I need some panels on the roof?!!?

                                          Actually I'm working on just making my own Cloud.
                                          Digital Audio makes me Happy.
                                          -Dan

                                          Comment

                                          • wkhanna
                                            Grumpy Old Super Moderator Emeritus
                                            • Jan 2006
                                            • 5673

                                            #22
                                            Originally posted by PewterTA
                                            Actually I'm working on just making my own Cloud.
                                            :roflmao:
                                            _


                                            Bill

                                            Practicing Curmudgeon & Audio Snob
                                            ....just an "ON" switch, Please!

                                            FinleyAudio

                                            Comment

                                            • impala454
                                              Ultra Senior Member
                                              • Oct 2007
                                              • 3814

                                              #23
                                              4 TB of software??!?
                                              -Chuck

                                              Comment

                                              • Chris D
                                                Moderator Emeritus
                                                • Dec 2000
                                                • 16877

                                                #24
                                                Yeah, that's the problem... you can make electronics as great as you want, to prepare for Armageddon. But unless you've got your own generator and self-sufficient fuel source, you're going to be SOL when power goes out. (which could be pretty quick)
                                                CHRIS

                                                Well, we're safe for now. Thank goodness we're in a bowling alley.
                                                - Pleasantville

                                                Comment

                                                • madmac
                                                  Moderator Emeritus
                                                  • Aug 2010
                                                  • 3122

                                                  #25
                                                  Just back it all up onto a netbook....DONE!!! Hehehehe!!!
                                                  Dan Madden :T

                                                  Comment

                                                  • Chris D
                                                    Moderator Emeritus
                                                    • Dec 2000
                                                    • 16877

                                                    #26
                                                    ... and then the netbook lasts for 6 hours of battery power, and it's done too...

                                                    (you see the problem here)
                                                    CHRIS

                                                    Well, we're safe for now. Thank goodness we're in a bowling alley.
                                                    - Pleasantville

                                                    Comment

                                                    • PewterTA
                                                      Moderator
                                                      • Nov 2004
                                                      • 2901

                                                      #27
                                                      Hence the solar panels... as long as they last all I need is sun.

                                                      But I'm in Pittsburgh...so we'd only get like 50 days of use a year! Or is even 50 days a stretch???! LOL

                                                      Yes 4TB of software. I have all my company's software that I use for repairing things on this server as well so if I'm out at a clients site and they don't have the discs we give them, I can get what I need off the server from their location. It's sort of become my store for everything I tend to possibly need to use on any given day. I basically got irritated having to go to a clients site twice because I wouldn't have the software I need to fix everything and to carry close to a 300 DVDs of software is just insane. Though I do have a lot of it on a 2TB drive that I keep with me most times as well...

                                                      Anything to make my day easier at work...well it's worth it! Plus it's nice to stream all my music right to my phone or computer from wherever I am, keeps me sort of sane...
                                                      Digital Audio makes me Happy.
                                                      -Dan

                                                      Comment

                                                      • impala454
                                                        Ultra Senior Member
                                                        • Oct 2007
                                                        • 3814

                                                        #28
                                                        300 DVDs of software??? And that's only 1.4TB I don't mean to keep questioning you but it just sounds nuts. I have a full MSDN subscription through work and I doubt I could download the entire thing and fill up 4TB... and that is every version of every piece of software MS has ever made, minus games. Just curious is all.
                                                        -Chuck

                                                        Comment

                                                        • PewterTA
                                                          Moderator
                                                          • Nov 2004
                                                          • 2901

                                                          #29
                                                          I have the full MSDN, I have entire Adobe collection, all the AutoCad, 10 different backup software sets, Audio Editing software, Video Editing software, POS (Point of Sale and Piece of Sh#t) software, etc... Cisco updates, WatchGuard updates, Barracuda updates, HP, IBM, Dell (firmware updates and patches). All the updated drives for all of the computers we have out from any manufacturer we've dealt with. We support 500 companies and beyond about MS & Adobe, we never know what we'll need and it all comes in handy to have on hand.

                                                          In total, work alone I probably have close to 2000 software packages and I have all updates for all of it organized and everything, it's amazing well organized if I say so.

                                                          Then included in that is my own personal software library collection of about 300 programs (probably of which, I only use about 50 now)...but it's never bad to "not" have them.

                                                          Oh... and we have images of all the types of machines we deploy for people, which is close to 50 I think at about 15GB per image, so it's close to if not about 1TB in that alone. It's nice to be able to just quickly (about an hour) completely rebuild a computer/server for a client when needed.

                                                          Again with it all... I'd say I don't use and probable don't need a good bit of it. But as soon as something is gone...that's when I'll be like, "I wish I had that piece of software still."
                                                          Digital Audio makes me Happy.
                                                          -Dan

                                                          Comment

                                                          • impala454
                                                            Ultra Senior Member
                                                            • Oct 2007
                                                            • 3814

                                                            #30
                                                            That is pretty crazy. Though yeah, you definitely don't need the full MSDN lol. Unless you're out there installing Office 95 or something :P

                                                            I'm guessing that storing all of that on your home setup isn't exactly kosher (though I'll never tell)
                                                            -Chuck

                                                            Comment

                                                            • madmac
                                                              Moderator Emeritus
                                                              • Aug 2010
                                                              • 3122

                                                              #31
                                                              Originally posted by Chris D
                                                              ... and then the netbook lasts for 6 hours of battery power, and it's done too...

                                                              (you see the problem here)

                                                              Oh I see more problems than that!!. If you tried to put 40TB of data onto a little netbook I think it would explode!!! :rofl:
                                                              Dan Madden :T

                                                              Comment

                                                              • PewterTA
                                                                Moderator
                                                                • Nov 2004
                                                                • 2901

                                                                #32
                                                                It might just be a little tougher to carry around... I don't think it'd explode though. Probably within 5 years those netbook/small laptop/tablet type of a thing will be standard with 40TB or 100TB!
                                                                Digital Audio makes me Happy.
                                                                -Dan

                                                                Comment

                                                                • madmac
                                                                  Moderator Emeritus
                                                                  • Aug 2010
                                                                  • 3122

                                                                  #33
                                                                  Originally posted by PewterTA
                                                                  It might just be a little tougher to carry around... I don't think it'd explode though. Probably within 5 years those netbook/small laptop/tablet type of a thing will be standard with 40TB or 100TB!

                                                                  I love my little MSI Netbook but it only has a 160GB hard drive!!. That's a far cry from the amount of data we're talking about here!! :E
                                                                  Dan Madden :T

                                                                  Comment

                                                                  • Hdale85
                                                                    Moderator Emeritus
                                                                    • Jan 2006
                                                                    • 16075

                                                                    #34
                                                                    If you're just file serving it doesn't take a lot of computing power so a netbook could likely do it. Anything more then that you'd have issues though.

                                                                    Comment

                                                                    • PewterTA
                                                                      Moderator
                                                                      • Nov 2004
                                                                      • 2901

                                                                      #35
                                                                      Dougie... that's like saying you'd take a vette with a Geo Metro engine in it.... :P HA HA HA.

                                                                      You are right, there's not much computing power needed for just fileserving...but it also depends on how much file serving you're doing also. If you're pushing 4 or 5 HD streams a netbook might have a slight issue with doing that (providing you have the connections for all the HD data storage built into it).
                                                                      Digital Audio makes me Happy.
                                                                      -Dan

                                                                      Comment

                                                                      • numberoneoppa
                                                                        Senior Member
                                                                        • Sep 2009
                                                                        • 535

                                                                        #36
                                                                        Just came in to say that you have good taste.
                                                                        -Josh

                                                                        That feeling when things are finally going right. Yeah, that one.

                                                                        Comment

                                                                        • impala454
                                                                          Ultra Senior Member
                                                                          • Oct 2007
                                                                          • 3814

                                                                          #37
                                                                          Netbook power is ridiculous. My two year old phone ousts most of those things. You definitely need more than that to host a 40TB file server. I didn't catch exactly what SAS card or whatever Pewter is using but it definitely requires more beef than any netbook is going to have.
                                                                          -Chuck

                                                                          Comment

                                                                          • PewterTA
                                                                            Moderator
                                                                            • Nov 2004
                                                                            • 2901

                                                                            #38
                                                                            Both SAS cards are Areca ARC-1880ix-24 4GB cards. Main server has the battery backup for the card (not the server), backup server does not *yet*.

                                                                            Both servers are on APC 1500 Battery Backups incase of power loss (well the whole network is completely on battery backup to keep it up should need be during a power failure).
                                                                            Digital Audio makes me Happy.
                                                                            -Dan

                                                                            Comment

                                                                            • knguyen429
                                                                              Member
                                                                              • Feb 2010
                                                                              • 37

                                                                              #39
                                                                              ;x( ;x( ;x( and i thought my 26tb was huge. i know what you mean about hdd prices. i've been at 95% capacity since the beginning of the year, but i can't justify paying these current prices on hdds.
                                                                              - Kenneth Nguyen

                                                                              Comment

                                                                              • Hdale85
                                                                                Moderator Emeritus
                                                                                • Jan 2006
                                                                                • 16075

                                                                                #40
                                                                                Yeah now that I have money to buy hard drives I can't lol. I've been floating between 95 and nearly 100% for quite a while

                                                                                Comment

                                                                                • Kevin P
                                                                                  Member
                                                                                  • Aug 2000
                                                                                  • 10809

                                                                                  #41
                                                                                  What gets me is the price of computers and laptops haven't gone up when HDD prices did. Which means either comp manufacturers stockpile drives to put in their machines, or the HDD price gouging is only for end-user purchased drives, and not OEMs.

                                                                                  Comment

                                                                                  • PewterTA
                                                                                    Moderator
                                                                                    • Nov 2004
                                                                                    • 2901

                                                                                    #42
                                                                                    It's a little bit of both Kevin. The big manufacturers have contracts that keep the stockpiled drives coming in and drives that are being slowly made are given to them first before any go out for consumers.

                                                                                    I'd say about a year and they might be back down to almost where they were. I've heard production is about 70% of what it was already...so that's getting there and the prices are slowly coming down...so...... who knows when it gets back to normal.

                                                                                    Dougie... I have 2 1.5TB drives if you need them... I could be willing to let them go at a decent price. Unfortunately, the other 2 1.5TBs are in use at my parents place and I don't know when I'll be getting them back (they are doing some video editing of my sister's wedding).
                                                                                    Digital Audio makes me Happy.
                                                                                    -Dan

                                                                                    Comment

                                                                                    Working...
                                                                                    😀
                                                                                    😂
                                                                                    🥰
                                                                                    😘
                                                                                    🤢
                                                                                    😎
                                                                                    😞
                                                                                    😡
                                                                                    👍
                                                                                    👎
                                                                                    Searching...Please wait.
                                                                                    An unexpected error was returned: 'Your submission could not be processed because you have logged in since the previous page was loaded.

                                                                                    Please push the back button and reload the previous window.'
                                                                                    An unexpected error was returned: 'Your submission could not be processed because the token has expired.

                                                                                    Please push the back button and reload the previous window.'
                                                                                    An internal error has occurred and the module cannot be displayed.
                                                                                    There are no results that meet this criteria.
                                                                                    Search Result for "|||"