Best way to backup terabytes of data?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Kevin P
    Member
    • Aug 2000
    • 10809

    Best way to backup terabytes of data?

    I know some of you have huge media servers with many terabytes of storage. How do you back those puppies up?

    Now that I got a modest raise, I'm planning some home network upgrades. Although my 6 year old file server still works fine and has ~1 TB of stuff on it, there's also the other computers that have varying amounts of stuff on them (my laptop being the biggest with hundreds of GB). And currently I don't have a good backup plan. Once in a while I'll back up the desktops/laptops to the file server and even rarer will I hook a 1 TB external drive up to the server and at least back up the important stuff. The MythTV HTPC hardly gets backed up at all, though most of what's on there I don't care much about, but that's about 800 GB of TV shows we never get around to watching.

    There's a number of possibilities. The simplest would be to pick up a couple ~4TB external HDs and back up onto those, rotating between them and keeping one offsite. I'll probably want to encrypt them in case the offsite one gets lost/stolen (I'll probably keep it in a drawer at work). The drawback to these is I'll outgrow them if my total storage exceeds 4 TB in a few years, though I can stretch the backups by only backing up important files.

    Another is to build a new file server and then use the old one as a backup. That's more complex and costly though and I don't have a lot of space to have a bunch of machines sitting around. Or buy a pre-built NAS.

    I know there's tape drives, but how many can store multiple TBs on a tape? And how expensive? I haven't researched them.

    The servers are all Linux (Ubuntu variants) so I'll probably just use rsync to back them up. Windows machines have Acronis on them and I generally back those up to the file server.
  • Hdale85
    Moderator Emeritus
    • Jan 2006
    • 16075

    #2
    I just have dual redundant drives. Haven't really had an issue over the years. I'll be replacing my drives soon with 8 new drives (all my new server holds). I mean if you have really important data on there you could build a second server with dual redundant drives that backs up only your really important data such as pictures and such. I might use my left over 3TB drives for something like that at some point in a NAS box.

    Comment

    • Kevin P
      Member
      • Aug 2000
      • 10809

      #3
      Redundant drives protect against drive failure, but not against fire, flood, theft, accidental deletion, or a hardware failure that takes out multiple drives. I had one machine lose its power supply a few years ago and took out the motherboard, drive and video card. If that had been my file server, well... you know the rest.

      I have my stuff on a RAID 5, but once again, I have no off-site backups. The most important stuff is photos and some video, and some documents.

      Right now I'm leaning toward a pair of large external drives, backup to them and rotate them, keeping one off-site. Down the road I may build a new server with more drive bays and make the old one the backup. Then I can switch to something like ZFS so I can more easily expand the array and use snapshots.

      Comment

      • BWLover
        Senior Member
        • Jan 2009
        • 552

        #4
        I use a service called Crashplan. It's $6/month (cheaper if you buy 1, 2, 3, or 4 years at a time). There is no limit to the amount of data you can upload to their servers. And the software will also back up to local external hard drives. You can also pay $125 (called seeded back up service) and they will send you a 1TB hard drive that you return to them. They then put that data on their servers to speed up the initial back up. Unfortunately 1TB is the biggest hard drive they will send you. How fast is your upload speed? The software (after you initial back up is done) monitors your selected folders and uploads any new files in the background. I've got a friend who has about 4TB of data in their cloud and it took about 4 months strait to back it up. Mind you Canadian upload speeds blow.


        Sent from my iPhone using the Tapatalk app
        Bowers & Wilkins 683 Speakers
        Rotel RB-1090 2 Channel Amp
        Rotel RC-1082 Stereo Pre Amp
        Rotel RCD-1072 CD Player
        Pro-Ject Debut Carbon w/ Ortofon 2M Red (sitting on a piece of slate supported by 3 "solid tech feet of silence" isolation feet)
        Rotel RLC-1040 Power Conditioner
        Shynyata Research SR-Z1 Power Outlet & Venom 3 Power Cords x 4
        Tara Labs RSC Vector 1 Speaker Cables & Interconnects
        Pioneer PDP-5070HD 50" Plasma
        Playstation 3
        Shaw HD PVR
        Primacoustic Room Treatments

        Comment

        • mjb
          Super Senior Member
          • Mar 2005
          • 1483

          #5
          I prefer to use HD's for backups. - and then hide them... low tech here, lol - my upload speeds don't allow me to use the "cloud" solutions.
          - Mike

          Main System:
          B&W 802D, HTM2D, SCMS
          Classé SSP-800, CA-2200, CA-5100

          Comment

          • Hdale85
            Moderator Emeritus
            • Jan 2006
            • 16075

            #6
            Well I'm not sure how recoverable normal raid arrays are but if I have hardware failure with my ZFS array its very easy to pop them in a new machine and recover the array. But I agree about fires and such, there is a "fireproof" NAS out there that's been tested to withstand quite a bit of fire but its not horribly cheap.

            My brother in law is going to be building a server soon and I think we are going to setup a backup between the 2 servers of the important stuff so I have copies in 2 locations in case of any sort of failure including fire flood etc.

            Comment

            • Kevin P
              Member
              • Aug 2000
              • 10809

              #7
              ZFS has about the same recoverability as an equivalent RAID array. So, a mirror is equivalent to RAID 1, RAIDZ is equivalent to RAID 5 (can handle loss of 1 drive), RAIDZ2 is like RAID 6 (can handle loss of 2 drives) etc. The problem is if you lose more drives than that (say, 2 drives in a mirror or RAID 5). Or, in a fire or flood, losing ALL your drives. The best protection there is an offsite backup. Where ZFS excels is in data integrity (better at detecting and fixing random errors/corruption than a standard RAID setup) and in management (snapshots, ability to divide your pool into dynamic filesystems with different attributes for different needs), etc. I don't know how solid ZFS on Linux is either. I know it was dicey for a while, but it sounds like it works pretty well now. I played with it in a VM one time and it seemed to work fine, but I didn't really bang on it that hard either. I could use FreeBSD but there's a learning curve there, I've been a Linux guy for decades so I know that like the back of my hand.

              Ironically, when I had a fire 10 years ago, my backups all melted (they were on DVD-Rs on a high shelf) but all my hard drives being in towers on the floor survived, so I didn't lose any data.

              Cloud solutions work if you don't have a lot of data to backup. Upload speeds tend to suck in North America in general, and on Comcrap there's a ~300 GB/month cap as well (not always enforced but something I have to keep in mind). Plus I don't trust cloud solutions as far as security or trust either. Do I really want my private files on a server I have no control over? I prefer to own the equipment my files are on.

              Then there's the logistics. Backing "everything" up is simpler to set up but you need more space, and not everything really needs to be backed up. Backing up just the important stuff saves space and time, but then there's the issue of configuring the backup to back up the things you want, and hope you didn't miss some odd folder somewhere. Most of my files are on a single share on my server, but not everything in that share is important enough to be backed up regularly.

              Probably what I'll do is buy two 4 TB USB 3.0 drives. Set up encrypted filesystems on them, and plug one into my server and have a cron job periodically rsync the entire RAID array to it. Then, once a week or so, switch the two external drives and take one offsite (stuff in a drawer at work). Then for my other computers, just back those up to the server.

              I tried the rsync thing with a 1 TB external I have now, and it seemed to work fine, but it doesn't have enough space to back up everything on the array that I want to backup. I figure a pair of 4 TB drives will be enough for several years. I just have to find some that are reliable. I tend to lean toward Western Digital as their drives tend to be better than average for reliability but I've seen mixed reviews on the external ones.

              Another reason I want to get a good backup scheme going is because my server is running an old version of Ubuntu (10.04) and I need to upgrade it, and I would like to back the thing up before I do this, in case something happens. Plus I can then switch from mdadm to ZFS which would require rebuilding the array from scratch and then restoring all my files.

              Comment

              • Chris D
                Moderator Emeritus
                • Dec 2000
                • 16877

                #8
                Kevin got a raise? Whoa... finally, some appreciation for your talents!! :T

                I'm running out of room on my Drobo, currently with 2 TB, have been looking at upgrading drives. Would be great to get whole new NAS, but no $$.
                CHRIS

                Well, we're safe for now. Thank goodness we're in a bowling alley.
                - Pleasantville

                Comment

                • knowledgebass
                  Senior Member
                  • May 2013
                  • 159

                  #9
                  Kevin, you may find the program rsnapshot to your liking. It's a backup script front-end that uses rsync for the sync part. It's meant for doing snapshot style backups, only copies new or changed files, and sure beats the series of scripts I was using before!

                  Comment

                  • aud19
                    Twin Moderator Emeritus
                    • Aug 2003
                    • 16706

                    #10
                    Well I'm actually in the process of building a new NAS right now so good timing Kev (and congrats on the raise!).

                    I'm building what will currently be a 4TB server (3X2TB) unRAID server and planning on backing it up via USB enclosures as you mentioned (though I only have 2X1TB drives so it will be important data only for now). The USB enclosures will be kept in our safe deposit box at the bank. (I'm also contemplating throwing a couple spare 500GB drives in the NAS and duplicating important files on them as well in case of drive failure.) As time goes on I'll replace and/or add 4TB (or 6TB) drives and use the current 2TB units for backup.

                    Really the most important things are our photos (especially of our kids) and some smaller documents, recipes etc and our music collection (so big it would be daunting to replicate). That shouldn't really ever take more than a couple TB (I think all of that together is ~ 700GB now). All of our backed up movies etc I'd have little qualms simply pirating to replace (*gasp*, I know :roll: ) if the worst happened.

                    But that's probably your best bet. A nice NAS and some external backups to be stored off site. Short of building duplicate NAS units and keeping one in an out building in a safe or something with a Cat6 line run out to it anyway :lol:

                    If you want input on a nice small, powerful but efficient setup, here's the hardware I used:

                    Discussion of HTPC systems and products, DVD drives, system configuration issues, personal computing and gaming systems of all kinds including internet applications, other peripheral devices, and game consoles.

                    -LIAN LI PC-Q25B Case
                    -SILVERSTONE SFX ST30SF 300W PSU
                    -ASUS H87I-PLUS Motherboard
                    -INTEL I3 4150
                    -SanDisk Cruzer Fit 16GB USB 2.0 Flash Drive

                    Also just ordered this fella for backups:


                    Jason

                    Comment

                    • Hdale85
                      Moderator Emeritus
                      • Jan 2006
                      • 16075

                      #11
                      Kevin, in ZFS generally you can have a lot more redundant drives than 2 if you want. You could have multiple pools of say 6 drives that each have 2 redundant drives, but then you could also have as many hot spares as you wanted as well. So those would swap in automatically if a drive failed. Granted none of this would save you from fire or flood.

                      Comment

                      • Chris D
                        Moderator Emeritus
                        • Dec 2000
                        • 16877

                        #12
                        Jason, wasn't Inatech one of the company names in the movie "Office Space"?

                        I'm looking more and more at NAS's, finding a different option than Drobo. One possibility is Synology:


                        Another is a Buffalo NAS:
                        CHRIS

                        Well, we're safe for now. Thank goodness we're in a bowling alley.
                        - Pleasantville

                        Comment

                        • Kevin P
                          Member
                          • Aug 2000
                          • 10809

                          #13
                          It was spelled "Initech" in the movie.

                          I need a new backup solution for Windoze too. I've used Acronis for the past few years but the 2014 version is causing issues on my lappy. I avoid Symantec like the plague ever since they ruined the Norton products, so Ghost is out. What else is out there that simply works without adding unneeded bloat?

                          Comment

                          • aud19
                            Twin Moderator Emeritus
                            • Aug 2003
                            • 16706

                            #14
                            Originally posted by Chris D
                            Jason, wasn't Inatech one of the company names in the movie "Office Space"?

                            I'm looking more and more at NAS's, finding a different option than Drobo. One possibility is Synology:


                            Another is a Buffalo NAS:
                            http://www.amazon.com/Buffalo-LinkSt...ds=buffalo+nas
                            Why not build your own Chris? Those pre-built units are nice and all and that's where I also started (a bre-built QNAP NAS) but really, building your own is better bang for your buck (better hardware for ~the same $) and generally far more expandable. Plus then you can pick whichever NAS OS you want and tailor it to your needs. Try and think of it as buying a HTIB when you darn well know you're going to end up with separates and tower speakers :P

                            Originally posted by Kevin P
                            It was spelled "Initech" in the movie.

                            I need a new backup solution for Windoze too. I've used Acronis for the past few years but the 2014 version is causing issues on my lappy. I avoid Symantec like the plague ever since they ruined the Norton products, so Ghost is out. What else is out there that simply works without adding unneeded bloat?
                            I've been looking in to CrashPlan for my unRAID server but it also runs on MS, MAC and Linux as well Kevin

                            Jason

                            Comment

                            • Chris D
                              Moderator Emeritus
                              • Dec 2000
                              • 16877

                              #15
                              Originally posted by aud19
                              Why not build your own Chris? Those pre-built units are nice and all and that's where I also started (a bre-built QNAP NAS) but really, building your own is better bang for your buck (better hardware for ~the same $) and generally far more expandable. Plus then you can pick whichever NAS OS you want and tailor it to your needs. Try and think of it as buying a HTIB when you darn well know you're going to end up with separates and tower speakers :P
                              Well, really the reason would be I don't know how to do it. But if you guys teach me, I will! It would have to be fast on the network, and devices would have to be able to play movies and music stored on it, particularly the iTunes library.
                              CHRIS

                              Well, we're safe for now. Thank goodness we're in a bowling alley.
                              - Pleasantville

                              Comment

                              • aud19
                                Twin Moderator Emeritus
                                • Aug 2003
                                • 16706

                                #16
                                It's really just like building a simple computer with a bunch of hard drives in it. Case, power supply, motherboard, CPU and RAM. That's about it.

                                Figuring out which OS to use for it is probably the hardest part. For example something like FreeNAS has many benefits (like ZFS which is generally agreed to be one of the better file systems and that it's free) but tends to be a little pickier about hardware (prefers expensive ECC RAM, have to have identical hard drives etc). Whereas unRAID is less picky about hardware and the ability to use whichever hard drives you want was a big seller for me as I have two 2TB drives of one model, another 2TB drive of another model, a couple 500GB drives and in the future I'll be upgrading to 4TB or 6TB drives. unRAID is the only OS I'm aware of that can pool that mish-mash of drives together as one networked drive with parity for redundancy.

                                That's where most of my research has gone so far beyond making sure I got hardware that was friendly to the OS.
                                Jason

                                Comment

                                • aud19
                                  Twin Moderator Emeritus
                                  • Aug 2003
                                  • 16706

                                  #17
                                  Kevin, you decide on anything yet?
                                  Jason

                                  Comment

                                  • Kevin P
                                    Member
                                    • Aug 2000
                                    • 10809

                                    #18
                                    Not yet Jason, but I'm leaning toward rsyncing to 4 TB external drives. For Windoze, using robocopy to back up to the file server, and then that in turn gets rsynced to the external drive.

                                    Any recommendations for reliable external 4TB+ drives?

                                    Comment

                                    • aud19
                                      Twin Moderator Emeritus
                                      • Aug 2003
                                      • 16706

                                      #19
                                      Originally posted by Kevin P
                                      Not yet Jason, but I'm leaning toward rsyncing to 4 TB external drives. For Windoze, using robocopy to back up to the file server, and then that in turn gets rsynced to the external drive.

                                      Any recommendations for reliable external 4TB+ drives?




                                      Just bought one myself today


                                      Have you figured out what you're doing for for the file server?
                                      Jason

                                      Comment

                                      • Kevin P
                                        Member
                                        • Aug 2000
                                        • 10809

                                        #20
                                        That's just a bare drive, though it could be installed in an enclosure. Guess I need to find a decent enclosure that supports USB 3.0 and a 4 TB drive. I figured just get one ready to go with drive(s) already installed.

                                        Workstations will get backed up to the server, and the server will get backed up to the external drive (using rsync). Then I'll swap external drives weekly or so and keep one offsite.

                                        Comment

                                        • aud19
                                          Twin Moderator Emeritus
                                          • Aug 2003
                                          • 16706

                                          #21
                                          Originally posted by Kevin P
                                          That's just a bare drive, though it could be installed in an enclosure. Guess I need to find a decent enclosure that supports USB 3.0 and a 4 TB drive. I figured just get one ready to go with drive(s) already installed.

                                          Workstations will get backed up to the server, and the server will get backed up to the external drive (using rsync). Then I'll swap external drives weekly or so and keep one offsite.
                                          I posted a 2-bay (they also make a 1-bay) USB 3.0 docking station earlier in this thread:
                                          Discussion of HTPC systems and products, DVD drives, system configuration issues, personal computing and gaming systems of all kinds including internet applications, other peripheral devices, and game consoles.


                                          But there's a TON of HD enclosures that are just enclosures.



                                          I prefer those to the "all in one" drive/enclosures as they allow you to swap multiple drives in and out, they let YOU spec whatever hard drives you want and upgrade them whenever you want etc. Just far more versatile/flexible.

                                          As for the file server, I was asking if you decided on what you're building or if you were just sticking with your existing unit for now?
                                          Jason

                                          Comment

                                          Working...
                                          😀
                                          😂
                                          🥰
                                          😘
                                          🤢
                                          😎
                                          😞
                                          😡
                                          👍
                                          👎
                                          Searching...Please wait.
                                          An unexpected error was returned: 'Your submission could not be processed because you have logged in since the previous page was loaded.

                                          Please push the back button and reload the previous window.'
                                          An unexpected error was returned: 'Your submission could not be processed because the token has expired.

                                          Please push the back button and reload the previous window.'
                                          An internal error has occurred and the module cannot be displayed.
                                          There are no results that meet this criteria.
                                          Search Result for "|||"