No announcement yet.

Media server capable of 20+ TB's

This topic is closed.
  • Filter
  • Time
  • Show
Clear All
new posts

  • Media server capable of 20+ TB's

    Ok so there has been a lot of interest in the media server I mentioned that I was working on. So for now here is the parts list and a bit of info about how I'm going to approach the whole thing.

    The motherboard, desktop level motherboards are quite capable these days as well as their processors. So I feel that in a media server sort of setup where it will be accessed by anything less then probably 20 computers you don't need to go crazy and spend big bucks on server style components. Now sometimes there are some good deals on the server stuff so sometimes its an option but usually just getting a good desktop motherboard and CPU is going to be cheaper.

    I picked this motherboard because it has several PCIE expansion slots as well as onboard video. Depending on how you plan on doing your raid you may not want a video card taking up an expansion slot in your server. I also picked this board because it has 6 onboard SATA ports which allows me to use my hard drives in software raid and gives me time to decide if I like software raid and if I want to just buy a few cheap SATA PCIE cards that will allow me to expand my software array or spend some big bucks and grab a nice Areca or similar 24 port card.

    Next up in my list is the case, now generall hot swap casses are very expensive. There has been a lot of buzz about a particular case though because of its insanely low price when it comes to a large hot swap case. This case is capable of handeling 20 hot swap drives and I believe it has space for 4 more internal drives. It also has the SATA/SAS backplane that has all the power connections on there so your not looking for a PSU that has a million connections. There have been a few complaints about the hot swap trays having to be wiggled around a bit to make them go in all the way. For the 1k or more in savings its something I'm willing to deal with. Also since its a home setting chances of me pulling hard drives in and out very often are slim.

    The power supply I picked out I got bundled with my CPU so it was cheaper. It's a 500 watt power supply which is probably more then enough for 20 drives. Someone on AVS is running a 48 drive array and recently measured that he's drawing something like 400 watts. You could go crazy on the PSU and it's not really a bad thing but you don't have to. Just clean and efficient in the 500-750 watt range is probably adequate.

    Buy PC Power and Cooling Silencer PPCS500 500 W ATX12V / EPS12V SLI Ready CrossFire Ready 80 PLUS Certified Active PFC Power Supply compatible with core i7 with fast shipping and top-rated customer service. Once you know, you Newegg!

    The processor, now depending on what your doing this may or may not make much of a difference. Some people do other things with ther server other then simply serving files. So if there are some other things you do with the server you may end up looking for more power. Also software raid, if you intend to use it your going to want a better CPU then you'd need if your doing hardware raid. I'm not sure of everything I'll do with my server. Also for the time being I'm doing software raid. The AMD Phenom CPU's are quite cheap so I just went with a quad core Phenom to serve the needs of whatever I end up doing.

    Memory is just stanard Crucial DDR2 2gb's for now. Not sure if I'd need more then this? If I run linux like I'm leaning towards it's probably more then enough.

    Ok now the hard drives. I decided to go with 1TB drives. For starters they are quite cheap and going to get cheaper. My server is capable of taking 24 drives total for now I just got 6. This way as drives get cheaper and as I need more space I can buy more and slide them in. There has been a lot of discussion about drives lately. Realistically the 1.5 TB drives are the best buy as far as cost/space. The problem is there is only one out there and they have been plagued with issues since release. There was a new firmware for them that came out a week or so ago and people have been saying it's fixed. I don't trust them yet with my data so for now I'm staying away.

    Ok now onto my OS drive. There was a conflict here. I didn't want to take up space that another storage drive could be taking up potientially. A lot of people have been using USB flash drives on their servers for their OS's and this sounded like an awesome idea and it's the path I chose.

    I will post up some pics and what not as everythign arrives. So far all I have are the CPU, memory, 8gb flash drive, and PSU. The rest should arrive tomorrow.

  • #2
    Wow, would you want to put your os which is intensive onto a flash drive which has a large tendency to fail? I mean I know anything can fail...but using a USB flash drive... eh, I've had 2x the amount fail as physical HDs.

    Just make sure you back up your system a lot incase the flash drive fails!

    And get a good tape backup system to back the whole thing up! ha ha. :T

    But that looks like it's going to be a really good server!!!
    Digital Audio makes me Happy.


    • #3

      A couple of thoughts:

      If you're going to put the OS on a USB flash drive, you should use a high-speed one like this one: You'll notice much faster boot/loading times.

      For the prior poster who is concerned about flash drive failure: the OS is relatively static, back it up once (even copy it to another flash drive) and if it dies, just pop another one in and you're back in business. Keep the important data on the RAID array and you're golden. That said, I wouldn't put any Windows on a flash drive, but there's ways to put Linux on a flash and optimize it to minimize writes to the flash drive (which they have a finite number-of-writes lifespan).

      2 GB RAM should be enough for Linux without a lot of other things running. If you go 4GB or higher and use a 64-bit distro, you could forego the swap partition for even more speed. You could even load the flash drive's contents into a RAM disk and then the OS will scream! Also, more RAM means more space for disk cache, improving performance.

      The quad-core CPU is probably overkill for even a large RAID setup. Maybe look at one of the 45-watt Athlon 64 dual cores, they'll run a lot cooler and draw less power. I'm going to be building a media server myself sometime and will probably go with that CPU, unless I decide to make it a screaming VM box in which case I might spring for the quad core.

      How do you plan to back up 20 TB of data?


      • #4
        Interesting project man. A USB OS drive sounds interesting too. With RAM as cheap as it is going to 4 wouldn't add too much cost. Which PSU did you get? The link is the same as the same as the case.


        • #5
          I fixed the link for the PSU. It's a PC Power and Cooling 500 watt that is 80%+ efficient.

          As for the USB flash drive yeah I noticed it wasn't one of the highspeed units after it had shipped. I may try it and if I find its to slow I'll just order another one and use this one for whatever. Maybe backup the OS onto it or something?

          Backing up 20TB's of drives haha. Well the guy with the 48TB server built another server to backup to. Me personally I don't want to lose my data but if I do it's all media stuff and not the end of the world. Running a raid 6 array and being able to lose 2 drives at the same time I feel is enough security for my needs. Some may not feel the same way and I understand everyone's needs and wants are different.

          I ordered extra ram for my desktop and I've been considering just sticking it in the server instead. I have a stick of 1GB laying around but then it wouldn't be running in dual channel so that's why it hasn't gotten used.

          Ah almost forgot. I have not experienced any issues with failing USB flash drives. I actually have a 2GB PNY Attache flash drive that I bought something like 4-5 years ago for 200 bucks when they first came out. I have not experienced any loss or corruption of data on it and it still works great.


          • #6
            You can never have too many flash drives. Use it as a backup for the OS, or just put your pr0n collection on it and keep it on your keychain.

            RAID isn't a replacement for backups, of course. You're protected from a hard drive failure (or 2, for RAID 6), but not from accidentally deleting files, or files getting corrupted for whatever reason, a virus overwriting stuff, etc. etc. I'd organize the data in such a way that the stuff that changes often/needs to be backed up can be kept in one place (folder, partition, etc.), and you'd back just that up regularly. You'll probably find that that data is minimal in size, and could just be backed up to an external drive. Static data like DVD/BD rips can probably be left unbacked up since you can just re-rip from your discs. DVRed TV shows are at your discretion... don't bother backing up the watch-once-and-delete shows, but if you're keeping any long-term, back those up.

            In fact, you might find that you have 5 categories of data:

            1. Important data that changes regularly, such as personal finances, email, etc. That's the stuff you'd back up frequently.
            2. Important data that tends to be static, but new files are added regularly - such as photos or downloaded music, or DVRd shows you want to save - just back up new/changed files, how often depending on how much the data changes.
            3. Archival data such as old photo collections, that don't change but you don't want to lose - make a couple backups, keep one offsite. This sort of thing could be burned to DVD or BDs.
            4. Data that can be easily recreated, such as rips from your physical media - no need to back up at all, unless you want to save time re-ripping. An offsite backup of this data could be handy in case of a disaster (e.g. fire or theft) that destroys your server and your media collection.
            5. Transient data that comes and goes, such as watch-and-delete DVR shows. You can get away with not backing this up.

            I have some plans for a media server too, so I'll be posting a "competing" thread on this, as I have some different ideas.


            • #7
              Love to see your idea's And who knows I may adapt some of yours. I have a solid foundation I think that I can build off of though. I figure even a slower flash drive is faster then an IDE drive. Which is what I have laying around that would end up being used. But certainly interested in how your setup will be


              • #8
                USB is slower than IDE though, and especially SATA. On the other hand, flash memory is generally faster than physical hard drives.

                Here's another idea to consider when you've added a few more drives to your box. Split them up into more than one array. A RAID 5 with a lot of drives has an increased chance of issues due to drives failing (the more drives you have, the better the chances of one or more failing), especially if you lose 2, you're screwed unless you use RAID 6. Plus, segregating the data can help with backups like I mentioned previously. And, it's easier to create another array than to expand an existing one. By the time you're ready to add more capacity, you might decide that 2 TB or larger drives are cheap enough and you could build an array with those, separate from the array of 1 TB drives you have now.

                Another thing you can do is designate 1 or 2 drives as hot spares. If you have more than one array, they can share the hot spares. Then, if a drive fails, the affected array will "grab" the hot spare and rebuild itself onto it. Then, at your convenience, replace the failed drive, and have the replacement be the new hot spare.


                • #9
                  The problem I see with more then one array is partially that it wouldn't all be shown as one big space right? If you run multiple arrays then doesn't the amount of space your losing for redundancy go up? Or does it work out to be the same?


                  • #10
                    Kevin, I have a 3800+ X2 in my desktop, do you think it would be safe to stick that in the server? I've been considering putting the Phenom in the desktop lol. Kind of silly to have an HTPC and server that have a better CPU then my desktop :B


                    • #11
                      Just to use my setup as an example, I have a 2TB server and a 1TB Western Digital "MyBook" for backup. I wrote a command line app that will check specific folders every night and copy over any files that have been modified. My documents, source code, music, and pictures (raw pictures only) all get copied over. Movie/TV rips and such do not get backed up. Works great, in a fire or something I could just grab the mybook and go and not lose anything important.


                      • #12
                        Yeah, even my music is getting rather large though lol. Something like 350gb's at the moment I believe. But even so a 1TB drive would probably be enough to keep up for at least a little while.


                        • #13
                          Originally posted by Dougie085
                          The problem I see with more then one array is partially that it wouldn't all be shown as one big space right? If you run multiple arrays then doesn't the amount of space your losing for redundancy go up? Or does it work out to be the same?
                          Assuming RAID 5 arrays, each array would lose 1 drive's worth of capacity for redundancy, or 2 for RAID 6. But, with a large number of drives, you'll want extra redundancy anyway. Where multiple arrays is better is if you have multiple RAID 5 arrays plus a shared hot spare vs. multiple RAID 6 arrays.

                          For example, taking 10 1 TB drives:
                          1 big RAID 5 = 9 TB + 1 for parity. Only 1 drive can fail without taking the array out.
                          1 big RAID 6 = 8 TB + 2 for parity. Up to 2 drives can fail without taking the array out.
                          Two equal sized RAID 5s: 4 TB + 4 TB + 1 parity per array. One drive can fail per array without data loss. Two drives failing in one array would cause loss of that array, but the other would remain.
                          Two equal sized RAID 6es: 3 TB + 3 TB + 2 parity per array. Plenty of redundancy, but only 6 TB out of 10 available.
                          Now, let's add one more drive, and make it a hot spare, and go with 2 RAID 5 arrays.
                          4 TB + 4 TB + 1 parity per array + 1 hot spare. Now, if a drive fails in either array,it will be rebuilt onto the spare. Once the rebuild completes, either array can then tolerate one more drive failure. So, two RAID 5 + 1 hot spare gives you (almost) the same redundancy as two RAID 6 but with 1 less drive.

                          But, going with multiple arrays depends on how you plan to organize your data. One big array might be better if you're ripping a lot of BluRay or something. But you should consider at least one hot spare, when you add some more drives.

                          I may organize my server into more than one array (or partitions within an array). Put the often-changed files in one array/partition, and archival stuff on another. Then I can backup each partition/array on different schedules.

                          Originally posted by Dougie085
                          Kevin, I have a 3800+ X2 in my desktop, do you think it would be safe to stick that in the server? I've been considering putting the Phenom in the desktop lol. Kind of silly to have an HTPC and server that have a better CPU then my desktop :B
                          That would work, as long as both chips/mobos are the same socket type. (Socket AM2/AM2+ are the current trend for AMD).
                          Originally posted by impala454
                          Works great, in a fire or something I could just grab the mybook and go and not lose anything important.
                          But what if you aren't home when the fire breaks out? (Speaking from experience here...)

                          Oh yeah, forgot to mention. If you do go with more than one array, you can use LVM (Logical Volume Management) to make the multiple arrays look like one big logical drive. This makes expansion easier. For example, say you start with an array of 1TB drives. You RAID them and then put a LVM partition on it, and then put your file system on that. Down the road, that volume is getting full and you want to add more space, but by then 2 TB drives are cheap and readily available, so you buy a bunch of those. Make those into an array and then expand the LVM and filesystem onto it. Now you have 2 arrays, acting as 1 big volume, and that also eliminates the issue of the different drive sizes in 1 array. A couple years later, you buy up a bunch of 4 TB drives, you can make another array with those, and so forth and so on. Well, by then, your old 1 TB drives are starting to die, so you copy everything over to the new array, and retire the old drives. Repeat ad infinitum...


                          • #14
                            Originally posted by Kevin P
                            But what if you aren't home when the fire breaks out? (Speaking from experience here...)
                            The stuff I care about most (photos, source code, documents) still takes up less than a DVD worth of data, so every couple months or so I burn them and stick the disc in my fireproof safe. I also occasionally copy my source code folder & a few documents up to my web server.


                            • #15
                              I guess I'm anti USB flash drives for important data since I send back so many every day. We've got a fail ratio of 1:8 and almost every one we've sent out has, at some point, come back to be replaced. Some last longer than others, but most fail within 1 to 2 years of heavy use...which all of our people use. Biggest thing that kills them is dust in the USB port...which you wouldn't have here keeping it plugged in all the time.

                              I do however, love all the solid state drives we use. They definitely show a nice performance increase over mechanical drives. But I'm still leery on them eventually failing. That's why I only use it as my OS drive and regular drives in my raid 6 config for my 4TB total on my server, but that's almost full so I'll have to be expanding.

                              That's why I use Veritas with a tape backup to backup the server and all PCs on my network.
                              Digital Audio makes me Happy.


                              • #16
                                Yes both CPU's are AM2. All 3 computers in my house are AM2/AM2+ except the notebook haha. Anyways I think that's what I'll do. The 3800+ should be good for the server.


                                • #17
                                  It should work perfectly don't need a huge CPU just to send data to the network.

                                  LOL my 2008 server is only a P4 3.2GHz cpu. Works perfectly fine for what it is.
                                  Digital Audio makes me Happy.


                                  • #18
                                    CPU isn't important if you just have a "dumb" file server, but make sure you keep in mind anything you might want to do to the files on the server. I.e. say you decide to re-encode a large amount of files or something. Just something to keep in mind.


                                    • #19
                                      Well got it all put together and what not. Instead of putting 6gb of ram in the desktop I put 4gb in the server and 4 in the desktop. I'm installing Ubuntu Server 64bit as we speak. See how this goes


                                      • #20
                                        Anyone got AIM or something and could help me out with samba? I've got Webmin installed which is a nice browser interface admin tool. I'm trying to setup Samba. I don't want it to require a password but it's not letting me browse the files at all. I can see the Media-Server on the network from my other computers but when I try and go into it it just says I don't have permission or what not.


                                        • #21
                                          Ah nevermind! I got it now

                                          Ok now I got it to where I can browse the files but I can't write to it. I set Samba to allow all users to write. The array has write protections on the server it self so maybe that's why?


                                          • #22
                                            If the server can't write to the array (due to mounting read-only or file permissions) Samba won't be able to either.

                                            If you want to talk to me on AIM, send me a PM with your screen name so I can add you to my buddy list. EDIT: never mind, I see you have it in your profile. I'll add you to my list now. If you want to IM me, my screen name is in my profile.


                                            • #23
                                              Looks like I got it working. I just changed the owner to nobody which is what I have Samba set to forced user. It seems to be working pretty quickly. Takes like 2 minutes to copy a 4.4gb file? I guess that's pretty quick lol.

                                              I setup VNC on it as well so I can just control it from my desktop.


                                              • #24
                                                That's pretty good speed, especially for Samba (or SMB, the underlying protocol, which, true to Mickeysoft form, is very inefficient).

                                                If you have other Linux boxes and you want to share between them, NFS is a lot faster. But for Winders you'll need to use Samba.


                                                • #25
                                                  Yeah my other boxes are all windows for now. I just have the HTPC, Desktop, and a notebook. Since I'm using a switch I need to figure out how to get my wireless router clients to show up on the same network. My friend is coming over today to help me tinker with it so we'll see how that goes.


                                                  • #26
                                                    One thing I'm going to say is I was actually surprised how easy it was to setup Samba and MDADM. It's all rather straight forward I'm not even sure I could have done it more quickly in a windows server setup. Even to add extra drives and what not it will be super easy in linux.


                                                    • #27
                                                      How is your network set up? Assuming the wireless router provides your Internet connection, just connect a port on the switch to one of the LAN ports on the router. If either device has an "uplink" port, use that.

                                                      Or, if you have enough LAN ports on the router, those act as a switch, and then you wouldn't need the other switch at all, unless it's in a different location or you need more ports, or want gigabit.


                                                      • #28
                                                        I have a Dell 16 port gigabit switch. I have 6 network connections. The wireless router has only 4. I have the wireless router connected to the switch via the routers WAN port I think they call it. So the computers on the wireless network (which currently is just the notebook) do not show up on the same network as my desktop and HTPC. I've been wondering if I just connect it to a lan port on the back of the wireless router if it would work then.


                                                        • #29
                                                          Yes, connect the switch to one of the LAN ports on the router. Just make sure to disable the DHCP server on the router to avoid conflicts.

                                                          Alternatively, you can connect your DSL modem to the router's WAN port, then connect a LAN port on the router to the switch. Then connect your computers to either the router or the switch (connect gigabit-capable machines to the switch). But, if your DSL modem has its own router/firewall, you could use the first method above, if you disable DHCP on the wireless router.


                                                          • #30
                                                            The DSL modem doesn't have router capabilities it's just a straight through DSL modem. So it should work fine connected to the wireless router and then to the switch through one of the lan ports.

                                                            On another note Kevin, disabling the swap seemed to only fix it for a little bit. It started acting up again but not as often so I'm going to pull the flash drive and install a regular hard drive or use the USB laptop drive I have.


                                                            • #31
                                                              Ok well I finally got the server up and going. I had quite a few instances where I ended up just formating and starting from scratch. Some was stupid mistakes on my part others was bad advice from someone on a different forum. Either way it seems to be running much better then previously now! It runs great really extremely fast and very easily serves up even the most demanding Blu-ray movies over the network. I snapped a couple pics. I didn't get as many of the server not sure why? I guess it's nothing to terribly exciting.

                                                              I will say that the cables are all bundled up and nice now. I also taped up the holes under the hard drive fans so that it pulls more air through the drive bays. This is something a lot of people have been doing.

                                                              You can also see my modest little network setup. The Dlink box is the wireless router. It does 108 super g or whatever they call it. Works very well and pretty quick. Certainly not quick enough for 1080p streaming as I've been there tried that. Then the DSL modem is on top of it. Next to that is my Dell 2716 Managed gigabit switch that I snagged on ebay for about 110 bucks shipped. Excellent switch extremely fast and very well built. The antenna for my wireless is out in the hall way. Basically goes up through the ceiling panels and it has magnets on it so it just sits out in the hallway. For some reason my wifes laptop is very sensative to the signal? It's not very far away so I need to look into that soon.


                                                              • #32
                                                                Sweet work man. Did you redo the network yet or??

                                                                I hope to have space to put use to my Cisco Fiber switch... ;x(

                                                                MMM 100Base-TX dual Mode fiber, should have zero issue streaming HD content over a network!


                                                                • #33
                                                                  Dude if you want to tame th level of noise in that bad boy here is what ya need!


                                                                  Lose the 60mm fans and replace them with this, it will pull a lot hotair out and it will be very quiet

                                                                  Even with 20HDD's 3 of these can move enough air to cool them


                                                                  Searching...Please wait.
                                                                  An unexpected error was returned: 'Your submission could not be processed because you have logged in since the previous page was loaded.

                                                                  Please push the back button and reload the previous window.'
                                                                  An unexpected error was returned: 'Your submission could not be processed because the token has expired.

                                                                  Please push the back button and reload the previous window.'
                                                                  An internal error has occurred and the module cannot be displayed.
                                                                  There are no results that meet this criteria.
                                                                  Search Result for "|||"