Show disks in a system as one and make a data (not disk) parity… Is it so hard?

OK people that have read my previous posts about my quest for the «perfect» home server, might have noticed that I consider a priority (a) to be able to unify the space of various disks (various sizes, makes, filesystems, buses) into one large bucket and (b) to have some data redundancy, without utilizing mirror techniques that would in fact half my storage capacity.

Well it seems both those things are tough nuts to crack! I wonder why.

Here is a summary of what I know or have tried on both those Fronts. Note that the article is Windows oriented unless otherwise specified.

Well at first I tried unRAID. I won’t spend much time on this as I have elaborate articles about the pluses and minuses of this system. It is Linux based (Slackware), rather closed (although finally it slowly shifts away from this philosophy), commercial and works – but just that. It gives you both my requirements:

  1. Disks, all together make one contiguous space and unRAID manages how to fill them up.
  2. System needs your larger (or same size as the larger) disk to build a parity of all the rest (whatever their size up to the size of the parity).

Good system, «just works». Why I took it out? Briefly:

  1. The author never really got what community presence is, never really got what «keeping current» is, never really got how «commercial software» ties the company.
  2. It was Linux. I could expand it to do other «server» things only by hacking things (and then braking the hack with every new build and trying to find another).

So I decided to move to Windows.
Window on the first part of the equation (JBOD) has various solutions:

Mapping drives instead of letters, to subfolders of a single disk (instead of mapping to a letter). This has been Windows functionality for quite some time. In fact it is the default for other OSes. Never really caught up in Windows. Unfortunately it doesn’t give you real «single» disk functionality as those mapped «folders» can actually be up to the size of the disk they represent.

Hard-links. Rather advanced function of the OS, again not really exploited in Windows (and very exploited in *x systems). Can lead to interesting results but needs very very complex administration as there is no automated system to handle the creation and change of hard-links to simulate properly JBOD-like functionality.

Dynamic disks. Dynamic volumes can extend over disks. Unfortunately if something happens, you die. As simple as that. No real tools to help you, disks cannot be seen if moved to another system (esp. separately) etc. Just seen some hacks and some commercial software from mysterious companies handle those kinds of disks. Last but not least, any possible high-layer (file/folder) parity solution would need to see distinct volumes too. So dynamic disks mess this thing up too.

Disk Extender version 1 (aka Windows Home Server). Well this was my second attempt. Seems it will be more sort lived than my (have to say) problem-free life with unRAID. Why? Let see first briefly what it gives you:

  1. Windows (and in fact Server 2003, same kernel as XP, the most compatible kernel of all). This means I can run other applications and services! …and I did, as a 24/7 system in a house really has to justify the electricity bill.
  2. Is normally easily manageable using a special remote console. In practice a more advanced user will need normal desktop (over RDP etc.).
  3. It gives you «JBOD» but uses a complex and NOT bullet-proof method («tombstones») for disks. It is a combination drive mapping and hard-links, both managed by the OS. At least it is (barely and by «hack») compatible with 3rd party parity solutions.
  4. Also gives you data redundancy but only with the not very smart method of mirroring specific folders. So if you want all your data safe, you half your storage.

Unfortunately WHS has MANY problems and MANY limitations. Currently my system is dying. Just like that for no reason. Services fail and crash etc. No data corruption (yet?) at least. Also the way the system is built doesn’t allow for fully utilizing the Windows core, nor allows for REAL recovery in case of serious failure. This is why it is going out the… erm… window. 🙂

Disk Extender version 2 (aka Aurora and Veil). Well last two years saw serious changes in the Windows Home Server and Windows Small Business Server (and Windows Essential Business Server). EBS was killed. WHS and SBS teams joined to one (HSBS team or something). This brought some fresh air in the form of Aurora (the new junior version of SBS which in a large part is about «the cloud») and Veil (the new WHS). Both of them share the same kernel (64bit Win 2k8R2), the same administration suite and very very very large part of the code. We can say they are the same product with a few «switches» on and off depending on the market targeted. This is not an article about them though; the part we care is disk extender. Well seems it got better in a way. It’s now done in lower level (something like dynamic disks), so it kills the compatibility with 3rd party parity solutions and «gains» all the disadvantages (in fact worse because it’s fully «custom» as it is) of dynamic disks as they cannot be read by ANY other system and right now NO recovery utilities exist. No thanks.

Those are the non-hardware solutions for «JBOD» kind of disk handling. In fact there are a couple more, but since they are 3rd party and more interconnected to error recovery too, we’ll mention them separately (for all their functionality) below.

First of all we have FlexRAID. This was my choice along with WHS. In theory (and I am sure in practice for many of us), it will build a snapshot parity of your data. Being «snapshot» (i.e. point in time, whenever you ask the system to do it), means it is great for static data but not very good for data that change frequently (video edits etc.). In my theory I would use the DE part of WHS and the parity part of FlexRAID. Well we talked about how good WHS was to me. FlexRAID was also not very good to me. Parity creation is not optimized for my spread of disks and controllers and in fact could take in my tests from half a day to two days! Same with rebuild. I saw that this won’t work. Potentially it could work but to be honest, the author doesn’t really seem to have taken his project very seriously. In the past he has scr**ed his users by disappearing for months, his last beta expiring. After (quite) some time, he resurfaced with a build from scratch and seemingly more dedication. This is where I went in (I already knew the past events but said «what a heck, he probably knows better now). Well now, I am not sure. Lately he also disappeared for months but now resurfaced just a few days before the expiration of his beta (we see a louse pattern?). A new beta is out now. I have to internally fight to trust this project. We’ll see. It shows promise as they author had said about a «live» version (not snapshot based) which could be the solution to my problems and has already developed a single view of disks I define, giving me the «JBOD» feeling I want (or close), although still with many bugs. It is still the solution I will be… forced to use if I move to SBS7 (as is my plan).

DisParity.Well DisParity hasn’t gone far yet. It’s built on the same idea of FlexRAID snapshot, but hasn’t been in development very actively. The interesting part is that DisParity was built as reaction by an angry user of FlexRAID that «suffered» from the first incident described above. As I said it hasn’t seen too much development. That doesn’t mean it doesn’t work.

Liquesce. Well this a fairly new project. Right now it concentrates on the JBOD part, but maybe in the future (possibly as separate project) it will move to the field of parity based redundancy. I giving this a good look (and suggest everyone else to that too) as the developer is doing a very organized job, in fact more organized that ALL the guys above and this project has the potential to go FAR. Go help him!

Well in my quest for the Grail of home data storage, this is what I’ve discovered. What would I do if I started today?

  • If you don’t care about anything else than a machine that you throw disks on, it «merges» them and keeps the data in there secured AND don’t mind a lazy developer, I suggest unRAID.
  • If you like an «add-on» thing more than a closed «native» thing and most of your data are «static» (which is usually the case for most of us), then I would suggest a Windows system along with a 3rd party solution.
    • WHS? Well it is the most «compatible» with 3rd party parity solutions while giving you the JBOD part of the equation, but really… I cannot suggest it today.
    • Veil? Still in its «preview» form, WAY more modern than WHS, but also way less developed. You will have a problem with parity solutions though (as the disks don’t show separately).
    • Aurora? Same thing in SBS costume (eg. if you don’t want -some- media capabilities and want domain users).
    • SBS2008. Only for the advanced (and with access to the license). Will give you many services, will leave you choose (erm, I mean find) your own solution for the JBOD AND redundancy problems.
    • SBS7. Well same thing way more modern WHEN it comes out. I am waiting for it as this is what I’ve decided to move to.
    • XP or 7 (not Vista). Why not? Keep things simple and go match your own JBOD and redundancy solution.
    • (yes not Windows) Linux and FlexRAID (as it’s the only one ported to Linux). Well you might want that mix.
  • FlexRAID if you feel you might get lucky. Still don’t bet too much on the developer too (it’s like unRAID’s Tom that doesn’t even have to pretend because the software is not commercial). 🙂 Still it’s way the more developed solution after unRAID. If it fits you.
  • If you want to play with smaller players, try DisParity. There are already (or «just» – depending how you see things) a couple of successful data recovery cases described in the forum, but things pretty much end there.
  • Liquesce. You should visit codeplex for Liquesce just because of the potential (at least for the JBOD part for now). We should help nicely organized and with the appropriate passion attempts.

Now in the end what I need? Let me make a nice «wishlist» of what I would like to see both for the JBOD and redundancy problems I have.

  1. Parity should be live. Snapshot solutions have some use, but limited if you ask me.
  2. Allow for different parity engines. Only FlexRAID has done some work on this yet.
  3. Configuration (both of parity and JBOD) with multiple backups (on system folders, on all data disks etc.) and the system to automatically search all those possible locations in case of recovery.
  4. Since I mentioned configuration. Web is number one nowadays. GUI second, console only for the geeks. Of course all methods should talk (read/write) to the same configuration files (duh!).
  5. Rich and user friendly reporting system. Not just logs, but also system prompts.
  6. If you talk about folder based disk distribution (to a single JBOD like view), then the system should both be smart AND allow for great user flexibility. I expect to see a choice of filling disks one by one (possibly up to a threshold – and with use choice «smaller-to-larger» or the opposite or even following a specific priority set by the user) OR evenly filling them AND allowing at all times the user to put something on a specific disk manually without messing the system’s logic.
  7.  Expand the above to «green». A single folder should remain on one disk if possible. Other disks don’t need to spin if files the user is looking (like «Music») are all together. In fact NO disk should spin if the system includes a nice smart directory listing cache (user configurable size and supporting maintenance in case something de-syncs it… manually putting a file on a specific disk shouldn’t normally de-sync it, as the OS has functions that can alert the system of the change).
  8. On a more mature system, a maintenance mode where the system will be able to move files around based on the logic of #6 above. In other words if the user decided in the past to evenly fill disks and then changed his mind to fill them one by one, to be able to do it. «Safe» moves of course. The same maintenance mode could also become smarter to support green philosophy mentioned above. For example system had decided that folder «videos» and «pictures» are on the same disk (user has decided to fill disks one by one). Then adding to videos makes this folder so large that «pictures» wouldn’t fit if videos where all in the same disk. So the system puts new videos in the next disk but when in maintenance, shifts the whole «pictures» folder on next disk and puts all videos together on the first. It’s a wishlist, ain’t it? 🙂 Of course this isn’t very «green» as it creates more disk I/O daily (or whenever system is doing maintenance). But give user the choice of what happens during maintenance
  9. The custom «unified» views should have their own user/group security and of course be able to be mapped to a drive letter and shared (else what is the point eh?).
  10. Should be able to work on various FS (in fact if possible not care for the FS as long as normal system calls work) – in fact even different FS at the same time.
  11. Folder caching should also support at some point, an off-line mode. In other words set some attribute for files on a missing disk and still show them (in fact it should be user choice to show or not). This will possibly help cooperation among parity systems.
  12. I would in the end love, to automatically support something like Windows 7 «library» logic. i.e. drop a file on a specific folder in a custom view based on file type.

 I ask to much? You bet.