Are There Reasonable Multi-Master Over the WAN Storage Options?


  • Service Provider

    Trying to help @stefuk come up with a reasonable option for a small client that wants a triple region multi-master SMB protocol storage solution. Meaning they want each office (three offices, each in a different country, roughly 50Mb/s and up to 300ms latency between them) to see their storage pool as local and to work with it like they "own" the storage. Total storage is about 10TB today but needs to scale up and daily change rate is, supposedly, about 1GB/day (but how did they get up to 10TB at that rate?)

    Complex, no matter how you look at it. The complications are, of course, that any change means that data has to go over the wire very quickly and locking needs to be multi-regional and any blip in the network means that the storage will be orphaned and some process has to get it all synced back up.

    There is not any normal product that handles this. Something like Samba on GFS2 on DRBD can do this, but will be super complex and I have no idea how poorly it will perform, but pretty poorly, is my guess. Even going to async will only do so much and will cause the locking to potentially get out of sync.

    Anything using DFS seems like a bad idea, DFS not having a great reputation for stability, and this being over the WAN would push it far more than normal. And DFS does not have locking, so a third party file locking system like PeerLock would be required. PeerLink is an issue but I have no experience with it and no idea of the cost (but they seem to imply that they normally run it on NetApp, if that gives you a feel for how expensive it must be.)

    My guess is that the multi-master concept is just a bad one and that there is no really reasonable way to do that. Instead I am thinking that regional masters, each of their own data, with bi-directional replication is going to be the best real world solution. I'm leaning towards Exablox as that will make most of the storage a lot easier. Thoughts? Ideas?



  • How does a regional master setup work? If someone is editing in the master region, and someone is editing it in a non master at the same time, the sync would end up with only a copy from the master, and the non regional changes would be lost?

    What kinds of files are we talking about there? I'm guessing not Office type documents as something like SharePoint would solve this problem.

    If the size is keeping things from going to SharePoint, what about some sort of archive, assuming that older files aren't actually being edited, and an archive point can be chosen and files moved to the archive as needed.


  • Service Provider

    @Dashrender said:

    How does a regional master setup work? If someone is editing in the master region, and someone is editing it in a non master at the same time, .....

    It wouldn't let them, same as if they were both local. It needs to lock the files as being open.


  • Service Provider

    @Dashrender said:

    What kinds of files are we talking about there? I'm guessing not Office type documents as something like SharePoint would solve this problem.

    No, bigger ones like engineering files.


  • Service Provider

    @Dashrender said:

    If the size is keeping things from going to SharePoint...

    Centralized hosting has been tested and is not fast enough for their use case.



  • @scottalanmiller said:

    @Dashrender said:

    If the size is keeping things from going to SharePoint...

    Centralized hosting has been tested and is not fast enough for their use case.

    Yeah - I assumed that had to be the case.

    Definitely curious how you solve this?


  • Service Provider

    My take on it is... you don't. It's not a reasonable thing to attempt to do. You make people change their processes.



  • @scottalanmiller said:

    My take on it is... you don't. It's not a reasonable thing to attempt to do. You make people change their processes.

    Where are the sites located?


  • Service Provider

    @wirestyle22 said:

    @scottalanmiller said:

    My take on it is... you don't. It's not a reasonable thing to attempt to do. You make people change their processes.

    Where are the sites located?

    UK, France and Malaysia. I think.



  • That just seems like a bad idea. If the WAN fails, you are going to have issues with different file versions.



  • @scottalanmiller said:

    @Dashrender said:

    What kinds of files are we talking about there? I'm guessing not Office type documents as something like SharePoint would solve this problem.

    No, bigger ones like engineering files.

    With the size of those files, and only 50mbps, I doubt that's going to cut it for what they want to do. Back in the late 1990 and early 2000 all the engineering software packages had integrated file management options. One of those with a WAN between sites. I would mean waiting a bit when someone checks out a file that's not local, but you wouldn't have to sync everything over the slow WAN connection.

    @scottalanmiller said:

    @Dashrender said:

    If the size is keeping things from going to SharePoint...

    Centralized hosting has been tested and is not fast enough for their use case.

    What makes them think self-hosting on the same connections is somehow faster? It's magic!


  • Service Provider

    At least self hosting, with the local connection, the local users will get local LAN speeds as long as there is no locking.



  • Why not go with Self-Hosting and replicated off-site?

    If somebody in France needs access to a file from Malaysia, then they should connect to the Malaysia file server via <insert your method here> to access the files.

    Like @scottalanmiller -- sometimes you have to change the processes.


  • Service Provider

    @dafyre said:

    Why not go with Self-Hosting and replicated off-site?

    If somebody in France needs access to a file from Malaysia, then they should connect to the Malaysia file server via <insert your method here> to access the files.

    Like @scottalanmiller -- sometimes you have to change the processes.

    That's what I am thinking. I want to look at using Exablox, one at each site. Each site with their own share of which they are the master that then replicates to the other sites.


  • Service Provider

    @dafyre said:

    Why not go with Self-Hosting and replicated off-site?

    If somebody in France needs access to a file from Malaysia, then they should connect to the Malaysia file server via <insert your method here> to access the files.

    Like @scottalanmiller -- sometimes you have to change the processes.

    That is the proposed solution from @scottalanmiller. It is not the solution they want though.


  • Service Provider

    Although I've only submitted that recommendation to @StefUk so it might be that with a talk with the business that they will understand and be ready to go that route.



  • @scottalanmiller said:

    @dafyre said:

    Why not go with Self-Hosting and replicated off-site?

    If somebody in France needs access to a file from Malaysia, then they should connect to the Malaysia file server via <insert your method here> to access the files.

    Like @scottalanmiller -- sometimes you have to change the processes.

    That's what I am thinking. I want to look at using Exablox, one at each site. Each site with their own share of which they are the master that then replicates to the other sites.

    Unless I missed it, we still haven't been told how large the files are. Is it better to deal with possible sync issues, or how about using RDS instead? If you really need to work on a file at relative performance, an RDS server in each location that users can share might be a better option.


  • Service Provider

    RDS to a central location with VDI is being proposed as a long term solution, but not something that they are prepared to deal with in the short term.



  • How does using Exablox solve a file versioning problem? What is the solution for that specific problem, assuming you can't force a lock out to all nodes?


  • Service Provider

    @Dashrender said:

    How does using Exablox solve a file versioning problem?

    Single site masters.



  • @scottalanmiller said:

    RDS to a central location with VDI is being proposed as a long term solution, but not something that they are prepared to deal with in the short term.

    Just before reading this, that is exactly where my mind leapt. Centralize the whole thing - RDS to a box near that storage pool. Problem solved.



  • @scottalanmiller said:

    @Dashrender said:

    How does using Exablox solve a file versioning problem?

    Single site masters.

    Please provide more details.


  • Service Provider

    @Dashrender said:

    @scottalanmiller said:

    @Dashrender said:

    How does using Exablox solve a file versioning problem?

    Single site masters.

    Please provide more details.

    UK Exablox with a share of which it is the master. Replication of that data goes to France and KL.

    France Exablox with a share of which it is the master. Replication of that data goes to UK and KL.

    KL Exablox with a share of which it is the master. Replication of that data goes to UK and France.

    Each site gets its own local data of which it is "in charge". It has the "write" share for that data. The replication is purely for reads.

    Each site can work with its local data as normal. It's just a normal mapped drive for them. If a site needs data from another site it grabs a read only copy super fast from the replication, makes changes and then saves those changes over the WAN to the location where the master for that share is. Cumbersome on the less common saves, but only one master for every file.



  • Awesome - thanks!


  • Service Provider

    @scottalanmiller said:

    @Dashrender said:

    @scottalanmiller said:

    @Dashrender said:

    How does using Exablox solve a file versioning problem?

    Single site masters.

    Please provide more details.

    UK Exablox with a share of which it is the master. Replication of that data goes to France and KL.

    France Exablox with a share of which it is the master. Replication of that data goes to UK and KL.

    KL Exablox with a share of which it is the master. Replication of that data goes to UK and France.

    Each site gets its own local data of which it is "in charge". It has the "write" share for that data. The replication is purely for reads.

    Each site can work with its local data as normal. It's just a normal mapped drive for them. If a site needs data from another site it grabs a read only copy super fast from the replication, makes changes and then saves those changes over the WAN to the location where the master for that share is. Cumbersome on the less common saves, but only one master for every file.

    Exactly, and if they restructure their shares correctly, those less common saves should really be uncommon.



  • Hi everyone,
    thanks for chiming in ... and thanks @scottalanmiller for posting this on my behalf.

    RDS is currently out of the question at the moment due to intense graphical resources that they need. We are looking at some long term solutions - Nvidia Grid of some sort but at the moment RDS will not cut it.

    stef


  • Service Provider

    @JaredBusch That's what I am thinking. Hopefully the workflow will make for a pretty solid local storage situation the bulk of the time.



  • @JaredBusch @JaredBusch @scottalanmiller

    that's good in terms of replication.
    but what about working on the same files - project ? what you are saying is that there is no way to get them to work on the same project without file version issues ?

    Stef


  • Service Provider

    @StefUk said:

    @JaredBusch @JaredBusch @scottalanmiller

    that's good in terms of replication.
    but what about working on the same files - project ? what you are saying is that there is no way to get them to work on the same project without file version issues ?

    Correct, there is not. If they each have a copy of their own data, they each have an opportunity to work on them at the same time. Once you have multiple masters, you have issues. No way around that.



  • @scottalanmiller said:

    @StefUk said:

    @JaredBusch @JaredBusch @scottalanmiller

    that's good in terms of replication.
    but what about working on the same files - project ? what you are saying is that there is no way to get them to work on the same project without file version issues ?

    Correct, there is not. If they each have a copy of their own data, they each have an opportunity to work on them at the same time. Once you have multiple masters, you have issues. No way around that.

    So really we're trying to figure out how to combine all of the changes, correct? Can't this be done with .tmp files fragmenting and then recombining? I'm sure their software doesn't support this but I'm just asking hypothetically for my own knowledge.


Log in to reply
 

Looks like your connection to MangoLassi was lost, please wait while we try to reconnect.