I have to change cloud drive service yet again
-
@Dashrender said in I have to change cloud drive service yet again:
That said - we have a process where faxes are dumped into a folder, then they are uploaded into our EHR, then they are moved to a trash folder where they are left for 30 days, then deleted. So we do a move process all the time, ever day.
Trashing a file isn't the same as a normal move, because it is always out of the sync system. So it would never be an issue, it's just a deletion in this context. We are talking exclusively about moving around a single file system, not about deletions.
-
@Dashrender said in I have to change cloud drive service yet again:
I also don't get why Scott is against people doing moves? Does the fact that this is a sync'ing system somehow change this process?
Yes, syncing breaks this process because different people are syncing up files from different places. Think about it from a meta-data standpoint.
But before we do that... of course I'm against moves, sync'd or not. Just moving files around all of the time is very inefficient, not just from a computer standpoint (but it means that folders are being used as organizational structure in a way that isn't good) but also from a human way (if files are regularly moving you risk all kinds of problems from slowness as people have to search for something that used to be one place and is not in another, to one person editing a file that is deleted out from under them.) It's not good for the computer nor the humans.
Now... why syncing can't support moving files....
-
The sync process scans the end point for files. Those files are copied up to the server. The data point on the server is where the meta data is attached and how it knows to look for changes. Move that file, and there is no connection to the file on the client any longer.... so...
Client A creates myfile.doc
Client A syncs to Server
Server now has a copy of myfile.doc and stores data about it like its version history, ownership, sharing, etc.
Client A moves file to new location. (Deletes the existing copy, makes a new file with the same name in a different spot. A move IS a deletion and recreation to the local filesystem.)
Sync to the server forces one to be deleted and a new one to be created. The server can't see that a file was "moved", that's not visible to it. It sees what really happened, one was deleted and another was created that just happens to have the same name and contents. But it is NOT the same file to the server (and potentially not to the end user.)
Now the server thinks it should delete the one that was there, and create a new one, so it does.
Now the server pushes down the deletion and the new creation to Clients B and C. Everyone is in sync, although now two users dont' know where their file is, all shares are broken, and all history and permissions are broken. The file is there, but things aren't set up like they were thought to be. No big deal.
Now Client B was working on that file at the time that it was moved and they save it. Since the file was deleted and no longer exists, when they save it it creates a new file.
Client B syncs the new file up to the Server. The Server has no idea that this is the same file that was deleted previously because that file doesn't exist, has no data, and this file is brand new.
Now Clients A and C get a copy of this divergent file, in the original location, from the Server.
To the end users it looks like files are reverting. But what really happened is that the user on Client A broke the meta-data because of their "moving" of files and created a break in the database so that the files being worked on by the end users are not associated with the file that was moved.
-
@Dashrender said in I have to change cloud drive service yet again:
@scottalanmiller said in I have to change cloud drive service yet again:
They should stop moving files around without coordinating with everyone.
How do you see this working?
11 people with a shared drive, all syncing the data local.
Train them to use computers properly.
THe bottom line, that we don't know is, WHY are people moving files around? What is broken that makes them want files to be shuffling regularly.
-
@Dashrender said in I have to change cloud drive service yet again:
I don't disagree with the OP that his issues definitely seem like shitty programming problems that simply shouldn't exist. 20 people all sync the same files - 5 people are offline most of the time - someone moves a file, one of the offline people come back online - the offline system should see a date notice of the file move and change the once offline client to match the server.
Should it? Should moving files even be really possible? Since there is no notice of file move, that's not a thing, what do you expect the system to see? There is no "move log" or anything of the sort. Simple one file gets deleted, another gets created.
To the cloud sync tool, this could happen for any of a number of reasons. It can't guess. It is the humans breaking the system here, not the computer.
You could complain that the OS' filesystem layer doesn't report "moves" somehow through an API if you want. That seems silly.
The real issue here is that exposing a raw database (filesystem) to end users is fundamentally bad computing, but necessary because people are so used to it. But this is why things like iOS devices refuse to do this and don't let people work with files at all. You stop thinking about it, but it's nothing to do with mobile computing, but lack of filesystem exposure on an iPhone or iPad that makes it so robust and easy to use. Each app gets its own space and works with its own files and you never need to manage file storage.
Letting users see the filesystem is truly too "under the hood" and allows users to do too much damage. We have two choices, move to OSes like iOS that forbid this, or train users on how to properly use a computer that exposes a filesystem. Those are the options.
-
cough SharePoint cough
-
@scottalanmiller While I don't disagree with what you are saying, The exposure of the file system has a very practical purpose in the real world in that it allows users, especially those that are not tech people, to understand that hard concepts of what a file is and where it exists. By presenting it the way that windows does, individual files can be treated in the users mind like their analog counterpart. People move physical files and folders around in real filing cabinets all the time and they expect computers to facilitate this as an extension of the old paper workflow. Windows calls these actions "move", but as you say, it's really a copy and delete action, and the resulting file is a new creation as far as the file system is concerned. But most people don't have that level of understanding of the file system and therefore it is very reasonable for people to expect them to behave like their physical counterpart. I also think using files moving between folders is a very legitimate workflow mechanism, maybe not the most efficient, and that people that use this method might have a folder for work in progress, a folder for submitted, or what ever. Yes, I know that in the end it is all meta data and we are just substituting the file location meta data for proper tags like "in progress" or "approved". But since our OS does not natively have these meta data tags (do any?), we can either invest in something like a document management system that adds and keeps track of those tags, or we can use the tools built into the OS.
One problem that I see from the iOS argument is when you have more than one program that is compatible with the same file. To give you an example, once a quarter I take an sql query and dump the results to a csv file. I then change the file extension on it to txt and upload that to the state. This is for quarterly taxes. If all of this is abstracted away by the OS, then trying to do something similar would be tedious at best, perhaps even impossible. You could argue that the application can be changed to output the desired file format the first time instead of using the csv middle man, but that is well beyond the scope of most people that can only use tools given to them, not create new ones.
-
@scottalanmiller said in I have to change cloud drive service yet again:
(Deletes the existing copy, makes a new file with the same name in a different spot. A move IS a deletion and recreation to the local filesystem.)
This is 100% true, and is exactly how it shows up in the audit logs for Windows file servers if you have auditing turned on.
-
is there a file system that will track all this meta data, even when the file is deleted? To me it feels like general ledger type transactions where the past history is part of the permanent record. I know its apples and oranges, but is there anything on the horizon that looks at this from the point of view of historical chronology? I would probably lead to meta data bloat, which would be my guess as to why it doesn't exist now.
-
@scottalanmiller said in I have to change cloud drive service yet again:
@Dashrender said in I have to change cloud drive service yet again:
@wirestyle22 said in I have to change cloud drive service yet again:
@guyinpv said in I have to change cloud drive service yet again:
@scottalanmiller said in I have to change cloud drive service yet again:
@guyinpv said in I have to change cloud drive service yet again:
@Obsolesce said in I have to change cloud drive service yet again:
Users aren't saving a copy to their hard drive, then putting it back after someone else moves or renames it?
Using the NextCloud Sync client, all the files are already local on their drives. So the users are using CTRL-X to cut files and paste them into other folders. And occasionally, those file operations reverse and a moved file goes back to where it was. Or a renamed files gets reversed. Or worse, a renamed and moved file because a duplicate in its original folder, leaving us with 2 copies with multiple names.
yeah, that's a behavioural thing that has to stop. Has to, period. Simply isn't something that they can do when using that kind of system, any system like that.
What behavior is better?
Assuming you mean they should stop using ctrl-x to cut and paste files. Are you suggesting they should, say, copy the file to their desktop, then delete the NC one, then copy the file back into the new folder location?
Funny thing is, when they ctrl-x and move a file, NC literally records the activity as a file move operation. It knows what just happened. That just makes everything that much more weird.
I know of no business that does this (ctrl+x to move files). If they are all doing this someone must have told them to do it this way. Was that you?
huh - about 50% of my users use ctrl + keys all day long. We do a lot of copy pasting between systems. I know I showed some of them, some knew from before.
They move files around the file system? Why? They want to hide them from other uses? Want to lower efficiency? What purpose does it serve to move files around all of the time?
That's not what I said - I said "50% of my users use ctrl + keys" they are using it mostly to copy paste text between apps, not to move files in the filesystem.
-
@scottalanmiller said in I have to change cloud drive service yet again:
@Dashrender said in I have to change cloud drive service yet again:
That said - we have a process where faxes are dumped into a folder, then they are uploaded into our EHR, then they are moved to a trash folder where they are left for 30 days, then deleted. So we do a move process all the time, ever day.
Trashing a file isn't the same as a normal move, because it is always out of the sync system. So it would never be an issue, it's just a deletion in this context. We are talking exclusively about moving around a single file system, not about deletions.
In my case they are not being trashed - they are being moved to a 'finished' folder. the folder is active and online in case there was an issue in the upload process.
-
@scottalanmiller said in I have to change cloud drive service yet again:
The sync process scans the end point for files. Those files are copied up to the server. The data point on the server is where the meta data is attached and how it knows to look for changes. Move that file, and there is no connection to the file on the client any longer.... so...
Client A creates myfile.doc
Client A syncs to Server
Server now has a copy of myfile.doc and stores data about it like its version history, ownership, sharing, etc.
Client A moves file to new location. (Deletes the existing copy, makes a new file with the same name in a different spot. A move IS a deletion and recreation to the local filesystem.)
Sync to the server forces one to be deleted and a new one to be created. The server can't see that a file was "moved", that's not visible to it. It sees what really happened, one was deleted and another was created that just happens to have the same name and contents. But it is NOT the same file to the server (and potentially not to the end user.)
Now the server thinks it should delete the one that was there, and create a new one, so it does.
Now the server pushes down the deletion and the new creation to Clients B and C. Everyone is in sync, although now two users dont' know where their file is, all shares are broken, and all history and permissions are broken. The file is there, but things aren't set up like they were thought to be. No big deal.
Now Client B was working on that file at the time that it was moved and they save it. Since the file was deleted and no longer exists, when they save it it creates a new file.
Client B syncs the new file up to the Server. The Server has no idea that this is the same file that was deleted previously because that file doesn't exist, has no data, and this file is brand new.
Now Clients A and C get a copy of this divergent file, in the original location, from the Server.
To the end users it looks like files are reverting. But what really happened is that the user on Client A broke the meta-data because of their "moving" of files and created a break in the database so that the files being worked on by the end users are not associated with the file that was moved.
OK this whole thing makes sense - So you really believe the OP's issue is that people have open files they are editing, and that is the root of the issue here? The OP hasn't mentioned anything about multiple people being in the same file and seeing files 'reappear'... granted he could have left that out.
-
@Dashrender said in I have to change cloud drive service yet again:
@scottalanmiller said in I have to change cloud drive service yet again:
The sync process scans the end point for files. Those files are copied up to the server. The data point on the server is where the meta data is attached and how it knows to look for changes. Move that file, and there is no connection to the file on the client any longer.... so...
Client A creates myfile.doc
Client A syncs to Server
Server now has a copy of myfile.doc and stores data about it like its version history, ownership, sharing, etc.
Client A moves file to new location. (Deletes the existing copy, makes a new file with the same name in a different spot. A move IS a deletion and recreation to the local filesystem.)
Sync to the server forces one to be deleted and a new one to be created. The server can't see that a file was "moved", that's not visible to it. It sees what really happened, one was deleted and another was created that just happens to have the same name and contents. But it is NOT the same file to the server (and potentially not to the end user.)
Now the server thinks it should delete the one that was there, and create a new one, so it does.
Now the server pushes down the deletion and the new creation to Clients B and C. Everyone is in sync, although now two users dont' know where their file is, all shares are broken, and all history and permissions are broken. The file is there, but things aren't set up like they were thought to be. No big deal.
Now Client B was working on that file at the time that it was moved and they save it. Since the file was deleted and no longer exists, when they save it it creates a new file.
Client B syncs the new file up to the Server. The Server has no idea that this is the same file that was deleted previously because that file doesn't exist, has no data, and this file is brand new.
Now Clients A and C get a copy of this divergent file, in the original location, from the Server.
To the end users it looks like files are reverting. But what really happened is that the user on Client A broke the meta-data because of their "moving" of files and created a break in the database so that the files being worked on by the end users are not associated with the file that was moved.
OK this whole thing makes sense - So you really believe the OP's issue is that people have open files they are editing, and that is the root of the issue here? The OP hasn't mentioned anything about multiple people being in the same file and seeing files 'reappear'... granted he could have left that out.
He has mentioned file locking a dozen or so times though and that would seem to confirm what @scottalanmiller is thinking.
-
@scottalanmiller said in I have to change cloud drive service yet again:
@Dashrender said in I have to change cloud drive service yet again:
I don't disagree with the OP that his issues definitely seem like shitty programming problems that simply shouldn't exist. 20 people all sync the same files - 5 people are offline most of the time - someone moves a file, one of the offline people come back online - the offline system should see a date notice of the file move and change the once offline client to match the server.
Should it? Should moving files even be really possible? Since there is no notice of file move, that's not a thing, what do you expect the system to see? There is no "move log" or anything of the sort. Simple one file gets deleted, another gets created.
To the cloud sync tool, this could happen for any of a number of reasons. It can't guess. It is the humans breaking the system here, not the computer.
You could complain that the OS' filesystem layer doesn't report "moves" somehow through an API if you want. That seems silly.
The real issue here is that exposing a raw database (filesystem) to end users is fundamentally bad computing, but necessary because people are so used to it. But this is why things like iOS devices refuse to do this and don't let people work with files at all. You stop thinking about it, but it's nothing to do with mobile computing, but lack of filesystem exposure on an iPhone or iPad that makes it so robust and easy to use. Each app gets its own space and works with its own files and you never need to manage file storage.
Letting users see the filesystem is truly too "under the hood" and allows users to do too much damage. We have two choices, move to OSes like iOS that forbid this, or train users on how to properly use a computer that exposes a filesystem. Those are the options.
SMB and I assume other local sharing protocols have been providing file locks for decades - so yes, I expect this.
That said - I know file locks aren't possible when you're offline - so a syncing type solution just can't reasonably be expected to use it.
I completely agree with you on hiding the filesystem from users. We need a way to have a central storage location for files when accessing them via the apps, as well as a personal location (like MS Office allowing local docs or OneDrive)... so once we have that, moving to the app only access to files would be pretty nice.
One issue with the app only access - what happens when you change apps? i.e. switching from using Adobe Acrobat to PDF Element? Is it now up to the OS to know what files belong to what app? or is there simply a single folderless dumping ground for all files?
Assuming you've been using Excel for years, you could have hundreds or thousands of xlsx files. searching through these is painful at best for people. forcing the use of tags will be helpful, but I haven't seen that deployed much if at all. simply searching titles are often useless, because we don't always remember the actual title we give something. etc etc.
-
@Dashrender said in I have to change cloud drive service yet again:
@scottalanmiller said in I have to change cloud drive service yet again:
@Dashrender said in I have to change cloud drive service yet again:
I don't disagree with the OP that his issues definitely seem like shitty programming problems that simply shouldn't exist. 20 people all sync the same files - 5 people are offline most of the time - someone moves a file, one of the offline people come back online - the offline system should see a date notice of the file move and change the once offline client to match the server.
Should it? Should moving files even be really possible? Since there is no notice of file move, that's not a thing, what do you expect the system to see? There is no "move log" or anything of the sort. Simple one file gets deleted, another gets created.
To the cloud sync tool, this could happen for any of a number of reasons. It can't guess. It is the humans breaking the system here, not the computer.
You could complain that the OS' filesystem layer doesn't report "moves" somehow through an API if you want. That seems silly.
The real issue here is that exposing a raw database (filesystem) to end users is fundamentally bad computing, but necessary because people are so used to it. But this is why things like iOS devices refuse to do this and don't let people work with files at all. You stop thinking about it, but it's nothing to do with mobile computing, but lack of filesystem exposure on an iPhone or iPad that makes it so robust and easy to use. Each app gets its own space and works with its own files and you never need to manage file storage.
Letting users see the filesystem is truly too "under the hood" and allows users to do too much damage. We have two choices, move to OSes like iOS that forbid this, or train users on how to properly use a computer that exposes a filesystem. Those are the options.
SMB and I assume other local sharing protocols have been providing file locks for decades - so yes, I expect this.
That depends if the application is holding the file open or not. If someone hits "save as", it's going to put it back. Problem exists regardless.
-
@Dashrender said in I have to change cloud drive service yet again:
One issue with the app only access - what happens when you change apps? i.e. switching from using Adobe Acrobat to PDF Element? Is it now up to the OS to know what files belong to what app? or is there simply a single folderless dumping ground for all files?
All modern OSes already handle that. It's a long ago solved problem.
-
@Dashrender said in I have to change cloud drive service yet again:
simply searching titles are often useless, because we don't always remember the actual title we give something. etc etc.
If titles (and presumable folder titles) don't let you find something, how does moving files around solve any problems?
-
@coliver said in I have to change cloud drive service yet again:
@Dashrender said in I have to change cloud drive service yet again:
@scottalanmiller said in I have to change cloud drive service yet again:
The sync process scans the end point for files. Those files are copied up to the server. The data point on the server is where the meta data is attached and how it knows to look for changes. Move that file, and there is no connection to the file on the client any longer.... so...
Client A creates myfile.doc
Client A syncs to Server
Server now has a copy of myfile.doc and stores data about it like its version history, ownership, sharing, etc.
Client A moves file to new location. (Deletes the existing copy, makes a new file with the same name in a different spot. A move IS a deletion and recreation to the local filesystem.)
Sync to the server forces one to be deleted and a new one to be created. The server can't see that a file was "moved", that's not visible to it. It sees what really happened, one was deleted and another was created that just happens to have the same name and contents. But it is NOT the same file to the server (and potentially not to the end user.)
Now the server thinks it should delete the one that was there, and create a new one, so it does.
Now the server pushes down the deletion and the new creation to Clients B and C. Everyone is in sync, although now two users dont' know where their file is, all shares are broken, and all history and permissions are broken. The file is there, but things aren't set up like they were thought to be. No big deal.
Now Client B was working on that file at the time that it was moved and they save it. Since the file was deleted and no longer exists, when they save it it creates a new file.
Client B syncs the new file up to the Server. The Server has no idea that this is the same file that was deleted previously because that file doesn't exist, has no data, and this file is brand new.
Now Clients A and C get a copy of this divergent file, in the original location, from the Server.
To the end users it looks like files are reverting. But what really happened is that the user on Client A broke the meta-data because of their "moving" of files and created a break in the database so that the files being worked on by the end users are not associated with the file that was moved.
OK this whole thing makes sense - So you really believe the OP's issue is that people have open files they are editing, and that is the root of the issue here? The OP hasn't mentioned anything about multiple people being in the same file and seeing files 'reappear'... granted he could have left that out.
He has mentioned file locking a dozen or so times though and that would seem to confirm what @scottalanmiller is thinking.
well, my memory of him mentioning file locking was that it didn't exist - not that he was getting "in use" messages because of file locking. But I'm to lazy to read up through the entire thread now.
-
@scottalanmiller said in I have to change cloud drive service yet again:
@Dashrender said in I have to change cloud drive service yet again:
One issue with the app only access - what happens when you change apps? i.e. switching from using Adobe Acrobat to PDF Element? Is it now up to the OS to know what files belong to what app? or is there simply a single folderless dumping ground for all files?
All modern OSes already handle that. It's a long ago solved problem.
By all modern, I assume you mean Chrome OS/android and iOS? but not windows?
-
@Dashrender said in I have to change cloud drive service yet again:
@scottalanmiller said in I have to change cloud drive service yet again:
@Dashrender said in I have to change cloud drive service yet again:
One issue with the app only access - what happens when you change apps? i.e. switching from using Adobe Acrobat to PDF Element? Is it now up to the OS to know what files belong to what app? or is there simply a single folderless dumping ground for all files?
All modern OSes already handle that. It's a long ago solved problem.
By all modern, I assume you mean Chrome OS/android and iOS? but not windows?
I'm not following? Windows can manage file types in this fashion as well. Must have missed something.
