Checking multiple Directories to confirm all files are identical
-
@eddiejennings Yeah I was thinking of the same solution as well, my trouble is how would I get the system to not try and store everything to memory first and then write to file. . . .
Some of these customer requests are insane...
-
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
@flaxking said in Checking multiple Directories to confirm all files are identical:
I would think you would be able to use robocopy to do a diff
Probably, but the issue still comes down to system resources.
Anything that is storing in memory will quickly consume the available resources.
Maybe if I pipe the about to a file it won't be so bad..
It's bound to be a lot more efficient than your powershell.
-
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
@eddiejennings Yeah I was thinking of the same solution as well, my trouble is how would I get the system to not try and store everything to memory first and then write to file. . . .
Some of these customer requests are insane...
If you do the equivalent of
md5sumwith subdirectories you will get md5 sums of all files. A diff with produce the different files.
File size or directory size will not matter at all for this operation.Get-FileHash seems to output multiple lines per file which is not good for this.
If you don't need hash to compare and just wanted to check filenames, file sizes and dates, maybe you should just do a directory listing for each tree and compare them. That would be very fast.
You could get
dirto provide a one-file-per-line output, with the proper options. -
@pete-s said in Checking multiple Directories to confirm all files are identical:
If you don't need hash to compare and just wanted to check filenames, file sizes and dates, maybe you should just do a directory listing for each tree and compare them. That would be very fast.
While I don't need the hash's of the files I was hoping to get some automated way of saying these files aren't*** in dir#.
But I can't for the life of me think of a good way to do that without eating up all of the ram in the world....
-
@flaxking said in Checking multiple Directories to confirm all files are identical:
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
@flaxking said in Checking multiple Directories to confirm all files are identical:
I would think you would be able to use robocopy to do a diff
Probably, but the issue still comes down to system resources.
Anything that is storing in memory will quickly consume the available resources.
Maybe if I pipe the about to a file it won't be so bad..
It's bound to be a lot more efficient than your powershell.
It's still going to consume more ram than any host in the environment has to process the job. Just between any 2 directories there's over 20 million files.
-
I know WinMerge has a folder comparison feature, but not sure it can handle your file count.
-
@danp said in Checking multiple Directories to confirm all files are identical:
I know WinMerge has a folder comparison feature, but not sure it can handle your file count.
It might be worth a try, I hadn't thought of it.
-
@dustinb3403 That's what I was thinking.
You'll still be in the shape of how do you compare two stupidly large files, though.
-
@dafyre said in Checking multiple Directories to confirm all files are identical:
@dustinb3403 That's what I was thinking.
You'll still be in the shape of how do you compare two stupidly large files, though.
Yeah, while that is certainly a part of the challenge, the larger portion is just checking to see if the bulk is all aligned and matching...
If any tooling had some way to "skip large files" and just jot down their names then a simple stare and compare might work in that case.
-
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
@flaxking said in Checking multiple Directories to confirm all files are identical:
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
@flaxking said in Checking multiple Directories to confirm all files are identical:
I would think you would be able to use robocopy to do a diff
Probably, but the issue still comes down to system resources.
Anything that is storing in memory will quickly consume the available resources.
Maybe if I pipe the about to a file it won't be so bad..
It's bound to be a lot more efficient than your powershell.
It's still going to consume more ram than any host in the environment has to process the job. Just between any 2 directories there's over 20 million files.
I don't know how it's implemented, so I can't say. Just create a new powershell script that doesn't store as much in memory. I think if you pipe to ForEach-Object it actually starts operating before the get-childitem gets all the objects and then don't store those objects in a variable. So maybe it will start garbage collection before you are done
-
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
@dafyre said in Checking multiple Directories to confirm all files are identical:
@dustinb3403 That's what I was thinking.
You'll still be in the shape of how do you compare two stupidly large files, though.
Yeah, while that is certainly a part of the challenge, the larger portion is just checking to see if the bulk is all aligned and matching...
If any tooling had some way to "skip large files" and just jot down their names then a simple stare and compare might work in that case.
So are you looking to compare bit for bit -- or just file name and size ?
-
@dafyre said in Checking multiple Directories to confirm all files are identical:
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
@dafyre said in Checking multiple Directories to confirm all files are identical:
@dustinb3403 That's what I was thinking.
You'll still be in the shape of how do you compare two stupidly large files, though.
Yeah, while that is certainly a part of the challenge, the larger portion is just checking to see if the bulk is all aligned and matching...
If any tooling had some way to "skip large files" and just jot down their names then a simple stare and compare might work in that case.
So are you looking to compare bit for bit -- or just file name and size ?
Name, size date ideally. bit for bit is overkill and I can't image the client would want to wait for who knows how long to get an answer for this.
-
One thought - run a md5 hash and output filename date hash to a file, then compare the contents of the files between the servers.
you could run the job individually on each server so all three plus devices could run at once - assuming not all on the same VM host.
-
@dashrender said in Checking multiple Directories to confirm all files are identical:
One thought - run a md5 hash and output filename date hash to a file, then compare the contents of the files between the servers.
you could run the job individually on each server so all three plus devices could run at once - assuming not all on the same VM host.
That actually isn't a bad idea, time consuming still but would probably be way more lightweight than trying to perform a live comparison between the systems.
Just use something like Meld to compare the text files after the fact.
-
@dashrender said in Checking multiple Directories to confirm all files are identical:
One thought - run a md5 hash and output filename date hash to a file, then compare the contents of the files between the servers.
you could run the job individually on each server so all three plus devices could run at once - assuming not all on the same VM host.
This appears to be working, thanks for the idea!
dir D:\Files -Recurse | Get-FileHash -ea Continue > C:\D-Files.txtDumps out one large file, I could then use a file comparison tool to quickly check these outputs.
-
@dustinb3403 You'll likely have to do some type of line fixup, i.e. if a file is missing, then every line after that would be a mismatch...
-
@dashrender said in Checking multiple Directories to confirm all files are identical:
@dustinb3403 You'll likely have to do some type of line fixup, i.e. if a file is missing, then every line after that would be a mismatch...
Maybe, Meld has actually been pretty good about that type of "issue" and can find the record anyways.
-
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
In a windows environment if you wanted to check multiple network directories, with millions of files ranging in sizes from tiny (a few KB) to large 4GB+ how would you do it.
If you need to search files on a server over network share, on the server you should install the windows search feature, and ensure the shared files and locations are indexed. That will greatly speed up the searching via Win10 computers.
-
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
Ideally I'd like to compare them all at once, but setting the "golden standard" here may be difficult.
Wait, what? Why do you have multiple directories that should have identical data? That makes absolutely no sense. There are methods to use in file server administration to avoid this...
-
@dustinb3403 said in Checking multiple Directories to confirm all files are identical:
I know I could use a tool like Create-Synchronicity to force 1 other directory to match the source, but I would prefer to find and list the differences in the directories.
Maybe powershell can help?Yeah, PowerShell can help with this in the same way closing the front door of a house will fix a fire inside of it.
