File Management removing unprintable characters

DustinB3403

@scottalanmiller said in File Management removing unprintable characters:

What OS or file system? Is this macOS?

I'm willing to use WSL, Mac OS or Powershell on Windows 10 to do this, I just need some simple way to find these characters and replace the automatically without me having to find out a backup failed and that is the one file we really needed.

And since I can't beat it into my users that these characters are bad I have to work around them.

DustinB3403

The file system is a Windows environment with mac users, even from a windows workstation you can put in these unprintable characters in the folder and file names.

So the issue is universal as far as I'm concerned.

scottalanmiller

@DustinB3403 said in File Management removing unprintable characters:

The file system is a Windows environment with mac users, even from a windows workstation you can put in these unprintable characters in the folder and file names.

So the issue is universal as far as I'm concerned.

The issue has to be on a specific file system. So NTFS, I'm assuming.

DustinB3403

@scottalanmiller said in File Management removing unprintable characters:

@DustinB3403 said in File Management removing unprintable characters:

The file system is a Windows environment with mac users, even from a windows workstation you can put in these unprintable characters in the folder and file names.

So the issue is universal as far as I'm concerned.

The issue has to be on a specific file system. So NTFS, I'm assuming.

Yes, NTFS, but also this affects our backup (it has a stroke on files and folders with unprintable characters and just moves along).

dafyre

@DustinB3403 said in File Management removing unprintable characters:

And since I can't beat it into my users that these characters are bad I have to work around them.

Keep on telling them, and then they'll learn when that one file they need can't be restored because they used some ungodly character in the file name.

DustinB3403

@dafyre said in File Management removing unprintable characters:

@DustinB3403 said in File Management removing unprintable characters:

And since I can't beat it into my users that these characters are bad I have to work around them.

Keep on telling them, and then they'll learn when that one file they need can't be restored because they used some ungodly character in the file name.

Unfortunately it isn't my position to tell them. I report to management, management hands out the beatings.

dafyre

@DustinB3403 said in File Management removing unprintable characters:

@dafyre said in File Management removing unprintable characters:

@DustinB3403 said in File Management removing unprintable characters:

And since I can't beat it into my users that these characters are bad I have to work around them.

Keep on telling them, and then they'll learn when that one file they need can't be restored because they used some ungodly character in the file name.

Unfortunately it isn't my position to tell them. I report to management, management hands out the beatings.

Then tell management to beat harder until they start handing out pink slips when a file gets broken so badly it can't be restored and costs the company money.

scottalanmiller

@DustinB3403 said in File Management removing unprintable characters:

@scottalanmiller said in File Management removing unprintable characters:

@DustinB3403 said in File Management removing unprintable characters:

The file system is a Windows environment with mac users, even from a windows workstation you can put in these unprintable characters in the folder and file names.

So the issue is universal as far as I'm concerned.

The issue has to be on a specific file system. So NTFS, I'm assuming.

Yes, NTFS, but also this affects our backup (it has a stroke on files and folders with unprintable characters and just moves along).

Right, but how we modify the source is the first step.

scottalanmiller

So assuming that we have access to PowerShell here, this should be a start...

$String -replace '[^\p{L}\p{Nd}]', '-'

scottalanmiller

That Regex should, in theory, allow you to pass in a file name, and resave it with a hyphen instead of any unprintable character.

You just need to loop through the files.

DustinB3403

@scottalanmiller said in File Management removing unprintable characters:

That Regex should, in theory, allow you to pass in a file name, and resave it with a hyphen instead of any unprintable character.

You just need to loop through the files.

No way to not have to pass an individual file through this? Hoping to just be able to point this at a directory and it searches file and folder for unprintable characters replacing as it goes.

(I know wish in one hand, crap in the other)

dafyre

@DustinB3403 said in File Management removing unprintable characters:

@scottalanmiller said in File Management removing unprintable characters:

That Regex should, in theory, allow you to pass in a file name, and resave it with a hyphen instead of any unprintable character.

You just need to loop through the files.

No way to not have to pass an individual file through this? Hoping to just be able to point this at a directory and it searches file and folder for unprintable characters replacing as it goes.

(I know wish in one hand, crap in the other)

Something like...

$myFiles=dir -recurse C:\Some\Folder

foreach ($file in $myFiles) {
  $newName=$file -replace '[^\p{L}\p{Nd}]', '-'
 ren $file $newName
}

scottalanmiller

@DustinB3403 said in File Management removing unprintable characters:

@scottalanmiller said in File Management removing unprintable characters:

That Regex should, in theory, allow you to pass in a file name, and resave it with a hyphen instead of any unprintable character.

You just need to loop through the files.

No way to not have to pass an individual file through this? Hoping to just be able to point this at a directory and it searches file and folder for unprintable characters replacing as it goes.

(I know wish in one hand, crap in the other)

Well, that's what the loop would be for. One way or another, the files have to be iterated through.

scottalanmiller

@dafyre exactly

psophos

@dafyre You might want to throw in an if statement there to check that the names are different before you rename.
I'm assuming the ren will rename the file even if the names are the same, but maybe it won't. Some very light testing suggests it may (no errors thrown).

1337

NTFS is utf-16. All characters are allowed in filenames on NTFS, except reserved characters like
< (less than)
> (greater than)
: (colon)
" (double quote)
/ (forward slash)
\ (backslash)
| (vertical bar or pipe)
? (question mark)
* (asterisk)
and chr(0) to chr(31).

If the backup system can't handle all allowed characters in a filename, then that is the problem that needs to be fixed.

There is no such a thing as unprintable characters. Just need the right font that has that character defined.

This is screenshot from Windows showing valid file and folder names:

JaredBusch

@Pete-S said in File Management removing unprintable characters:

Unrelated to the OP....
Why do those kanji look familiar? Like seriously...

Obsolesce

@DustinB3403 said in File Management removing unprintable characters:

So long story short I have users who use unprintable characters in file and folder paths, such as  or the little floating dot.

Can anyone think of some quick way to replace all of these in every folder and sub folder and file with a normal hyphen?

There's some built-in cmdlets to do this pretty easily.

Building on @scottalanmiller's regex, I added the exclusion of punctuation characters, because in my testing, it was replacing the "dot" before the file extension. I did not go looking for a way to just exclude dots. Someone else can do that.

This line will get each item in a directory and subdirectories -Recurse, and replace any non-"your-language"alphabet character, ignoring regular aphabet/number/punctuation characters.

Here's how I'd go about it:


# Remove the -WhatIf when you are ready to make the changes.
(Get-ChildItem -Path "C:\test" -Recurse | Rename-Item -NewName {$_.Name -replace '[^\p{L}\p{Nd}\p{P}]','-'} -WhatIf)

Using the -WhatIf switch will allow the code to be ran while telling you exactly what will change, without actually doing it. Remove the -WhatIf when you are ready to make the changes.

DustinB3403

@Obsolesce That is absolutely the perfect answer.

DustinB3403

@Pete-S said in File Management removing unprintable characters:

If the backup system can't handle all allowed characters in a filename, then that is the problem that needs to be fixed.

While I would generally agree, I have no way to control the file system in my backup location, which complains about the files and just skips them.

So while I agree, use a better back tool that isn't the answer I'm looking for.

@Obsolesce this is great, thank you!