@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

What do the event logs say?

Nothing for the offending jobs. I've been scouring the "Operational" event log for Task Scheduler and I see every time the working jobs are fired, and even when jobs are not fired because they're still running (which are expected for some as they run every minute, but sometimes certain maintenance jobs cause them to take a little longer). But the jobs that aren't running...just radio silence. It's as if they're being ignored.

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

I am having an interesting problem with the Windows Task Scheduler under Server 2012. I have a couple dozen scheduled tasks that do various things via PowerShell scripts (mostly pull data from a SQL database then ship it off to their respective recipient). These scripts have been in place and scheduled for years. However, recently, about a half dozen of the scheduled tasks have plain stopped triggering and I have no idea why.

For example, I have a scheduled task that's set to run at 9PM every Monday, Tuesday, Wednesday, Thursday, Friday of every week. It's "Next Run Time" is tonight at 9PM. It's "Last Run Time" was 1/24/2020 at 9:01:20 PM. Whaa?? The "Last Run Result" was "The operation completed successfully. (0x0)".

I can right click the offending jobs and run them manually no problem. I've "poked" at the scheduled tasks, enabled/disabled them, the trigger, changed the time back/forth, even changed the "Configure for" option from the default "Windows Vista, Windows Server 2008" to "Windows Server 2012". One thing I have not tried exporting/importing the offending tasks.

I'm curious if any of y'all have experienced anything like this?

If the task doesn't complete, it will not run again. Also the task scheduling service have to be running for any tasks to be triggered.

Right. I have "Stop the task if it runs longer than" set to 3 days. So I should see them in the "Running" state if that were the case, no?

Depends on if they are running right now. Anyway, how long does the task take to complete? Do your scripts produce a log file?

They are all "Ready" as of this moment. They run nightly, so I should see them as "Running" right now.

Some of them do, some of them do not (I've been adding Start/Stop Transcript to them as I've needed to troubleshoot). For those that do log, the log's last modified date/time corresponds to the last time the Task Scheduler reports the task ran.

I just edited my original post to add that the script I talk about as an example does log, and the log has not been touched since Task Scheduler reported it ran successfully.

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

I am having an interesting problem with the Windows Task Scheduler under Server 2012. I have a couple dozen scheduled tasks that do various things via PowerShell scripts (mostly pull data from a SQL database then ship it off to their respective recipient). These scripts have been in place and scheduled for years. However, recently, about a half dozen of the scheduled tasks have plain stopped triggering and I have no idea why.

For example, I have a scheduled task that's set to run at 9PM every Monday, Tuesday, Wednesday, Thursday, Friday of every week. It's "Next Run Time" is tonight at 9PM. It's "Last Run Time" was 1/24/2020 at 9:01:20 PM. Whaa?? The "Last Run Result" was "The operation completed successfully. (0x0)".

I can right click the offending jobs and run them manually no problem. I've "poked" at the scheduled tasks, enabled/disabled them, the trigger, changed the time back/forth, even changed the "Configure for" option from the default "Windows Vista, Windows Server 2008" to "Windows Server 2012". One thing I have not tried exporting/importing the offending tasks.

I'm curious if any of y'all have experienced anything like this?

If the task doesn't complete, it will not run again. Also the task scheduling service have to be running for any tasks to be triggered.

Right. I have "Stop the task if it runs longer than" set to 3 days. So I should see them in the "Running" state if that were the case, no?

Depends on if they are running right now. Anyway, how long does the task take to complete? Do your scripts produce a log file?

They are all "Ready" as of this moment. They run nightly, so I should see them as "Running" right now.

Some of them do, some of them do not (I've been adding Start/Stop Transcript to them as I've needed to troubleshoot). For those that do log, the log's last modified date/time corresponds to the last time the Task Scheduler reports the task ran.

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

I am having an interesting problem with the Windows Task Scheduler under Server 2012. I have a couple dozen scheduled tasks that do various things via PowerShell scripts (mostly pull data from a SQL database then ship it off to their respective recipient). These scripts have been in place and scheduled for years. However, recently, about a half dozen of the scheduled tasks have plain stopped triggering and I have no idea why.

For example, I have a scheduled task that's set to run at 9PM every Monday, Tuesday, Wednesday, Thursday, Friday of every week. It's "Next Run Time" is tonight at 9PM. It's "Last Run Time" was 1/24/2020 at 9:01:20 PM. Whaa?? The "Last Run Result" was "The operation completed successfully. (0x0)".

I can right click the offending jobs and run them manually no problem. I've "poked" at the scheduled tasks, enabled/disabled them, the trigger, changed the time back/forth, even changed the "Configure for" option from the default "Windows Vista, Windows Server 2008" to "Windows Server 2012". One thing I have not tried exporting/importing the offending tasks.

I'm curious if any of y'all have experienced anything like this?

If the task doesn't complete, it will not run again. Also the task scheduling service have to be running for any tasks to be triggered.

Right. I have "Stop the task if it runs longer than" set to 3 days. So I should see them in the "Running" state if that were the case, no?

@Dashrender said in Windows Server 2012 - Task Scheduler Issue:

when was the last reboot?

Last Thursday, believe it or not.

I am having an interesting problem with the Windows Task Scheduler under Server 2012. I have a couple dozen scheduled tasks that do various things via PowerShell scripts (mostly pull data from a SQL database then ship it off to their respective recipient). These scripts have been in place and scheduled for years. However, recently, about a half dozen of the scheduled tasks have plain stopped triggering and I have no idea why.

For example, I have a scheduled task that's set to run at 9PM every Monday, Tuesday, Wednesday, Thursday, Friday of every week. It's "Next Run Time" is tonight at 9PM. It's "Last Run Time" was 1/24/2020 at 9:01:20 PM. Whaa?? The "Last Run Result" was "The operation completed successfully. (0x0)".

I can right click the offending jobs and run them manually no problem. I've "poked" at the scheduled tasks, enabled/disabled them, the trigger, changed the time back/forth, even changed the "Configure for" option from the default "Windows Vista, Windows Server 2008" to "Windows Server 2012". One thing I have not tried exporting/importing the offending tasks.

I'm curious if any of y'all have experienced anything like this?

EDIT: I have this script in the example above set to log to a file. The log file has not been touched since Task Scheduler reports the task was run. So I'm pretty confident it's just plain not running these tasks for some reason

@stacksofplates Thanks! I started down the path of Logstash/Greylog/others and realized that it's going to take a bit more of my bandwidth than I can dedicate at the moment. So I ended up throwing together a BASH script that'll copy the current log files over to the share every minute. It works for now...

I personally use a Netgear LB1120 and have no complaints. It will act as a router or you can put it in bridge mode and use your own. I've used it in both configurations and it seems to work well. I've used it with AT&T, Verizon, and Ting (T-Mobile MVNO). It's integrated antennas have been fine for me, but it has the ability to connect external antennas if needed.

@dafyre I am going to go down the rabbit hole of Logstash first, actually. I can think of many other logs I'd throw at something like that. It would be a very nice add to the arsenal.

If I fail, or if folks are getting impatient, a cron job to do a simple copy will be plan B.

@Danp Wow, I can't believe I didn't think of doing a simple search for "centralized log management". Take my IT card...

Hey All,

I've been asked to set up a script (or something) that will copy the log file of one of our applications (running on CentOS 6 at the moment) over to a Windows file server. Apparently SSHing into a Linux box and grepping through logs is intimidating to some...who would've thought? Heh...

At any rate, my initial thought was to write a script that basically does a "tail -f" on the log and writes to a file on the file share, but I think this may be problematic even with using autofs to mount the Windows share.

I could just write a script that copies the entire log over every x minutes and overwrite the existing file on the share. But that's not very efficient.

Or perhaps I could do it in 2 stages. Stage 1 would be a tail -f that writes to a local file, then a second process (stage 2) takes the "tail" results and tacks it to the end of the existing file on the share every x interval. Stage 2 can ensure the file share is available, and if it's not exit and try again on the next interval.

Or, maybe there is a completely better way to approach this all together?

What are y'all's thoughts?

A quick update for y'all that are watching/participating in this thread (thank you, by the way!).

Late Friday I realized where the lockouts where coming from. We have a Windows VM that has a suite of applications that folks need to use every blue moon or so, and they access the VM via RDP. Of course, users don't log out, they just close the RDP client (I am going to fix this). The user in question had an old logon session on this VM. Killing the user's session (I just rebooted the VM) seems to have done the trick.

Now the goal is to better position myself for the next time this happens. I also figure it's probably not a bad idea to have more visibility on account lockouts and where they are coming from in general.

@PhlipElder said in Active Directory - Finding Source Of Repeated Lockouts:

@anthonyh said in Active Directory - Finding Source Of Repeated Lockouts:

I have an account that is being repeatedly locked out. The user recently changed their password, so I'm pretty sure there is something out there still trying to authenticate using their old credentials. I've been analyzing the Security log on both of our DCs (Server 2016), but it's not super helpful. I see Audit Failures, but these are attempts after the account has become locked. I am having a hard time finding the login event that triggers the lockout. Various articles I've read say to look for event 4740, but these don't seem to exist in the Security log on either DC. Is there some additional logon auditing I need to enable via GPO?

Some resources for you:

https://community.spiceworks.com/how_to/48758-trace-the-source-of-a-bad-password-and-account-lockout-in-ad
https://jackstromberg.com/2013/03/finding-the-source-to-something-that-keeps-locking-a-domain-user/
https://support.microsoft.com/en-us/help/4469275/introduction-to-the-account-lockout-and-management-tools
https://www.microsoft.com/en-us/download/details.aspx?id=15201 (LockoutStatus.EXE tool)
https://www.netwrix.com/account_lockout_examiner.html

Funny story, I downloaded the Account Lockout Tools from Microsoft and have been using LockoutStatus to track when this user's account became locked out. However, the Security log on either DC has been less than helpful.

I downloaded the Account Lockout Examiner from Netwrix and am going to put that on a box to test drive.

@petergregg85 said in Active Directory - Finding Source Of Repeated Lockouts:

@anthonyh

Lepide have a new Account Lockout Examiner freeware that may help you on this.

Else, get help from this article which lets you how to troubleshoot account lockout issue using LockoutStatus, EventCombMT and Netlogon.

Are you sure you enabled auditing policy?
Computer Configuration > Policies → Windows Settings → Security Settings → Advanced Audit Policy Configuration → Audit Policies → Account Management: Audit User Account Management → Define → Success and Failures.

Try running on DC:

auditpol /get /category:Logon/Logoff

do you see "Account Lockout" set to Success and Failure?

Most of the time, its Active sync that i have seen locks out the user's account

Did you tried to clearing out cached credentials.

Steps to track locked out accounts and find the source of Active Directory account lockouts: https://www.lepide.com/how-to/identify-the-source-of-account-lockouts-in-active-directory.html

Here's what I get when I run auditpol /get /category:Logon/Logoff

Should I enable any other categories for auditing?

@manxam said in Active Directory - Finding Source Of Repeated Lockouts:

@anthonyh : for the next time you need a "throwaway" email account

https://temp-mail.org/en/
https://10minutemail.com/10MinuteMail/index.html

Heh...I actually ended up using temp-mail.org

@Dashrender said in Active Directory - Finding Source Of Repeated Lockouts:

@anthonyh said in Active Directory - Finding Source Of Repeated Lockouts:

@wrx7m said in Active Directory - Finding Source Of Repeated Lockouts:

Mapped network drives? Activesync device?

You can also checkout Netwrix AD lockout examiner.
https://www.netwrix.com/account_lockout_examiner.html

Used it in the past and found it was usually those two.

I came across that tool, but I'm hesitant to give them my email address. I get enough marketing nonsense as it is, haha.

GOOD CALL!

make a google account and use that. The tool is worth it.

Alright, I'll do that.

@wrx7m said in Active Directory - Finding Source Of Repeated Lockouts:

Mapped network drives? Activesync device?

You can also checkout Netwrix AD lockout examiner.
https://www.netwrix.com/account_lockout_examiner.html

Used it in the past and found it was usually those two.

I came across that tool, but I'm hesitant to give them my email address. I get enough marketing nonsense as it is, haha.

@black3dynamite said in Active Directory - Finding Source Of Repeated Lockouts:

https://silentcrash.com/2018/06/using-powershell-to-trace-the-source-of-account-lockouts-in-active-directory/

This script outputs something, but I'm not sure how to interpret it. It's, uh, odd. It shows the User Name as "S-1-5-18" and the Source Host as "[OURPDC]$"

I have an account that is being repeatedly locked out. The user recently changed their password, so I'm pretty sure there is something out there still trying to authenticate using their old credentials. I've been analyzing the Security log on both of our DCs (Server 2016), but it's not super helpful. I see Audit Failures, but these are attempts after the account has become locked. I am having a hard time finding the login event that triggers the lockout. Various articles I've read say to look for event 4740, but these don't seem to exist in the Security log on either DC. Is there some additional logon auditing I need to enable via GPO?

@black3dynamite But why would I see some not elevated?

I checked with @jrc and he ran it as a Domain Admin but not elevated.

I am confused, but since this works it's time to move on...haha.