Need live network monitoring

JaredBusch

@Dashrender said in Need live network monitoring:

Transmit is higher than receive? I'm betting someone is infected... possibly with a ransomware, and the baddies might be stealing your data right now. Just a thought at least.

Can you narrow it down to what client is generating that traffic?

The long views are smoothed averages..

once afternoon hit and people start leaving it dropped off.

now at 5pm, there are still little spokes. I'm hoping to narrow it down now that the network is less busy.

jt1001001

Saw this behavior when a user installed a Canon printer at home and put the software on their work laptop. The software kept scanning for a printer. The software slammed the router but the traffic never left it.

JaredBusch

@Dashrender said in Need live network monitoring:

Transmit is higher than receive?

If you wwere commenting about the TX > RX on the device view, recall that it is TX from the router to the endpoint. aka the endpoint downloading something.

DustinB3403

Is it possibly torrent traffic? Do you have anything like zabbix on the end points and servers to see what they are doing?

JaredBusch

@JaredBusch said in Need live network monitoring:

@Dashrender said in Need live network monitoring:

Transmit is higher than receive?

If you wwere commenting about the TX > RX on the device view, recall that it is TX from the router to the endpoint. aka the endpoint downloading something.

This assumption was wrong. I jsut tested by downloading the current fedora desktop ISO to the user machine..

So user is sending data.

JaredBusch

@DustinB3403 said in Need live network monitoring:

Do you have anything like zabbix on the end points and servers to see what they are doing?

No, because they are not paying for type of ongoing work.

That said I could set it up if that is the easiest option for a hopefully one off.

DustinB3403

@JaredBusch it would be pretty quick to get setup, the servers and endpoints are going to be the most time consuming for getting each setup.

Edit installing each agent

dafyre

@JaredBusch said in Need live network monitoring:

@DustinB3403 said in Need live network monitoring:

Do you have anything like zabbix on the end points and servers to see what they are doing?

No, because they are not paying for type of ongoing work.

That said I could set it up if that is the easiest option for a hopefully one off.

If you can identify which device it is, just block it from the internet until you can get your remote tools installed on it and check it?

JaredBusch

Killed the most likely suspect machine and things calmed.

Now nothing abnormal. The big upload spike is offsite backups sending.

krzykat

@JaredBusch Sure would be nice if the Edgerouter DPI would know about off-site backups. Would have allowed you to isolate it real quick.

JaredBusch

@krzykat said in Need live network monitoring:

@JaredBusch Sure would be nice if the Edgerouter DPI would know about off-site backups. Would have allowed you to isolate it real quick.

That has nothing to do with the problem.

The problem occurred all day long, while no backups were running.
Here is the DPI for the offsite backup.
Exactly as I expect, never knew why it was classified as LE. But it always has been.

JaredBusch

So the problem is ZeroTier 1.6.0
It was pushed to chocolately on November 22.

All of the client machines updated on the 22 & 23 automatically via chocolatey.

A branch location called today reporting issues and their only have 6 computers. It was much easier to analyze.

Only 2 computers were causing problems. The only thing the same on those computers but different from the rest was ZeroTier. Stopped the ZeroTier service and poof, network normal.

So I hopped in to ScreenConnect and globally stopped the service at the main office.

#!ps
#maxlength=200000
#timeout=90000
Stop-Service -Name ZeroTierOneService

Magic. Things are normalized. Now to troubleshoot ZeroTier.

dbeato

Found this and might be helpful to this traffic issue. In a Windows Computer you could use Glasswire on a wim to find out what traffic is going out of it:
https://github.com/zerotier/ZeroTierOne/issues/1174
https://github.com/zerotier/ZeroTierOne/issues/1018
https://github.com/zerotier/ZeroTierOne/issues/867