Fail2ban on load balancer



  • Anybody using fail2ban on a nginx load balancer? I've been using it on nginx on my personal server, and it's been jailing bots quite well.

    I'm thinking about trying it on load balancers in a test environment for aws.



  • @IRJ said in Fail2ban on load balancer:

    Anybody using fail2ban on a nginx load balancer? I've been using it on nginx on my personal server, and it's been jailing bots quite well.

    I'm thinking about trying it on load balancers in a test environment for aws.

    no, only because I have never taken the time to learn how to set it up well.



  • @JaredBusch said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    Anybody using fail2ban on a nginx load balancer? I've been using it on nginx on my personal server, and it's been jailing bots quite well.

    I'm thinking about trying it on load balancers in a test environment for aws.

    no, only because I have never taken the time to learn how to set it up well.

    This is how I have done it on my web server and it works well. I am constantly banning not only ssh , but also scripting, bots, etc.

    https://hostpresto.com/community/tutorials/how-to-secure-nginx-using-fail2ban-on-centos-7/

    It seems to work on a single, low traffic web server. I am not sure how well it would handle significant traffic and create a possible bottleneck on a load balancer that is handling many requests.

    It looks like it is possible to host behind a load balancer as well.

    https://github.com/fail2ban/fail2ban/issues/2258



  • Fail2 ban is pretty powerful. I find these could be useful in addition to the 6 nginx jails above.

    Geo-Blocking (I can hear @scottalanmiller moaning)

    https://thecustomizewindows.com/2016/11/fail2ban-geoip-action-script-block-ssh-country/

    SQL Injection and Shellshock blocking

    https://www.booleanworld.com/blocking-bad-bots-fail2ban/

    Wordpress specific blocking

    https://www.linode.com/docs/security/using-fail2ban-for-security/



  • @IRJ said in Fail2ban on load balancer:

    Geo-Blocking

    I get why/how this can be circumvented, but I don't know why you wouldn't just do it. Looking at a new WP site we just setup, I see in wordfence that it has been hit hundreds of times from Russia in the last week.



  • Oh this is really cool. Block IPs based on community reputation using fail2ban. You can also contribute to community as well.

    https://www.howtoforge.com/tutorial/protect-your-server-computer-with-badips-and-fail2ban/



  • @wrx7m said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    Geo-Blocking

    I get why/how this can be circumvented, but I don't know why you wouldn't just do it. Looking at a new WP site we just setup, I see in wordfence that it has been hit hundreds of times from Russia in the last week.

    Yeah. I have a situation where it is a requirement, but it does seem that a majority of the script kiddies overseas don't change their IPs



  • @IRJ said in Fail2ban on load balancer:

    Yeah. I have a situation where it is a requirement, but it does seem that a majority of the script kiddies overseas don't change their IPs

    The world is running out of IPv4 and IPv6 is a pain in the ass 😛



  • @wrx7m said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    Geo-Blocking

    I get why/how this can be circumvented, but I don't know why you wouldn't just do it. Looking at a new WP site we just setup, I see in wordfence that it has been hit hundreds of times from Russia in the last week.

    Because it blocks lots of legitimate traffic, too. WordFence is a great example, businesses often lose customers who think that their website are down because their geo-blocking blocked them. It can be because your customers are traveling, work from hosted machines, or just get mis-identified. About 50% of the time when I am in Dallas (no VPNs, no remote, just using my local desktop normally) I am identified as being in Alberta, Canada. Thousands of miles away in a different region, in a different country. When I worked in NYC, I was always showing up as Frankfort, Germany to detection devices.

    So there are extremely good reasons why you don't want to geo-block by default. You have to remember that "geo-blocking" means "blocking IP ranges", not "blocking people by country." Those IP ranges have tendencies to be associated with certain countries, but nothing more. It's not that people doing nefarious things can easily get around it that is the problem, that's a red herring. The real issue is presenting your services as being down, randomly, to some small subset of your potential or current customers. I've literally mistaken businesses as having gone out of business and switched vendors for this reason before - lots of people probably do all of the time. It's rare that I take the time to understand and look into "outages" and sometimes figure out that it was geo-blocking and not a full outage causing the site to appear as offline undefinitely.



  • @IRJ said in Fail2ban on load balancer:

    Oh this is really cool. Block IPs based on community reputation using fail2ban. You can also contribute to community as well.

    https://www.howtoforge.com/tutorial/protect-your-server-computer-with-badips-and-fail2ban/

    That's pretty cool.



  • @wrx7m So the real question for geo-blocking is...

    Pro: Blocks attacks from script kiddies and other annoying but essentially non-threatening entities.
    Con: Blocks a small number of potential customers, likely causing them to not do business with you (because they don't know that they can.)

    So put a dollar value on each, or ask your CFO/CEO to do so. How much is the value of the annoyance versus how much is the value of the lost revenue?

    Both numbers are small. For me, I can't even imagine the second number, it's got to be tiny, but I can't estimate it. The first number I can estimate, approaching zero. Script kiddie attacks, those that can't change their IPs and aren't targeted, don't pose a real threat. They use a trivial amount of bandwidth, fill up some logs, but that's it. They don't impact me. So while I can't put a number of the "con", I know that there is one even if it is $5. But I can put a value on the "pro", and it is essentially zero. So for me, that's a crystal clear "don't take this risk without another factor" answer.



  • @wrx7m False positives too. If I blocked all Canada traffic as an example, LogMeIn will intermittently not work because they have data centers in Canada. There is essentially no benefit to it and a few, mostly small downsides. It just aids sale because it sounds cool.



  • @wirestyle22 said in Fail2ban on load balancer:

    @wrx7m False positives too. If I blocked all Canada traffic as an example, LogMeIn will intermittently not work because they have data centers in Canada. There is essentially no benefit to it and a few, mostly small downsides. It just aids sale because it sounds cool.

    That's not what's being talked about here. We are talking about doing in on a load balancer for a specific service. In this case https traffic to a specific application.



  • @IRJ said in Fail2ban on load balancer:

    @wirestyle22 said in Fail2ban on load balancer:

    @wrx7m False positives too. If I blocked all Canada traffic as an example, LogMeIn will intermittently not work because they have data centers in Canada. There is essentially no benefit to it and a few, mostly small downsides. It just aids sale because it sounds cool.

    That's not what's being talked about here. We are talking about doing in on a load balancer for a specific service. In this case https traffic to a specific application.

    I was talking specifically about potential problems with geo-location. I understand what is being discussed



  • @wirestyle22 said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    @wirestyle22 said in Fail2ban on load balancer:

    @wrx7m False positives too. If I blocked all Canada traffic as an example, LogMeIn will intermittently not work because they have data centers in Canada. There is essentially no benefit to it and a few, mostly small downsides. It just aids sale because it sounds cool.

    That's not what's being talked about here. We are talking about doing in on a load balancer for a specific service. In this case https traffic to a specific application.

    I was talking specifically about potential problems with geo-location. I understand what is being discussed

    If you do geo-blocking then it becomes part of the process of installing new software and services to add a whitelist of IPs for that service. Any cloud hosting provider has something similar to this.

    https://help.logmein.com/articles/en_US/FAQ/Whitelisting-and-LogMeIn



  • @scottalanmiller said in Fail2ban on load balancer:

    @wrx7m So the real question for geo-blocking is...

    Pro: Blocks attacks from script kiddies and other annoying but essentially non-threatening entities.
    Con: Blocks a small number of potential customers, likely causing them to not do business with you (because they don't know that they can.)

    So put a dollar value on each, or ask your CFO/CEO to do so. How much is the value of the annoyance versus how much is the value of the lost revenue?

    Both numbers are small. For me, I can't even imagine the second number, it's got to be tiny, but I can't estimate it. The first number I can estimate, approaching zero. Script kiddie attacks, those that can't change their IPs and aren't targeted, don't pose a real threat. They use a trivial amount of bandwidth, fill up some logs, but that's it. They don't impact me. So while I can't put a number of the "con", I know that there is one even if it is $5. But I can put a value on the "pro", and it is essentially zero. So for me, that's a crystal clear "don't take this risk without another factor" answer.

    I agree with @scottalanmiller that it is hard to take someone as a serious threat when they don't even bother to change their IP address. On the flip side, it is something that IT management generally expects for whatever reason. I think a big part of comes down to the due diligence factor. It is easy for a CEO to say yeah we block China and Russia as part of our security. It sounds good on a high level which is where IT management lives.

    I can see the requirement from their point of view, though. If you are breached and by chance it happens to be from a Russian IP, what is the first thing any security firm is going to ask. Why weren't you blocking IPs from high risk locations? In this case, I think for IT management it is has value. Losing .001% of sales will likely never be noticed and definitely not something where the IT team will receive credit.



  • @IRJ said in Fail2ban on load balancer:

    @scottalanmiller said in Fail2ban on load balancer:

    @wrx7m So the real question for geo-blocking is...

    Pro: Blocks attacks from script kiddies and other annoying but essentially non-threatening entities.
    Con: Blocks a small number of potential customers, likely causing them to not do business with you (because they don't know that they can.)

    So put a dollar value on each, or ask your CFO/CEO to do so. How much is the value of the annoyance versus how much is the value of the lost revenue?

    Both numbers are small. For me, I can't even imagine the second number, it's got to be tiny, but I can't estimate it. The first number I can estimate, approaching zero. Script kiddie attacks, those that can't change their IPs and aren't targeted, don't pose a real threat. They use a trivial amount of bandwidth, fill up some logs, but that's it. They don't impact me. So while I can't put a number of the "con", I know that there is one even if it is $5. But I can put a value on the "pro", and it is essentially zero. So for me, that's a crystal clear "don't take this risk without another factor" answer.

    I can see the requirement from their point of view, though. If you are breached and by chance it happens to be from a Russian IP, what is the first thing any security firm is going to ask. Why weren't you blocking IPs from high risk locations?

    Its become a CYA product and that's about it.



  • @IRJ said in Fail2ban on load balancer:

    @wirestyle22 said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    @wirestyle22 said in Fail2ban on load balancer:

    @wrx7m False positives too. If I blocked all Canada traffic as an example, LogMeIn will intermittently not work because they have data centers in Canada. There is essentially no benefit to it and a few, mostly small downsides. It just aids sale because it sounds cool.

    That's not what's being talked about here. We are talking about doing in on a load balancer for a specific service. In this case https traffic to a specific application.

    I was talking specifically about potential problems with geo-location. I understand what is being discussed

    If you do geo-blocking then it becomes part of the process of installing new software and services to add a whitelist of IPs for that service. Any cloud hosting provider has something similar to this.

    https://help.logmein.com/articles/en_US/FAQ/Whitelisting-and-LogMeIn

    For what real benefit though? That is kind of my point. It's a PITA but also doesn't benefit you really



  • @wirestyle22 said in Fail2ban on load balancer:

    @wrx7m False positives too. If I blocked all Canada traffic as an example, LogMeIn will intermittently not work because they have data centers in Canada. There is essentially no benefit to it and a few, mostly small downsides. It just aids sale because it sounds cool.

    That's not false positive, that's intentionally blocking those services. If they are really in Canada, geo-blocking Canada is because you don't want those services, no matter how "legit" they are.



  • @IRJ said in Fail2ban on load balancer:

    @wirestyle22 said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    @wirestyle22 said in Fail2ban on load balancer:

    @wrx7m False positives too. If I blocked all Canada traffic as an example, LogMeIn will intermittently not work because they have data centers in Canada. There is essentially no benefit to it and a few, mostly small downsides. It just aids sale because it sounds cool.

    That's not what's being talked about here. We are talking about doing in on a load balancer for a specific service. In this case https traffic to a specific application.

    I was talking specifically about potential problems with geo-location. I understand what is being discussed

    If you do geo-blocking then it becomes part of the process of installing new software and services to add a whitelist of IPs for that service. Any cloud hosting provider has something similar to this.

    https://help.logmein.com/articles/en_US/FAQ/Whitelisting-and-LogMeIn

    In that case, and I know there are places where the middle ground could make sense, does it make sense to geo-block but leave the main attack networks open? Rather than just block universally, and whitelist only what is wanted?



  • @IRJ said in Fail2ban on load balancer:

    On the flip side, it is something that IT management generally expects for whatever reason.

    It's something random non-technical people sometimes given power over IT want for some reason. Not people looking out for security, IT, or business interests, generally. It's the people who are trying to "do something to look like they are doing their job in meetings".



  • @IRJ said in Fail2ban on load balancer:

    It sounds good on a high level which is where IT management lives.

    Only in bad companies / gov't. No healthy IT works that way. No, granted, the majority of any field is absolute crap.



  • @IRJ said in Fail2ban on load balancer:

    Losing .001% of sales will likely never be noticed and definitely not something where the IT team will receive credit.

    This technique is also used by social engineers... the problem with lost sales is that they are never noticed. Silently turning away customers is almost always safe, and often IT management will throw away sometimes big numbers of sales (or similar) because it is impossible to detect. They do it because they are lazy, or sometimes malicious.

    At one F500 I was at, HR was doing this... turning away candidates before they recorded them in the system, so the company management had no idea that most job candidates were being told not to apply before we even stored their resumes! What seems like it would have been a tiny number turned out to be the majority and it crippled hiring practices.

    Not that geo-blocking will ever hit 50%, but it might be numbers far higher than people will guess, and there is no way to ever know. It's always a mystery.



  • @pmoncho said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    @scottalanmiller said in Fail2ban on load balancer:

    @wrx7m So the real question for geo-blocking is...

    Pro: Blocks attacks from script kiddies and other annoying but essentially non-threatening entities.
    Con: Blocks a small number of potential customers, likely causing them to not do business with you (because they don't know that they can.)

    So put a dollar value on each, or ask your CFO/CEO to do so. How much is the value of the annoyance versus how much is the value of the lost revenue?

    Both numbers are small. For me, I can't even imagine the second number, it's got to be tiny, but I can't estimate it. The first number I can estimate, approaching zero. Script kiddie attacks, those that can't change their IPs and aren't targeted, don't pose a real threat. They use a trivial amount of bandwidth, fill up some logs, but that's it. They don't impact me. So while I can't put a number of the "con", I know that there is one even if it is $5. But I can put a value on the "pro", and it is essentially zero. So for me, that's a crystal clear "don't take this risk without another factor" answer.

    I can see the requirement from their point of view, though. If you are breached and by chance it happens to be from a Russian IP, what is the first thing any security firm is going to ask. Why weren't you blocking IPs from high risk locations?

    Its become a CYA product and that's about it.

    Problem is, it covers your ass to clueless management. But it exposes your ass to finance, marketing, etc. If someone who really has a clue goes to the business and says that you might have thrown away customers because you don't like people who live in, work from, or are detected as being in certain countries... that could be some big expose. So it's not a standard CYA where you do it to be safe, it's a "pick one CYA or the other." In any company I've worked in, it would be the opposite, you'd avoid geo-blocking to CYA. In banking, as a big example, it would violate SEC regulations as it is illegal to block communications channels in that way (assuming you did it universally) for trading purposes. I've seen people do it accidentally and be told that they had minutes to reverse it before looking at jail time. That's a dramatic example, but shows that even government regulations can make avoiding geo-blocking a CYA exercise in many cases.



  • @wirestyle22 said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    @wirestyle22 said in Fail2ban on load balancer:

    @IRJ said in Fail2ban on load balancer:

    @wirestyle22 said in Fail2ban on load balancer:

    @wrx7m False positives too. If I blocked all Canada traffic as an example, LogMeIn will intermittently not work because they have data centers in Canada. There is essentially no benefit to it and a few, mostly small downsides. It just aids sale because it sounds cool.

    That's not what's being talked about here. We are talking about doing in on a load balancer for a specific service. In this case https traffic to a specific application.

    I was talking specifically about potential problems with geo-location. I understand what is being discussed

    If you do geo-blocking then it becomes part of the process of installing new software and services to add a whitelist of IPs for that service. Any cloud hosting provider has something similar to this.

    https://help.logmein.com/articles/en_US/FAQ/Whitelisting-and-LogMeIn

    For what real benefit though? That is kind of my point. It's a PITA but also doesn't benefit you really

    The point is, it's about politics. The value has nothing to do with IT when it does have value. Any (or essentially any) value comes from unhealthy organizations driven by politics and not by results. Which, as I've said many times, is the average company.


Log in to reply