395 comments
  • BLKNSLVR1y

    I self-host a (non-critical) mail server and a few other things and occasionally look at live firewall logs, seeing the constant flow of illegitimate traffic hitting random ports all over the place, some hitting legitimate service ports but others just probing basically anything and everything. I decided to setup a series of scripts that detect activity on ports that aren't open (and therefore there's no legitimate reason for the traffic to exist) and block those IP addresses from the service ports since the traffic source isn't to be trusted.

    Something that came out of analysis of the blocked IP addresses was that I discovered a few untrustworthy /24 networks belonging to a bunch of "internet security companies" whose core business seems to depend on flooding the entire IPv4 space with daily scans. Blocking these Internet scanner networks significantly reduced the uninvited activity on my open service ports. And by significantly I mean easily over 50% of unwanted traffic is blocked.

    Network lists and various scripts to achieve my setup can be found here: https://github.com/UninvitedActivity/UninvitedActivity

    Internet Scanner lists are here: https://github.com/UninvitedActivity/UninvitedActivity/tree/...

    Large networks that seem responsible for more than their fair share of uninvited activity are listed here: https://github.com/UninvitedActivity/UninvitedActivity/tree/...

    I'm semi-aware of the futility of blocking IP addresses and networks. I do believe, however, that it can significantly reduce the load on the next layers of security that require computation for pattern matching etc.

    Be aware: there are footguns to be found here.

    • TacticalCoder1y

      One thing I do is I blocklist entire countries' and regional ISP' CIDR blocks. Believe it or not: straight to firewall DROP.

      China, North Korea, so many african countries who's only traffic is from scammers, tiny islands in the pacific that are used for nothing but scamming...

      Straight to DROP.

      And I do not care about the whining.

      • mmsc1y

        Had a travel insurance do this and when I was in hospital in Asia I couldn't start a claim and the hospital nearly kicked me out. I'm sure the sysadmins thought it was a great way to reduce hacking attempts by blocking Asia.

        • O5vYtytb1y

          That's so remarkably stupid for travel insurance, it's unbelievable.

          • mmsc1y

            I wrote a cynical take on "how it happened" at the time: https://joshua.hu/losing-sight-vision-mission-of-your-role

            I think it comes from the divorce of what people are hired to do versus what their work actually contributes to. I also remember the countless cloudflare turnstiles that I've had to get through one way or another on airlines' websites which reset every minute (looking at you, airserbia, for being the worst).

        • dahart1y

          If there’s one single business that I might expect to honor traffic from foreign countries, it would be the travel industry. I can suddenly envision using a VPN to route through Asia and check a travel agent’s site access before purchasing.

        • steelframe1y

          Why couldn't they fix this with a phone call? So much suckage.

          Mental note 1: Verify whether I can file a claim over the phone before I give a travel insurance company my money.

          Mental note 2: Don't travel without being able to VPN through a U.S. endpoint. Preferably something sitting in my house.

          • mmsc1y

            Mental note 3: Verify you can call international numbers?

        • egorfine1y

          GP said he does "not care about the whining".

        • lopkeny12ko1y

          Ironic that GP commenter said "I do not care about the whining" about regional IP blocks and the first reply is just someone whining about it.

        • boredtofears1y

          That’s awful but why is the onus on random sys admins around the world to deal with this correctly and not the government hosting the problem entities?

          • AJayWalker1y

            I would say because it’s their job to serve their customers, even if they’re abroad? Especially for a travel insurance company.

          • account421y

            You don't think a travel agency selling policies covering china should have their sysadmins ensure that their customers can actually make use of those policies? They can always explicitly exclude china if they don't want to deal with this but then they wouldn't have gotten GP's money.

          • Thorrez1y

            It's not a random sysadmin. It's a sysadmin of a travel insurance company.

          • krsdcbl1y

            if the government in question is supportive of said problem entities, they won't "deal" with it

            If the government in question has free reign on regulating said traffic, it's an avenue for repressions and censorship

            Otherwise it's a legal matter to seek action against such entities, which is already how it works

            (... but I'm afraid we're actually mostly talking about "scenario 1 entities" here, which makes it futile to seek action from the very offices that already play a role in making it harder to use existing legal means)

          • kjkjadksj1y

            Government needs lobbying to act

          • belk1y

            That's like asking why don't we expect burglars to not burgle, they won't, but that doesn't mean walling off a whole neighborhood is the solution either.

            • tracker11y

              You haven't seen new construction in many upper end places then... High exterior walls and gated entry. Not that it adds much practically.

      • grishka1y

        As a Russian, I hate it when people do this. It's extremely annoying when you just click some random interesting-looking link from HN or Reddit or Twitter only to be greeted by a 403 or a connection timeout. Then you turn your VPN on, and magically, it loads just fine.

        • __turbobrew__1y

          For many services, the expected value of letting people from Russia access their service is negative. The reality is that Russia contributes a large portion of hacking attempts while providing very little to no revenue for the service. At the end of the day it is just business, and sometimes letting countries access your service is bad for the bottom line.

          • pcthrowaway1y

            I think you and the person above you can both have valid concerns at the same time. If someone said "~50% of theft is from <insert minority group> while they only account for 5% of my business, so I'm not going to let them in the door", assuming the absence of social and legal consequences which would realistically occur, it could be argued that it's the right move for their "bottom line" or whatever. Does that mean it's right, or good, or equitable?

            Of course at the same time, if you hold yourself to a much higher standard than what's socially or legally acceptable, there's the inevitable fact that your competitors aren't. So it's a fine balance.

            • tracker11y

              If <minority group> is covered by the same jurisdiction as <business>, then it's not close to a 1:1 comparison.

              It's perfectly reasonable to not do business with people in countries that support piracy. And I'm referring to the Arrg/EyePatch type and the Buh/KeyboardWarrior type. In the end, it's a choice. If you don't have a legal means to deal with illicit activity, and blocking mostly works, there you go.

        • tiberious7261y

          Your country is a bad global citizen. If they started taking action against the groups trying to break into my systems every minute of every day then I wouldn't need to block the entire jurisdiction.

          Geoblocking all sanctioned countries was the best thing I ever did

        • snapplebobapple1y

          Your annoyance is a feature, not a bug. You are supposed to get annoyed enough as a group to lobby your government to fight the internal problem

          • grishka1y

            You're very naive to assume that this government takes any feedback.

            I'll just leave this thread here: https://twitter.com/IrineKuklina/status/1578339408801304580

            • hellojesus1y

              I am powerless to prevent even my local county from voting to steal my income to fund nonsense welfare, so I can only imagine how much less hope you have for political change and in your ability to meaningfully enact any.

              Good luck, and I hope you stay out of harm's way.

              • account421y

                How do you think any political change was ever achieved then?

                • hellojesus1y

                  Anyone can attempt political change, but it all comes down to EV.

                  I live in the US. I can openly speak my mind with relative safety. And I mean relative. My physical safety will likely not be risked, nor the physical safety of my family. But we are very much at a stage where any dissent is accompanied by internet mobs and unemployment.

                  Do I think that I can convince > 50% of voters in my county to rescind a 1% tax on my household income over $200k? Unlikely. Near zero probability. And my guess is that that probability is certainly less than the probability I am called a racist, transphobe, white supremacist. And that may reduce my income to $0. The EV play doesn't make sense when I have children to raise.

                  I imagine the above weighted by an openly corrupt gov willing to imprision and kill further diminishes the EV for an individual.

              • digitity1y

                [flagged]

            • justinclift1y

              Page doesn't exist?

            • alandarev1y

              Sorry, can't access, I'm from sanctioned country

            • account421y

              They would take feedback the same way Napoleon did.

            • snapplebobapple1y

              you are naive to think whether your government takes feedback is relevant or not (or that I was specifically talking about Russia, That is just one of many countries with shitty internet crime prevention that are routinely blocked and each of those shite countries have varying levels of shite leadership with varying levels of responsiveness).

            • type01y

              oh but it does, you can submit it directly to Roskomnadzor so it can cooperate with said hackers and then GRU might even hire them directly /s

          • nullifidian1y

            Ah, yes, the remaining English speakers in Russia will overthrow the literal millions of the silovik class whose entire job is to repress (with violence) any independent political activity. There is no "lobbying" in Russia, if you didn't know.

            If you hate all Russians just say you hate all Russians. No need for this "lobby your government" euphemistic BS.

            • sqeaky1y

              We in the west can't change your government to ban hacking requests.

              We can block whole countries and make a practical reduction in hacks. Sorry that you got caught in the middle and feel you have no options.

              Maybe someone who does have options and makes their money from non-hacking will be inconvenienced and ask for change instead.

            • wredcoll1y

              So political change in russia is literally impossible and everything will be exactly the same 50 years from now?

              Obviously not. Is such change easy? Again, obviously not, but the only way countries change is their own citizens wanting to make the change.

              • nullifidian1y

                >So political change in russia is literally impossible

                Precisely. It's basically impossible. There has to be at least be a generational change, or a severe economic / military loss if we are talking about this decade, but even that isn't a guarantee since the system is perpetuating itself with force, with economic self-interest to continue doing so. Isolating Russian citizens from western sources of information (in addition to what the Russian government is already doing by itself) is not only not helping, it's counterproductive, since rejection engenders a rejection in return, lowering the probability that an inflection point in the Russian history would result in anything western.

                >countries change

                Authoritarian countries change when their enforcement class relaxes and loses control. It takes decades for it to occur. If there is no relaxation, then no change occurs, as demonstrated by numerous countries, not only Russia. Right now the control and propaganda are very tight. "Wanting to make change" publicly is literally a life-threatening activity.

              • grishka1y

                Oh we do want to make this change. Desperately. The only minor issue with that is that we lack any means to do so. I'll be sure to do my part as soon as the window of opportunity opens.

                • brokenmachine1y

                  It's probably risky, but absolutely there's a means to do so.

                  Be the change you want to see in the world. Change happens slowly at first, and then all at once.

                • firesteelrain1y

                  Sure hope your govt is not monitoring your posts

            • dead5ea1y

              The idea that Westerners might "hate" Russians (the people -- not the dictators and their regimes' activities) always seemed so silly to me that I assumed the majority of the related propaganda would be laughed off.

              In my experience, the worst general case you have from Americans is absolute "other side of the planet" indifference. Hence the apathetic practice of blocking Russian-originating IP traffic... This may be arguably worse than hate.

              A slightly better case, I think, is a healthy segment of the American populace thinks Russians are like the FPSRussia YouTube channel from a few years ago. (Disclaimer: Not sure what the status of that channel is now. Plus, I always figured he was geographically in the southern USA.)

        • mistrial91y

          people here are not thinking in whole systems-- roads have dual purpose.. there is security AND there is trade .. a world without trade is a poor world.. that includes the intellectual arts, civilian institutions cooperating, common issues like Climate.

          The voices here that say "I block everyone, don't bother me with your whining" .. it is a security practice.. OK. security is not the whole story of civilizations; obstinate thinking leads to ignorance, not evolution.

          The topic is SSH, an administrative and secured access. Yes security applies. to be on-topic

          • grishka1y

            Of course one can obfuscate and secure their own SSH access as much or as little as they want. Run sshd on a different port, require port knocking, ban IPs after failed login attempts, all that kind of stuff.

            I'm, however, specifically talking about public-facing services like HTTP(S), which also get blocked with this "I'll just indiscriminately blacklist IPs belonging to countries I don't like" approach.

            • phsau1y

              Malicious traffic is not limited to ssh and comes from the same usual suspects. Automated attacks against web applications is constant. I wouldn't say it's indiscriminate, it's practical.

            • tiberious7261y

              Yeah exactly, try running an esp VPN on a different port and see how well that works.

        • NicoJuicy1y

          Had a reddit clone. The amount of Russian spam coming in was nuts.

          Blocking the ru language blocked all spam. And since it didn't have Russian users, it was an easy choice to make.

        • EasyMark1y

          I think it’s harmless though if say it’s a business site or mail site that is only meant to do business with a subset of people, like a country or region. That said, I think it’s of highly limited value though because any hacker above Lvl 1 will know how to use a bot, remote box, or VPN from a more local IP.

        • egorfine1y

          > It's extremely annoying

          Now imagine how annoying is russian traffic to world's sysadmins. Then could you please point your finger to who's more wrong here: your government or sysadmins of the world?

      • nequo1y

        I assume you don’t host anything that could be useful to the 1.5 to 2 billion people that you’re blocking.

        • luma1y

          Or they host a business site that doesn't do business in those countries and so nothing of value is lost to them. For example, it's literally illegal for me to accept payments from .ru, so why bother wasting their time and my bandwidth?

          • ajsnigrutin1y

            I live in EU,and a bunch of american sites just block the whole EU due to GDPR laws.

            Then someone in US uses my email by accident to subscribe to some newsletter (not the first time, I also get personal emails for that person, since it's just one letter difference, and i'm guessing it's someone old, considering the emails I get), i try to click "unsubscribe", and it just redirects me to "<site> is unavailable in EU, blah blah" page, without unsubscribing.

            I make sure to report that site to every goddamn spam list possible.

            • rapind1y

              IMO replying unsubscribe should always work for marketing emails and if it doesn’t then I flag the email as spam. Nope, I’m not going to visit that tracked / info gathering unsubscribe link.

              • dheera1y

                I only use unsubscribe links from things I voluntarily and willingly subscribed to.

                If I was involuntarily subscribed to something, or subscribed because of an inconspicuous "subscribe me" checkbox that I probably didn't notice, including from a legit business that I purchased an item, it's getting reported as spam in Gmail.

                • account421y

                  This is the right approach. Usually I also avoid any future business with a company that starts spamming me.

            • DEADMINCE1y

              > a bunch of american sites just block the whole EU due to GDPR laws.

              Which is incredibly reasonable. If the EU didn't try to claim EU law applies globally, those sites might still be up.

              • robin_reala1y

                The US is just as bad at extraterritorial law, see FATCA for just one example.

                https://en.wikipedia.org/wiki/Foreign_Account_Tax_Compliance...

                • DEADMINCE1y

                  That situation is quite different. The US is using its significant power and weight to coerce those non-US banks into compliance with FACTA. Those banks don't have to comply, but they want to do business with the US and US companies, then they don't have much of a choice.

                  It's not like they just made a law and now insisted it applies globally, which is what the EU did.

              • arp2421y

                > If the EU didn't try to claim EU law applies globally, those sites might still be up.

                It doesn't; it applies to EU residents. Your non-EU business is free to do whatever it wants, but as soon as you do business with EU residents EU law applies.

                This is more or less how it works everywhere (with some exceptions).

                And deciding not to do business with EU residents (i.e. block in EU) is of course perfectly valid and reasonable choice. But not because "EU laws apply globally".

                • DEADMINCE1y

                  > It doesn't; it applies to EU residents. Your non-EU business is free to do whatever it wants, but as soon as you do business with EU residents EU law applies.

                  See, you say it only applies to EU residents, but that isn't the case.

                  The real issue is where you say but as soon as you do business with EU residents EU law applies., and, well, that's just nonsense.

                  I have a US site. I can operate my business any way I like as long as I don't break any Federal or State laws, and I can break every single EU law that doesn't have an equivalent US law.

                  The EU can't touch me. EU law doesn't apply to me, even if I advertise the hell out of my site to try and attract as many EU citizens as possible.

                  All the Eu can do is firewall me off, prosecute me if I come to the Eu and police or punish its citizens.

                  > This is more or less how it works everywhere (with some exceptions).

                  It's really not. The EUs claim of global jurisdiction is unique and a first. There may have been loosely similar things, but nothing quite like this.

                  > But not because "EU laws apply globally".

                  You should inform the EU they should correct their legislation then.

              • ajsnigrutin1y

                Sure, but if some Little Whinging news from North Arizona (fictional newssite) starts spamming me, because some grandma there can't remember his email address, and won't let me unsubscribe, I'll do everything I can do within my five minutes of anger to make them rethink.

              • 38362936481y

                What? No

                Claiming jurisdiction by server location is the stupidest thing ever if you trying to have any kind of customer protection laws. You have to go by customer location.

                However, the claim that they have jurisdiction over EU citizens abroad is very questionable.

                • talldatethrow1y

                  If a European travels to grocery store in Nevada, assuming they'd be protected by EU laws is a bit goofy.

                  If they travel to my US server digitally and want my data back, I shouldn't have to know EU laws. They came to me.

                  I guess you could argue that if I'm then willing to send them data, then I need to play the game. Like a Nevada store that ships to France.

                • jkaplowitz1y

                  > However, the claim that they have jurisdiction over EU citizens abroad is very questionable.

                  The GDPR makes no jurisdictional claims at all based on citizenship, despite a lot of inaccurate summaries saying otherwise. For those cases where the GDPR cares about individuals being EU or non-EU, it only cares about their location, not about their citizenship / nationality or their residence.

                • DEADMINCE1y

                  > Claiming jurisdiction by server location is the stupidest thing ever if you trying to have any kind of customer protection laws. You have to go by customer location.

                  I disagree, because that's impossible. That's why the EU's attempt is largely a joke. Literally - it seems to get mocked a lot when I tried reading up on the credibility and practicality of what they claim.

                  > However, the claim that they have jurisdiction over EU citizens abroad is very questionable.

                  It's the claim that they have jurisdiction over non-EU citizens and businesses in their own countries which is so laughable.

      • michaelcampbell1y

        Same here. I country-block I think 4 countries and my "not-me" ssh login attempts dropped 90+%. As I run funzies sites, I couldn't care less about the reduced legit traffic.

      • Dah00n1y

        I'd do this too except by far the most scam traffic I see are US in origin. I'm in the EU.

      • normie30001y

        > so many african countries who's only traffic is from scammers

        Which countries specifically? Asking from Africa, and not sure I've encountered this.

      • ajsnigrutin1y

        Personal page.. sure.

        Business? You're a pain to many people and don't care.

        I live in EU and many US pages just block the whole EU due to GDPR laws... then someone (by mistake) subscribes me to their newsletter, and the "unsubscribe" links leads to "this page is unavalable in EU"? I'll goddamn make sure your domain ends up on every goddamn possible antispam filter I can find.

        • jkaplowitz1y

          That's often worth an FTC complaint for a CAN-SPAM Act violation: https://www.ftc.gov/business-guidance/resources/can-spam-act...

          The FTC wouldn't accept "we didn't want to deal with GDPR" as an excuse for a business violating that law.

        • cdelsolar1y

          Why? Are they spam pages?

          • ajsnigrutin1y

            For me? Sure. I never subscribed to them. Ans the unsubscribe links doesn't work, probably illegal, although not sure if they can spam an EU citizen from usa, and which/whose/what law are they breaking.

        • DEADMINCE1y

          > I'll goddamn make sure your domain ends up on every goddamn possible antispam filter I can find.

          Honestly, individuals can't really do much to change the reputation of a domain.

          Maybe petition your representative to adjust the GDPR so they don't claim it applies globally?

          • account421y

            > Honestly, individuals can't really do much to change the reputation of a domain.

            Your hosting provider and ISP will see this differently. So will the FTC.

            > Maybe petition your representative to adjust the GDPR so they don't claim it applies globally?

            Your butthurt about the GDPR doesn't absolve you from your obligations under the CAN SPAM act.

            • DEADMINCE1y

              > Your hosting provider and ISP will see this differently. So will the FTC.

              No. They absolutely won't. Not if I'm not breaking any US laws. The EU bitching would have as much impact as a government official from say Narau doing the same. None.

              > Your butthurt about the GDPR doesn't absolve you from your obligations under the CAN SPAM act.

              No. You are misunderstanding and conflating things. My point is I can do whatever I want so long as I am in compliance with US law including CAN-SPAM, and even if I violate GDPR as much as I want (again, as long as it doesn't violate US law).

              • ajsnigrutin1y

                It's a greyzone situation, but if you started sending (for me) spam emails to me, and your unsubscribe link doesn't work, because you decided to block the whole eu from all of your services, including the unsubscribe feature, you probably are breaking the US spam laws too.

                • DEADMINCE1y

                  I agree that's likely. Then I guess it would matter what recourse the EU citizen would have. They would have to file suit in the US I would think.

      • DEADMINCE1y

        That's very computationally inefficient.

        • TacticalCoder1y

          > That's very computationally inefficient.

          It's O(1) with iptables/nftables ipsets. Moreover as I blocklist entire CIDR blocks, there aren't that many entries in those ipsets.

        • aforwardslash1y

          You can trivially maintain a list of the size of the whole ipv4 space by using bitmaps

      • ransom15381y

        [flagged]

      • tiahura1y

        The Biden administration needs to explain why they allow ISPs to import data from these countries.

        • hahajk1y

          I'm not sure I understand what you're suggesting. Are you saying that the US govt should make it illegal for people in its borders to communicate with people in those countries?

    • tomxor1y

      > and block those IP addresses from the service ports since the traffic source isn't to be trusted

      Don't get me wrong, I want to do the same, I run a lot of servers and see all the automated nonsense aimed at public servers. However, you should consider the fact that today blocking an IP is akin to blocking a street, a village or sometimes even a town. For ~better or~ worse we now live in the age of CGNAT.

      If your threat model and use case means you only care about a known subset of users with static IPs who are lucky enough to not share IPs then fair enough; but if you are running services intended for wide spread consumption you are likely blocking legitimate users without even knowing it.

      • BLKNSLVR1y

        I have thought about that and, as you say, my use-case is entirely "hobby" so there's nothing I host that's of much interest to others (if things break, which they have, it inconveniences me rather than other people).

        Having said that, the websites I host are behind Cloudflare and so port 443 allows Cloudflare's ASN, but blocks everything else. This way, any of the IP addresses that are blocked from direct access to port 443 can still access the websites, just through Cloudflare's added layer of protection.

    • nilsherzig1y

      Try running some of your blocked ips through greynoise, they usually have some interesting information about them

      • BLKNSLVR1y

        Thanks for the tip. Looks like greynoise use ipinfo.io for IP metadata.

        I use https://www.abuseipdb.com/ for any manual IP address checks, and https://hackertarget.com/as-ip-lookup/ for finding what ASN an IP address (range) is a member of. I'll check out greynoise and see what extra info may be provided.

        • reincoder1y

          I (DevRel of IPinfo) run Fail2Ban on a VM as well. Protip use the CLI.

          - The CLI has the `grepip` command that extracts all the IP addresses from a text. You do not have to parse your logs.

          - Analyze your data. After you have extracted your IP addresses from your logs, pipe them to the `summarize`, `map`, and `bulk` commands on the CLI.

          - If you are doing bulk enrichment with the `bulk` command, you can use some kind of CSV query tool like CSVtoolkit, DuckDB, or Python-Pandas.

          - Look into the ASN data. ASN data is always going to be the more interesting IP metadata for honeypots IPs. Summarize the IP addresses with the `summarize` command; it will give you a high-level report. If you want a web-shareable report, make a POST call to that endpoint. Docs: https://ipinfo.io/tools/summarize-ips

          https://github.com/ipinfo/cli

          You can always send your logs to me and ask what I think of them, and if I can find common patterns based on IP metadata. I am running our API and database services 24/7 and enjoy looking at logs. I can suggest firewall configurations based on country and ASN information provided by our free data.

    • shaky-carrousel1y

      Good idea. What I do is, I disallowed password login in my ssh server, and I permanently ban whichever address that tries to log in using a password.

      • BLKNSLVR1y

        I use a bastion host on a VPS as the only source IP address allowed to ssh into my systems, so any attempts to connect to ssh (from any IP address other than the bastion) are both blocked and logged into "the list" to be blocked from connecting to any other service ports.

      • PokestarFan1y

        I did this but added an "escape hatch" that allowed password logins from the local network only.

    • pgraf1y

      Just be aware that with your strategy “blocking 50% of unwanted traffic” means blocking non-attack traffic, as these Internet security companies are mostly legitimate. The automated attack traffic that you actually want to block is in the other half and will frequently change IPs.

      • BLKNSLVR1y

        > these Internet security companies are mostly legitimate

        This is both subjective and highly dependent upon the scope of services being run. My setup would probably progressively create more hassle than it saves as on a scale from small business to large business. For the setup I have, I quite specifically want to block their traffic.

        I'm possibly overly militant about this, but they keep databases of the results of their scans, and their business is selling this information to ... whoever's buying. I don't want my IP addresses, open ports, services or any other details they're able to gather to be in these databases over which I have no control and didn't authorise.

        To steal an oft-used analogy, they're taking snapshots of all the houses on all the streets and identifying the doors, windows, gates, and having a peek inside, and recording all the results in a database.

        I believe all of them are illegitimate. They 'do' because they can, and it's profitable. "Making the internet safer" is not their raison d'être.

        Happy for any else to form their own opinion, but this is my current stance.

        • drpixie1y

          Yes - Anyone who's FAQ answer to "How to avoid being scanned" is "We don't have an opt-out, you must block all these addresses" isn't behaving like a legit business.

          "Nice network you've got there."

          "We noticed something might be open. We're not telling you what it is."

          "It would be a pity if something happened to your business."

          "Give us lots of money."

          Sounds like a movie strong-arm thug.

        • appstorelottery1y

          Would be cool to have a "don't scan me bro" list of IP's that engage in this that we could share - is there such a thing?

          • BLKNSLVR1y

            The problem is that becomes a concentrator of IPs behind which privacy conscious individuals exist, which probably has higher value to "whoever's buying". It's a conundrum.

            • yesbabyyes1y

              It sounds like what GP is suggesting is to collect ips of all the scanners, and share the list of ips among ourselves, so we can collectively route their traffic to /dev/null.

              • BLKNSLVR1y

                aaaaah, that makes sense. See the links in my original post.

              • kjkjadksj1y

                Why not also sell the scans of scanners to the scanners customers and make a little pocket change?

          • zbentley1y

            There's a comment downthread discussing something similar; I haven't tried it though: https://news.ycombinator.com/item?id=40695179

          • dataflow1y

            You're being sarcastic, right? We did this for telephone numbers and saw how it turned out...

      • nubinetwork1y

        > these Internet security companies are mostly legitimate

        Act like a bot, get treated like a bot.

        > Just be aware that with your strategy “blocking 50% of unwanted traffic” means blocking non-attack traffic

        You don't block them forever, just enough for them to move on to someone else.

        • slt20211y

          they dont move on to someone else, they scan entire internet on a regular basis, just like gogle crawls web pages

      • wl1y

        My experience is that after blocking Censys, unwanted traffic on non-standard ports from other IP blocks has basically gone to zero. It appears to me that some bad actors are using Censys scans for targeting.

        • rolph1y

          i get similar results

      • chipdart1y

        > (...) as these Internet security companies are mostly legitimate.

        Note that you're basing your assertion on the motivation of random third parties exclusively on the fact that they exist and they are behind active searches for vulnerabilities.

      • moffkalast1y

        Lol legitimate. As legitimate as door to door salesmen. OP just put up a proverbial "no soliciting" sign.

    • k8sToGo1y

      Have you considered using crowdsec?

      • BLKNSLVR1y

        I set it up in a fairly superficial way, and there are only a handful (two or three) rules that can be applied on the free tier, and I'm a tight-ass.

        It's still running, but it doesn't seem to block much - but that might be because I didn't put enough time into "doing it properly".

      • teruakohatu1y

        Are there any downsides to crowdsec?

        • snorremd1y

          You end up sharing signals (IPs) to their crowd-sourced bad IP databases, but only get 3 free IP lists on the free plan. To get some of the bigger IP lists you need an enterprise plan at $2500 a month.

          Essentially they use the free customers to build the lists that drive their enterprise sales, which is fair enough as you get to use their free dashboard and open source software. But to me it seems they're really only targeting enterprise customers as a business.

          • Philippe_H1y

            Hi all and @snorremd, (Philippe from the CrowdSec team)

            The $2.5K / month was for enterprise, but we didn't correctly understand the need and converted it to 2 optional prices: $1K for LTS and $1K for support. This will be reflected in an update on our pricing page this week; thanks, everyone, for your patience in this matter.

            It took us time to segment our four products properly. We wanted to avoid pivoting later, as it happened to so many other open-source tools recently.

            * The Security Engine (IDS+WAF+IPS) is for everyone. (Free / MIT license, three free blocklists)

            * Its SaaS companion is made for anyone with a security engine. (Generous free tier, $31/engine/month for pro industrialization features, 3 premium blocklists + all free ones. Volume discounts avail. We'll soon merge SecOPS and enterprise plans, all features at the price of the SecOPS plan)

            * Blocklists are made for M/L entities to use. (In the range of a few ten of K$ yearly, all blocklists, unlimited)

            * The Full CTI database is intended to be used by L/XL Corps. (It contains 32 fields about ~25M IP, with industry targeted, country targeted, tech stack targeted, AS and range reputation, etc. Local replication at your place, several updates/day. 10 to 20K$ / month, depending on some parameters)

            PS: As we did for the Olympic Games 2024, we'll also give away a blocklist for the US presidential election of the most aggressive IP against US assets. With a quarter of a million machines running CS, we have a fairly good overview of this, in real-time.

            Safer together.

    • Bengalilol1y

      I was about to say out loud that it was a (kind of) relief not finding Google in your lists, then I found https://github.com/UninvitedActivity/UninvitedActivity/blob/...

      • BLKNSLVR1y

        I need to check my exact configuration, but whilst I've got 1e100 in a list, I think I've got an exception for it elsewhere.

        Ie. Whilst it's been detected as uninvited activity, it causes issues when blocked, so it's excluded from the blocking.

    • cranberryturkey1y

      Just install fail2ban.

      • WhackyIdeas1y

        For SSH, changing to a random port number resulted in zero connection attempts from bots for months on end. It seems bots just never bother scanning the full 65535 port range.

        • dizhn1y

          For most of my VMs there's no ssh running. I use wireguard to connect to a private IP. I haven't done this on the bare metal yet but I might. Though barring exploits like we had recently nobody is getting into a server with either strong passwords or certificates. Fail2ban in my eyes is a log cleaner. It's not useful for much else.

        • account421y

          > the full 65535 port range

          Note that putting SSH on a high port has security implications.

      • speleding1y

        A server with fail2ban can be DOSed by sending traffic with spoofed IP addresses, making it unavailable to the spoofed IP addresses (which could be your IP, or the IP of legitimate users).

        That is typically a bigger problem than polluting your logs with failed login attempts.

        • CreatedAccount1y

          What would spoofing the IP of a packet when the underlying protocol requires a two-way handshake accomplish?

          • ajsnigrutin1y

            With CGNAT, a prepaid sim card and some effort, you can make them block a whole legit ISP in a few days without spoofing anything.

            • tiberious7261y

              But the SIM card would need to from the particular ISP you are trying to block, otherwise you would be coming out of a different isps cgnat range, no?

              • ajsnigrutin1y

                Yeah, but many ISPs, especially smaller, have a same pool of ip addresses for all of their users in that 'region' (for whatever size and definition of a "region").

                So with some effort, reconnections from/to a mobile network and many tcp/ip connectons, you can achieve that your device is connecting to the attacked site with many different (if not all) IP addresses from the ISPs pool, and if each of those is blocked, none of the legit users (using the same IP address pool) can access those services anymore.

                Look at services like digitalocen with cheap virtual machines... even amazon... so many of their IP addresses were used for something "bad" and got blocked, that running a legit service on any of them can mean that a portion of your potential users won't be able to access them, because they'll be on some block list somewhere.

        • tiberious7261y

          Don't most isps check the source address before relaying traffic nowadays? I know at least one of mine started a few years ago (and we had no idea we were asymmetrically routing our traffic till then...)

      • hypeatei1y

        fail2ban is another layer which is susceptible to abuse and vulnerabilities. It might keep noise out of your logs but at a huge cost. I'd rather just change the SSH port to something non-standard and write it down.

        • gnuser1y

          Add it port knocking and this is how I do it. nftables ftw

    • tredigi1y

      > and block those IP addresses from the service ports since the traffic source isn't to be trusted.

      This means that you are locking out anybody using a paid VPN service, if any other customer of that same VPN service does any kind of scan.

      • BLKNSLVR1y

        Something I didn't mention in my original comment, but have mentioned in another reply somewhere, is that I have the websites running behind Cloudflare, and I allow Cloudflare's ASN into port 443 but block everything else.

        Essentially outsourcing the security of port 443 to Cloudflare.

        My use-case is "hobby / enthusiast", so I believe I'm losing nothing and the "world at large" is losing nothing from this setup. Having said that, all policies on this kind of thing need to be strongly thought about in terms of their applicability to the use-case.

        Were I running a small or even medium business, I'd probably do it exactly the same with maybe a bit more of an eye on what's being blocked and the ownership of the IP addresses, and I'd have some stats to point to on the range of sources of legitimate traffic. It'd have to be a pretty big, international business for it to cause much of an effect (although I'm talking well out of school here because I don't have anything at stake).

        Flipside, though, I have my outgoing traffic routed through a couple of different exits, and I've had to make specific rules for some websites that block traffic from VPNs and VPSs, which is annoying, so I'm not completely dismissing your point.

        Lastly, however, at all scales I'd still block the Internet Scanners for reasons I've given elsewhere. Blocking them massively cut down on the uninvited activity - again, it's not about making clean logs, but it really helped clear a lot of the noise.

  • noduerme1y

    Good grief. A couple days ago I re-enabled password logins on a server that normally only accepts private keys, just to check something from a third location, and then forgot to turn it off. Two days later the server's logs were full of thousands of failed login attempts that started a few hours after I enabled passwords and then ramped up to dozens per minute.

    Just because it didn't instantly say "Goodbye".

    I checked ip locations on the biggest offensing addresses; all were in China.

    I don't know what to call the idiocy and amorality that leads people to scan port 22 for a living (or the stupidity that leads them to guess random passwords for random usernames that don't exist), but I suppose that for every gardener there are a billion ants.

    • p_l1y

      There's a cottage industry of shitty mass-scanning attacks that continue onto getting root on badly setup fresh installs of various linux distros and drop a rootkit on them.

      Some other common targets are websites to be reused for spam (hello, Wordpress!) or to hijack things like gitlab (again to drop a rootkit.

      The rootkits are then usually used either for DDoS extortion rackets (usually against game servers, including online gambling), spam (might be less big today than it used to be), and cryptocurrency mining (from my experience mainly monero).

      One time it happened in a network I set up due to miscommunication and misunderstanding of how vendor's install scripts worked (by vendor technicians!). During investigation, we found out that this particular "kit" was sold cheaply on a chinese forum (used to be russian forums back in the day, eh), as complete package to run on Windows to attack linux hosts for DDoS botnet purposes.

    • jimbobthrowawy1y

      I always install fail2ban or something like it on servers I want to have SSH on. Really cuts down on the log volume, even if I have locked myself out occasionally. The thing about port scanning is that it's cheap as hell. There's less than 4 billion IP4 addresses and zmap can hit them all within an hour on a decent network connection.

    • mmcnl1y

      I have SSH access to my server behind a VPN. Not opening port 22 makes life a lot easier.

    • beastman821y

      The name for it is "authoritarian government"

  • simonmysun1y

    Coincidently, I recently visualized the scanners for fun by plotting them on a globe[1]. It gives a more comprehensive view of the locations and ASNs of the scanners. The demo data is generated from 1 day of logs.

    [1]: https://github.com/simonmysun/where-are-the-scanners

    Amazingly there's no request from same ASN. I believe this is because the VPS provider has a quite strict validation process, e.g. you have to upload a photo of yourself with your ID and your handwritten username, etc. I would suggest we consider the reputation or credibility of the data centers so that the data centers have the motivation of banning such users. In my case, a lot of the requests were sent from Tencent or Alibaba data centers.

  • jsiepkes1y

    If you have only public key authentication enabled with SSH I honestly don't understand why people bother with things like fail2ban. It just adds more moving parts with very little security gain.

    The real risk is a zero-day in OpenSSH and fail2ban probably isn't going to protect you from that. In that case you are better served by putting another layer of defense in front of SSH like a VPN.

    • BrandoElFollito1y

      fail2ban is the kind of pseudo-security applied just because someone's cousin mentioned that in his blog.

      It provides zero security. If your endpoint uses default usernames you will be shot anyway because of IP spread. If your security is good you add something that will block your legitimate connection when you are in the middle of nowhere and, shit, cannot access your <some service>.

      • d-z-m1y

        "security" is a term that has to be defined in relation to a threat model. If your threat model is an attacker with a static IP hammering your server, fail2ban does provide some security against that sort of attacker.

        • BrandoElFollito1y

          No it does not. If the packet is at your door it is too late already. Then either it does not matter in which case you do nothing, or it matters (DoS) and then you have other problems.

          You are right that security works in the context of a threat model. There are however useless tools that give a false sense of "security" that do not fit in any reasonable model.

          I have cases where I block whole ranges of IPs for "legal" reasons - it does not make sense but there you are, the ones who write the rules are not the ones who actually know the stuff.

          • d-z-m1y

            > No it does not. If the packet is at your door it is too late already.

            Too late for what? Again, it only makes sense to talk about "security" in the context of a threat model. You can debate the reasonableness of that threat model, but that's another discussion.

            My threat model(for the sake of argument :^)) is an attacker with a static public IP address trying to bruteforce access to my service via repeated login attempts.

            I'll maintain(for now) that fail2ban can be an effective tool that does provide some security against an attacker of this kind.

            • BrandoElFollito1y

              You wrote that someone is hammering your IP. This was for me the definition of a DoS. Nothing on your side will mitigate that.

              But it does not really matter anyway. Your threat model is a single IP attacking you. What are you concerned about? That they will find services that are exposed and attack them? You should be securing these.

              You will never be attacked by one IP. The exact same attack will be done from many, many IPs and you do not want to defend against IPs attacking you, but against them exploiting a vulnerability on your side.

              Of course there is the "why not an extra layer of protection". This is great when you want to obscure something (moving a port for instance) because this does not have an effect on your system. Just imagine what happens when fail2ban goes south and blocks all addresses, or half of them, or yours because you tried too many times. This is a moving part that is actually dangerous.

        • SahAssar1y

          If your server is on the internet with a public ssh server then it is probably providing some sort of internet service. That internet service is almost always easier to DoS than your openSSH server. If you are not providing a internet service then why is your SSH open to the internet?

          • kloop1y

            > If you are not providing a internet service then why is your SSH open to the internet?

            So that I can ssh into it from various places and do stuff on my home server from elsewhere

            • SahAssar1y

              So you are accessing that server's services from some network, why are you not only allowing SSH over that network?

              Or, if your service is open to the internet then why does not what I said above hold true?

              • kloop1y

                I guess I am technically, but only for myself

                • SahAssar1y

                  What is the networking difference between a service for yourself that you want to access from "various places" and a public service with auth checks for your key?

          • jszymborski1y

            Maybe the service is provided over SSH via e.g. port-forwarding (or is simply "SSH access to a server").

            • SahAssar1y

              Sure, but are L7 attacks easier than L4 against those servers? Adding more layers/software has a cost in configuration, maintenance, attack-surface, etc.

      • zbentley1y

        You're not wrong, but I'd say fail2ban still has value for junior operators seeking to reduce load and increase stability. If you don't know how to harden SSH, fail2ban is offers a much friendlier way to reduce the volume of logspam, CPU burn, and network traffic. It's just a pity that it's understood/documented/pitched as something that substantially increases security.

        • BrandoElFollito1y

          > If you don't know how to harden SSH

          then you do not open it to Internet. Otherwise you patch aggressively, you use ssh keys and not passwords and you move it to some random port to hide it a bit (it actually helps)

          > logspam

          you can filter this out in your log management tool

          > CPU burn

          if this is your concern, then you have a hep of issues you need to address. I have never seen a CPU perf hit because of such behaviour (there are cases where it happens, butthis is due to a vulnerability of the service)

          > network traffic

          the packet is here already, there is nothing to reduce

          • Karunamon1y

            Moving ssh off of port 22 makes it a pain in the ass to work with. Ports are standardized for a reason.

            Authentication attempts are a useful security signal; I don't want to filter them out. I want hosts running dictionary attacks to not be able to connect to my services in the first place. If you are running an SSH bot, then I don't want you on my website or anything else.

            • BrandoElFollito1y

              > Moving ssh off of port 22 makes it a pain in the ass to work with. Ports are standardized for a reason.

              yes, they were standardized in the ol' good times :) If you have a limited amount of people/services connecting then it is manageable. But of course YMMV.

              > Authentication attempts are a useful security signal; I don't want to filter them out. I want hosts running dictionary attacks to not be able to connect to my services in the first place. If you are running an SSH bot, then I don't want you on my website or anything else.

              enumeration and brute force on SSH fail by design when using keys.

              As for other services I do not see how this helps - you will block random IPs hoping that a vulnerable site is not taken over if they happen to get back. It is not common (at least in my monitoring of several honeypots in various locations) to have the same IP being particularly visible. Sure they are back sometimes but this is quite exceptional. Anyway - it is not worth the hassle, better have proper hardening.

              • throwitaway11231y

                > yes, they were standardized in the ol' good times :) If you have a limited amount of people/services connecting then it is manageable. But of course YMMV.

                Agreed. I've never found it difficult to manage this. I already tend to configure SSH hosts in my ~/.ssh/config file anyway so that I don't have to remember every IP and port combination for every host I have access to when I want to use SSH (or something that relies on the SSH protocol like rsync or scp).

      • ars1y

        fail2ban increases your server performance. It cuts down on enormous amounts of logging from failed attempts, and reduces the CPU used to deal with the failures.

        Some sites get a mind boggling amount of attempts. For example I sysadmin some Jewish sites, and they get exponentially more hacking attempts than the sites not mainly used by Jews. (This was before the current war mind you, I'm sure it's worse now.)

      • mmsc1y

        People don't believe it's possible for software to be secure, and need a secondary defense to "protect them".

        • catalypso1y

          > People don't believe it's possible for software to be secure

          Rightfully so. You'd statistically be almost always right considering a software unsecure given enough time (for the vulnerabilities to be introduced then found).

          > need a secondary defense to "protect them"

          Nothing wrong with that. It's called Defense in Depth and is rather advised. Once you understand that security measures are not bulletproof, stacking them proves to be an easy way to increase protection.

          The case of fail2ban is not trivial: reducing log noise is a great perk, and can indirectly help with monitoring (you'd more easily notice suspicious behaviour if it's the only thing on your logs), but it comes at the small cost of setting it up, and accepting the risk of having a shared IP unwillingly blocked.

        • marcosdumay1y

          Except that it explicitly doesn't protect against security bugs.

      • eikenberry1y

        I always read the main use case had nothing to do with security, but was to reduce log spam.

    • mekster1y

      Repetitive log is something you appreciate by reducing and you don't have to give it unnecessary CPU cycles too.

    • jcynix1y

      Fully agree. Limiting the networks which can access your server will help, e.g. limit access to just your local provider or your workplace and you'll see no attempts from Brazil, China, ... unless you are located there, of course ;-)

      • ajsnigrutin1y

        It's all fun and games, until you travel outside of your country, and try to access stuff at home.

        • jcynix1y

          That's manageable with a bit of preparation: when I'm travelling, I allow access from other networks, e.g. those from phone providers. Or add a web form where I activate the IP address with a cryptographically signed "token" which the server can verify and then add the IP address to the set of allowed ones.

          Used one or the other every now and then in the last 10+ years and still have my attackable footprint small the rest of the time.

        • euler21001y

          Tailscale (et al)?

    • Too1y

      How do you protect your vpn?

      • d-z-m1y

        use a vpn that does not advertise its presence, like wireguard.

  • danielovichdk1y

    I am not sure why this should keep anyone from hosting their own servers and services.

    I find it positive to know that whatever and whomever expose anything on the Internet someone will try to exploit it.

    For 443 and 80, why the concern ? Outsiders can try all they want bit if you are certain the software you use is secure, there will be no cigar.

    I'd much rather have these things out in the open than hiding things away with some obscure thought about that should help anything.

    If something is difficult do more of it. The same goes for understanding security.

    • dotancohen1y

        > if you are certain the software you use is secure
      
      This is the problem right here. You can be certain that the software you use has security issues.
      • lofaszvanitt1y

        And who will fire a 10k+ exploit on your server? So you could record it and resell? In the early days, surfing shady sites with Internet Explorer, you could net a lot of interesting js that exploited the browser.

        • dotancohen1y

          My server is an attack vector for my 10k+ users, and all their contacts. A 1% ransomware infection rate could net them $1 million USD worst case, and potentially an order of magnitude more if one of my users is browsing from a work machine in their network.

          Don't underestimate the security value of people hitting your servers, even if all you think you're serving is emojis.

          • lofaszvanitt1y

            I'm not underestimating. All I'm saying if someone pays 10k or more for an exploit against ssh/nginx/whatever, nobody is gonna pepper your server with it. They will sell it to a broker and pocket the money, end of story.

            You will be targeted if your server seems to be the lowest hanging fruit or most easily exploitable or the target is most easily reachable through your site. Otherwise noone will bother with your setup.

            • tiberious7261y

              This is very much so sticking your head in the sand. Some attacks are sold to highest bidder, others are deployed wide and fast. Some of us are responsible for securing high-sensitivity systems where such a shoot from the hip and trust everything will be okay attitude isn't acceptable.

            • elintknower1y

              Yeah, this is also a huge concern of mine. There's also nearly no standardization / information as to how to harden just a bit more than is commonly suggested by web devs / bad tutorial sites.

              • dotancohen1y

                Seriously. When you find something, please let me know too!

              • lofaszvanitt1y

                Reading the manual.

      • input_sh1y

        The question isn't does the software I run have some sort of yet-undetected security issues, but am I a valuable enough of a target for someone to waste their yet-undetected exploits specifically targeting me?

        If the answer's no, then your only job is to keep up with software updates.

        • lazide1y

          If you’re exposing your software to the external internet, you’re potentially valuable enough to get a drive by.

          • input_sh1y

            Assuming your software is fairly up to date and/or you haven't badly misconfigured it, they're not gonna do anything. There are a ton of routers and IoT devices that are a much easier catch than a machine run by someone that actually gave a thought or two about securing their server.

      • danielovichdk1y

        Sure. And so what ? Should I stop using it ?

    • tjoff1y

      > if you are certain the software you use is secure

      The entirety of the problem is that you can't be certain the software you use is secure.

      • danielovichdk1y

        Exactly. And to overcome this you as a user of that software has to be aware of that specific software.

        Most people doesn't give a shit, they pull down or introduce dependencies and think "wauw that was easy and fast".

        Of course there is secure software, otherwise we wouldn't be able to live as we do.

        • lazide1y

          As history has shown repeatedly, there is no secure software - just software that folks have not yet discovered how to exploit widely and effectively yet.

          • oopsallmagic1y

            Then why bother? I'm sorry, but where did this meek, defeatist attitude come from? It pervades software now. Sure, you're right, I guess I could get hit by a bus today, but that won't stop me from crossing the street, because there are a lot of things I can do to minimize my risk, like looking both ways, listening, and crossing at a signal. Software is similar. "Nothing means anything, all is chaos" might poll well on Reddit, but it's not good engineering.

            • lazide1y

              Who says it’s defeatist? It’s realism. You might as well say noting mild steel only has a 60-80kpsi yield strength ‘defeatist’.

              That attitude allows practical risk management and effective engineering. Pretending software can be secure or mild steel has infinite yield strength cannot.

              There is no lock that can’t be picked either, which is why no one leaves millions in cash protected just by a lock without guards and a surveillance system. And why they insure large amounts of cash.

              At this point it should be pretty obvious - don’t put important secrets on computers without a way to expire/revoke them. If it’s a secret that can’t be expired/revoked, think long and hard about if you need it on a computer - and if you do, use a SCIF.

              Monitor any connected computer systems for compromise. Use encryption extensively, preferably with hardware protection, because software is insecure, etc.

              Same with controlling dangerous equipment - don’t rely on pure software or someone will get killed. Use hardware interlocks. Use multiple systems with cross checking. Don’t connect it to the internet. Etc.

              This is all industry best practice for decades now.

              • wruza1y

                But the initial dialog was more like

                  Q: this is good steel still, why not use it?
                  A: steel is never ideal, that's the problem.
                
                Oh really.

                Risk manage us nginx please. At least write out the steps, you must have a checklist or something, right?

                Let's be honest, we just apt install it and read vulnerability reports when they hit /news.

                • oopsallmagic1y

                  Exactly. I don't believe that the argument that some software somewhere at some point could have some vague security flaw in it is usually good enough to justify not running the kinds of software most of us here work on. It's solipsistic, and honestly seems a little in bad faith.

                  But it's also moot: if you're that afraid of vague security threats, then just don't expose your software to the internet. It's not difficult.

              • oopsallmagic1y

                > Who said it's defeatist?

                Uh, me, I did. I thought I was pretty clear. Please refer to my previous comment.

                > It's realism.

                Okay. How are you going to change your behavior?

                I'm not sure what point you're trying to make. If you want to put your recipe website behind a SCIF, be my guest. Some of us aren't quite so afraid.

                • lazide1y

                  Haha, pot calling kettle black. I don’t need to do a damn thing different. Cars are still dangerous 100 years after they were invented, and the world still turns.

                  You’re the one trying to turn this into some kind of existential emergency. What are you going to do differently?

            • kloop1y

              > Then why bother?

              Because software is fun, and I get to work with cool things. There is a joy in programming in and of itself.

              I guess your question doesn't make sense to me. Just because it will eventually be broken, does that automatically mean there's no value in software? I don't think that's true, it just probably means you should have an analog backup process if possible, especially for critical things like government services.

            • tiberious7261y

              It's not defeatist, it's called defense in depth

          • hollerith1y

            That gives the misleading impression that it is impossible to create and maintain a truly secure software system.

            • lazide1y

              I have yet to find any such system - given enough time and exposure.

              What makes you think such a thing is possible? In reality, not theoretically.

              I also have yet to find an unpickable lock, given the same constraint. Locks still have utility.

              But only fools protect something very valuable with just a lock.

              • hollerith1y

                >What makes you think such a thing is possible?

                The main source of my confidence is extrapolation from the results of successful initiatives to improve security. Rust is one such initiative: at relatively low cost, it drastically improves the security of "systems software" (defined for our purposes as software in which the programmer needs more control over resources such as compute time and latency than is possible using automatic memory management). Another data point is how much Google managed to improve the security of desktop Linux with ChromeOS.

                There's also the fact that even though Russia has enough money to employ many crackers, Starlink's web site continued operating as usual after Musk angered Russia by giving Starlink terminals to Ukraine -- and how little damage Russia has managed to do to Ukraine's computing infrastructure. (It is not credible to think that Russia has the ability to inflict devastating damage via cracking, but is reserving the capability for a more serious crisis: Russia considers the Ukrainian war to be extremely serious.)

                Sufficiently well-funded organizations with sufficiently competent security experts can create and maintain a software-based system that is central to the organization's process for delivering on the organization's mission such that not even well-funded expert adversaries can use vulnerabilities in that system to prevent the organization from delivering on its mission.

            • kjkjadksj1y

              Is that impression not accurate? Everything is possible to exploit imo. Its why the us government spends a mountain on cyber defense and offense.

      • moffkalast1y

        Haveibeenpwned paints a pretty good picture. Breaches, breaches everywhere. The average piece of software cannot be trusted with keeping any data secure for any notable amount of time.

        It's funny that password managers and random generated single use passwords are so popular now, because the greatest risk to one's credentials isn't direct attacks, but having them leaked by someone's half assed backend. It gets even funnier when the service that gets breached has some arcane password security rules with two symbols or whatever, the ultimate hypocrisy.

        • otherme1231y

          Almost all stories you read about data leaks are some variation of "I installed XXX database and forgot to limit access" or even "and I wrongly supposed it wasn't listening to an internet exposed port". Breaches are just queries.

        • oopsallmagic1y

          To be blunt, those breaches are the result of software written by people I wouldn't trust to bag my groceries. I've never had a database get leaked, because I'm not a hack, and I know how to do the bare minimum above professional negligence to secure internet-facing services. I wish I could say the same about most of the industry.

        • withinboredom1y

          A “breach” usually means they got access to the database, which is much different to access to the underlying server. We aren’t talking about databases, we are talking about servers.

          • moffkalast1y

            It really depends on the architecture. At least I think it's fairly common for people to have some sort of database proxy running beside the static serve, so there isn't any direct public access and to do some caching, but once you're there it should be pretty wide open.

            • withinboredom1y

              In my experience, it is much more likely someone forgets to escape some input and opens the database up (via SQL injection) than it is for someone to break in via ssh or gain access to the shell.

      • quaintdev1y

        Common the web servers like Nginx, Caddy are not secure? If they found a zero day in these application whole Internet will go up in flames.

        • robertlagrant1y

          The whole internet keeps patching those flaws as they are found. The problem with self-hosting is patching.

          • wruza1y

            This is a non-problem since the invention of unattended updates. This whole subthread spreads uncertainty and doubt over simple things like nginx or ssh. Service providers don’t patch their software by hand either.

            20 years ago, when I was still young and naive, I took these concerns way too serious, remapped ports, believed in pwn, set up fail2ban and knocking, rotated logs. Later I realized it was all just FUD, even back then. You run on 22, 80 and 443 like a chad, use pw-based auth if you’re lazy, ignore login attempts and logs in general and never visit a server until it needs reconfiguration. Just say f* it. And nothing happens. They just work for years, the only difference is you not having tremors about it.

            The only time a couple of my vpses were pwned in decades was a week after I gave a sudoer ssh key to some “specialist” that my company decided to offload some maintenance to.

            What changed from back then is that software became easier to set up and config and less likely to do something stupid. Even your dog can run a vps with a bunch of services now.

            • denton-scratch1y

              > And nothing happens.

              Good luck. Some people have different experiences.

              • wruza1y

                Some people install every php plugin they can find. Recently I gave my coworker an access to a gui server and next day he complained he can't install some chinese malbloatadware on it. People have different experiences due to different paradigms. My message is about not being anxious, not about being clueless.

                With opensource and how code works in general, we are all in the same boat with bigcorps and megacorps. And they receive the same updates at the same rate (maybe minutes faster cause they host repos).

                This quote, "you can't be certain the software you use is secure", is technically true but is similar to the "you can't be certain you won't die buying groceries". Perfectly useless fearoid for your daily life.

                • tjoff1y

                  I get what you are saying, and if anything all the "attacks" in the logs should build you some confidence. Oh, so 98% of all attacks assume I haven't changed the root password? I must be ahead in the game then.

                  But the way you phrase it isn't really convincing, and for singling out 443 and 80 ports. As the subthread of breaches hint towards. You might not need to be worried about nginx, but whatever you host on nginx might be a problem and being "certain the software you use is secure" is also pretty darn useless as guidance.

            • ricardo811y

              >pw-based auth

              better off using key only logins and forgetting IMO

        • mr_mitm1y

          Even OpenSSH almost got a fatal backdoor recently.

        • tiberious7261y

          What planet are you on? Nginx had a 0 day as recently as April 2022 https://www.accuknox.com/blog/nginxday-2022-nginx-ldap-zero-...

          This happens _all_ _the_ _time_

          • account421y

            A very specific one that doesn't affect 99.99% of nginx servers.

            • tiberious7261y

              "If they found a zero day in these application whole Internet will go up in flames."

              Don't move the goalposts. I'm certainly not saying that nginx is insecure. I'm saying that if you think any piece of software written after the 80s has reached the point where it won't have 0 days anymore you just haven't been paying attention

    • e12e1y

      This seem hopelessly naive just after the windows php bug bit?

      https://arstechnica.com/security/2024/06/thousands-of-server...

  • kristopolous1y

    in the early 2000s I kept an anonymous ftp server open and would routinely get the latest cracked software delivered right to my hard drive. It was very convenient.

    • sattoshi1y

      Cracked software can contain extra features. Especially when delivered in this way.

      • seanthemon1y

        Ooo like that awesome techno music on startup, or maybe bee movie during install

        • Etheryte1y

          I like the idea that someone embedded an entire movie as a malicious payload in an installer.

          • account421y

            Well not a hollywood movie, but cracktros are a thing.

          • seanthemon1y

            I'm sold, send me the link

      • 1y
        [deleted]
      • input_sh1y

        In the early 2000s it was pretty much expected that each and every computer you encounter is full of viruses. That is, viruses on top of viruses that come by default from everyone running a cracked version of Windows XP.

        • welder1y

          Most people on here didn't use Windows in the early 2000s, or ever.

          • PHGamer1y

            I remember a very popular windows xp key that always circulated on the net. it was funny as hell doing some contract work and a client giving me this code his former IT used to install windows.

            Im like is this what I think it is.. anyways he ended up buying a legit license since you know lawyer but still funny that I saw it in a business environment.

          • account421y

            A very optimistic estimate of this websites demographic.

    • lofaszvanitt1y

      Oh, when you needed specific ftp clients, because most of them couldn't handle special characters needed to access the directory containing the LOOT :D.

    • qingcharles1y

      Back in the early 90s my first "job" on the Internet was for a pirate group -- I had to scan and find open FTP directories where the group could upload and store their output. There was a specific job title for this position, but I no longer remember it.

    • throw_m2393391y

      "H2O, try before you buy..."

  • mianos1y

    Over 90% of the ssh logins come from just a few China Telecom addresses. They just keep trying random ssh accounts over and over all day. I just geoblock China now. Maybe occasionally unblock it for a few minutes if the kids want to buy something from Shien. Then I honeypot the rest with the continuous ssh banner script.

  • mtekman1y

    I have a utility that parses ssh failed attempts and creates iptables blocklists:

    https://gitlab.com/mtekman/iptables-autobanner

    For those just wanting the blocklist, here is a table of malicious IP addresses, with columns of: address, number of ports tried, number of usernames tried.

    https://upaste.de/bgC

    • securethrowaway1y

      I simply run fail2ban with a whole bunch of customer filters that will ban people very quickly. There's no need to request php or malformed urls when php is not used for example.

      • mtekman1y

        I used to run fail2ban, but I found it (or at least its defaults) ineffective against discouraging further requests. With iptables, you can specify the connection to hang for a period and then drop

        • justsomehnguy1y

          Defaults are set to reject. Just configure the jails or a global config.

    • miah_1y

      A iptables hashlimit rule can do the same. Your firewall rules get to be more readable and you don't end up relying on the security of a log parser.

      The biggest win comes from just disabling password authentication in sshd though.

    • Phelinofist1y

      I run endlessh, I always giggle when I see some connection that last for 2d

    • sambazi1y

      a lot of ppl thought this would be a good idea at some point

    • eps1y

      upaste link is 404

  • frankohn1y

    Some time ago I set up a server for a website and I was appalled, like many others, by the number of SSH connection attempts. I decided to open SSH only in a randomly chosen port number above 1024 and now I have essentially zero probing attempt. It is trivial but for me is a satisfying configuration.

    • usr11061y

      This was true in 2018. In recent years I get 100s, sometimes 1000s of login attempts a day on high addresses.

      My servers are on AWS addresses. If someone searches for servers (as opposed to routers, phones etc.) AWS might be a preferred address range. No experience whether scan rates depend on the address used.

      • eps1y

        It appears to be two-stage process.

        There are open port scanners that just check what ports are open on which IPs, and there are separate ssh login brute-forcers. Once your machine gets picked up by the former, the latter will pile up.

        I have two servers on adjacent IPs, both with ssh listening on a high port. One gets hammered with login attempts and the other does not.

        • nonamesleft1y

          A lot of these seem to use zmap (https://github.com/zmap/zmap) or masscan (https://github.com/robertdavidgraham/masscan) for the initial scan.

          Often with default parameters such as zmap setting ip id to 54321, having tcp initial window at 65535, having no SACK bit set and masscan with no SACK bit either, tcp initial window at 1024, tcp maximum segment size 1460 (which is strange to put below initial window size!), (older versions having fixed src port 61000 or 60000 from documentation examples and no MSS set), all of which are extremly uncommon in legitimate traffic and thus easily identified.

          Even those so called "legitimate" scanners (emphasis on the "") seem to use these tools with little or no extra configuration.

          With this setup the last time my high-port ssh (key-only) has got an attempt on it was 2023-07-26 (previous intruders get permanently firewalled).

        • gradschool1y

          This might not matter for your setup, but I would have thought it's bad in general to have sshd listening on a high port because then any non-root user who finds a way to crash it can replace it with his own malicious ssh server on the same port.

          • usr11061y

            You mean non-root local user? We don't have non-trusted users on the system.

            Well, unless the http server or our dns resolver has a remote code execution vulnerability.

            So directly I don't see the risk you describe. Of course considering maximum defense in depth you might have point.

          • 20after41y

            That's a good point, though you could use some firewall rules to rewrite the port number so that the local daemon is listening on the normal port but accessible via an alternate high numbered port.

        • usr11061y

          Maybe that's the case. The machines where I am seeing a lot of ssh login attempts on high ports have been on the same IPv4 address for years. Some since 2018.

        • frankohn1y

          Interesting to know. For the moment, several months, I still have no login attempts but so that means my server didn't get picked up by any port scanner.

      • gsich1y

        addresses == ports in your view?

        • usr11061y

          Yeah, sorry about the mistake. Too late to edit the comment :(

  • josephcsible1y

      > 1016 cd ~; chattr -ia .ssh; lockr -ia .ssh
    
    Does anyone know what the "lockr" command is? I can't find any references to it besides people saying they observed malware trying to run it, usually (as is the case here) right after a chattr command with the same arguments.
  • eps1y

    > 8181 root

    In 30 days? That's tad unrealistic.

    Just checked and there are dozens root login attempts per minute on my colo'ed server in the EU. Virtually all from the Chinese and post-Soviet IP space. But mostly Chinese.

    • nubinetwork1y

      I see ~1000 unique IP addresses hitting SSH every day.

  • gunapologist991y

    > In conclusion, these commands represent a clear strategy to infiltrate, assess, and establish control over targeted systems.

    Oh hello, ChatGPT. You seem to be everywhere these days.

  • hugocbp1y

    Amazing article!

    It is actually amazing how fast and thorough the connection attempts happen as soon as you put anything online.

    I've been playing around Hetzner and Coolify recently, and notice that, as soon as port 22 is opened, it is bombarded by those attempts. Several per second. It might be due to Hetzner IPs being reused, but happened to me every single time. Same with Postgres default port (those were the ones I've seen).

    I have defaulted to use Terraform and bash to only open those ports in the Hetzner firewall (and more common ones like 3000 or 8000) to my own current ip. It does mean I'll get drift and need to reapply the Terraform code if I change ips, but seems to be at least one way to defend.

    I fear that a lot of devs jumping into the "you only need a VPS" crowd on Twitter will end up with a huge attack surface on their apps and machines and most won't even know they are being targeted like that most of the time.

    To this day I still find it hard to find a comprehensive security guide for those newer Linux fresh boxes (and the ones you find are all so very different with different suggestions). If anyone knows of a good one, please share with me!

    • fsmv1y

      You just need to turn off password authentication so it's keys only. They can attempt logins all they want and never get in.

      Also if you run ssh on a nonstandard port you get many fewer attempts. There are several groups that constantly scan all of ipv4 for open ports, if you use ipv6 they cannot scan that space anymore.

      Optionally you can set up fail2ban but I find it's not a big deal.

      • ogud20251y

        I changed my SSH configuration to only listen on an IPv6 address 6 months ago and since then the number of SSH attacks has fallen from 1000+/day to less than 10/week.

      • hugocbp1y

        Thanks!

        That is usually what I already do. Good to know I'm on the right path.

        When possible I disable root login as well (though Coolify seems to need it on, even if without password).

    • e12e1y

      I would recommend just using a VPN, like tailscale, for all non-public resources - rather than IP whitelisting.

      Ed: including private web services like self-hosted gitlab not used for publishing public projects.

      • hugocbp1y

        It's on my list to try. Haven't sat down to actually try using Tailscale with servers yet but seems like a good option. Thanks!

  • bobbob19211y

    Not sure if op will see this, but with regard to his comments on MikroTik routers and frequently seeing in his honeypot logs, the command: /ip cloud print

    he is correct, This is a MikroTik command- although mikrotik has this feature, disabled/ off by default, a lot of users make use of it, and running that command will (if cloud dns enabled), will show the dynamic DNS entry of the device he is connected to. Ie if the cloud DNS is enabled, the output from that command will be something like: Detected public ip: 34.2.82.3 DynDns: djwisyehd.clouddns.mikrotik.com (which will always be updated to the detected public IP address of the router)

    So I assume the attackers run this command so that they can still reach the router in case it’s public IP address changes at some point. (And assuming that the device will still be accessible after any public IP address changes).

    (or perhaps they run that command to see if the cloud DNS service is disabled, which is the default, in which case they will then enable it so that they will have a dynamic DNS entry for the device).

  • laktak1y

    What does `echo -e "\x6F\x6B"` do?

    • ggambetta1y

      If you say it 3 times in front of a mirror, it summons Stallman

    • raverbashing1y

      Maybe I should create a honeypot where cat, echo, sed, and curl/wget all drop random bytes in all commands they execute

      Would be fun

      • thesnide1y

        Better would be to just subtly change the output...

        Like do a +1 on the byte every 7 bytes. Bonus to do it only on every 7 printable chars.

        And you can even do A/B testing on the constant 7.

    • gpvos1y

      Tests whether `echo` supports the `-e` option.

    • zh31y

      It prints "ok" and shows they got in (it relies just on a shell, nothing else).

      • lucianbr1y

        Why not do 'echo "ok"'?

        • kynetic1y

          As shown by someone having to ask what it does, it obscures what it does.

          • lucianbr1y

            Doesn't seem terribly useful. I mean it only obscures that it prints "ok". If you're looking at the logs, you probably already figured out someone is attacking you, and if you didn't, seeing "echo ok" will not help you figure it out.

            If the only thing the command does is "obscure what it does", then the only thing it obscures is "obscure what it does". I guess there's no requirement that whoever writes these scripts is a genuis.

            • Retr0id1y

              People writing malware generally don't want to deploy it on honeypots, because then they're handing their payload (and other tradecraft) directly to analysts.

              So often the first stage is an attempt at honeypot detection, or more broadly, device fingerprinting.

              A bad honeypot might not even run a real /bin/sh, and this detects that right off the bat.

              • lucianbr1y

                That makes a lot more sense than "it obscures the obscure thing it does to obscure itself".

    • ynoxinul1y

      This look like a simple test to see if remote command execution works.

    • spc4761y

      It echos "ok".

    • 1y
      [deleted]
    • Mxrtxn1y

      Prints out `ok`

  • pingec1y

    A bit tangential but is there a service or self hosted solution that would take a list of IPs and then keep scanning them periodically and alert me if any new ports have suddenly open?

    • bluish291y

      I think shodan could br useful in this regards

      https://www.shodan.io/

    • cranberryturkey1y

      hmmm....you could do that with nmap script and a cronjob.

      • cranberryturkey1y

        I just scanned my domain for all 65k ports and it took 20 seconds with a 10gbit pipe. i could scan yours for you and shoot you an email if a new port is discovered. Would charge you Like $100/year or something.

  • FredPret1y

    I simply block traffic from countries where I do not do business in.

    I used to see constant attempts to mess with Wordpress URLs, which I know is not legitimate because I don't run Wordpress.

    Cutting out Russia & China basically removed this problem. I really hate locking up my tiny corner of the internet but I don't see another way.

  • micheloosterhof1y

    If you want to try this yourself, although he carefully doesn’t mention his software, it appears he is running the Cowrie honeypot (https://github.com/cowrie/cowrie)

    (Note, I’m the maintainer)

  • willfiveash1y

    This story reminds me of the time I accidentally/naively set up a ssh honeypot when I configured the router I was using at the time (this was a long time ago) to forward incoming SSH connections to a ReadyNAS (which was using a Sun SPARC processor) in my house. I did that so I could log into it while I was away from my house. One day, I ssh'ed in and notice that the ReadyNAS was running very slow which surprised me because I thought it was idle. I checked the CPU usage and the sshd was using 100%. First, I thought it was a bug but it occurred to me I should check my incoming ssh connection attempts in my router log. Turns out there were a ton of ssh connection attempts coming from an IP address assigned to China. In response I changed the router port forwarding for incoming ssh connections to use a non-standard port number, like 55,243, and after that my ReadyNAS was no longer bombarded with ssh connection attempts. Lesson: try to avoid forwarding standard port numbers.

  • agilob1y

    There's a project for running Honeypot as a Service: https://haas.nic.cz The data is public and you can register your router too

  • ciebie1y

    What is a `lockr` command? Is it file system specific or something? Never seen anything like this. It probably should lock permissions on .ssh, but how?

  • nisa1y

    Somewhat related due to a weak password a mail server from a community I'm involved in send out lot's of spam mail, after analysing the log files I've had over 1500 different IP addresses that logged in to send spam, about 10 mails for each address. ASN and subnets where spread across over the whole world. It seems like these attacks are coordinated using vast botnets and the use of single ssh public key here seems to confirm this. I had similar experiences going after attacks on WordPress instances and there I've also found attacks spread out across lots of hosts.

    I'm wondering if it's possible to pin down those behind these attacks, there must be mistakes.

  • Tiberium1y

    Interesting article, sadly due to my exposure to LLMs I couldn't help but notice that the parts about "oinasf" and sakura.sh are AI-edited at least. Kind of a weird choice considering that a lot of the article was clearly human-written.

  • ricktdotorg1y

    3 simple sshd config lines remove a gigantic amount of worries if you run open ssh servers:

       Port NN
       PasswordAuthentication no
       AllowUsers user1 user2 user3
    
    change those, sleep at night.
  • nilsherzig1y

    Check out https://viz.greynoise.io/ especially the trends > anomalies tab is very interesting

    • jslakro1y

      How do you use that information?

  • ibbtown1y

    Had a own server in university during mY PhD. Most request were trying to download scientific papers from large journals using absolute and not relative URLs after request.

  • poikroequ1y

    I once tried hosting a web server at home by exposing ports 80 and 443 to the Internet. Hours later I reviewed the logs, thousands of attempts to hack into my lil Linux server. It spooked me to say the least, so I switched to using cloudflare tunnels instead.

    Exposing ports on the Internet is dangerous, especially SSH. You're much safer using a proxy or gateway of some sort, or better yet a VPN if it doesn't need to be publicly accessible.

    • waingake1y

      Is it? If you've got `PasswordAuthentication` disabled, only allow public key logins and keep your system up to date. Honest question.

      I self host my email ( docker-mailserver ) and host my personal website on an old laptop with a static IP. Have done for years now without issue.

      • pkrotich1y

        The keyword is diligently keeping your system up to date! That said you’ll still have exposure to zero day vulnerabilities and DOS attacks.

        • Fabricio201y

          But an attacker with one of the biggest vulnerabilities on earth (hell, ssh noauth 0day) would very likely use it against big cloud providers and infrastructure (isps and others) and not burn it on your home server! Keeping it reasonably up to date with your distro's cycle is probably enough for most people doing this home server thing.

          So of course, as things always are with security this is a matter of risk assessment and understanding your attack surface, a server with only public key and maybe on a special port goes a very long way, add fail2ban on top and i'd say it's probably fine for quite a while.

          But that does make me think... what if... a wormable noauth 0day like that on ssh or some other popular system... how fast could it replicate itself to form the biggest botnet.. how long would it take, to take over all visible linux servers on the internet (so that your little home box ends up being a target)?

          I guess at that point you are limited by bandwidth, but since you can scale that with every compromised server... hope someone does the math on that one day!

          • rcxdude1y

            Ipv4 is only 4 billion addresses. It doesn't actually take very long to just try all of them. If you're running a service exposed to the internet and it has a published exploitable vulnerability, it's just a matter of time before it gets exploited. (that said, that time does give a little buffer for patching)

        • kristopolous1y

          https://wiki.debian.org/UnattendedUpgrades Most distros have something like this.

      • Beijinger1y

        "PasswordAuthentication disabled" not sure I can even do this on my shared BSD server. I have ssh access via pw and need it. Is this really dangerous?

        • Scramblejams1y

          Yes, it's risky to accept password auth if someone sharing the box with you has a poor password. They could do things like:

          . Install a spam or brute force password bot, which could get the machine kicked off its internet connection (in addition to whatever havoc it causes first)

          . DoS the server by filling up the disk or using too much RAM (are quotas enforced?)

          . Exploit a local vuln to get root, if such exists on that box. (Is the kernel promptly patched and the box rebooted?)

          . Explore other users' directories (are permissions locked down correctly across users?)

          …and more thrilling possibilities!

          Embrace key auth. Future you will thank you.

        • johnklos1y

          It is, if for no other reason than you never know when some other user has a guessable password. You should switch everyone to ssh keys. It's a good excuse to learn :)

        • sneak1y

          Yes. Authenticating with passwords is obsolete and dangerous. Use keys and disable password auth.

          • tpoacher1y

            And if you really like passwords, you could always enable both, too!

        • fragmede1y

          How good is your password? If it's long, with special characters, it's fine. Install fail2ban. The problem with auth keys is you can't get into the server if you don't have your laptop/phone/NFC device because you got pickpocketed/mugged?

      • Beijinger1y

        "I self host my email "

        Is this still possible? Are your emails getting delivered?

        Downvoted. I don't know when the downvoter tried the last time to "host their own email". Yes, DMARC, DKIM und SPF. Good luck trying to get your email deliverd to t-online or something.

        https://forum.hestiacp.com/t/t-online-curious-story-about-th...

        They may even check if your domain has an "imprint". I kid you not. I use my own domains too, but I piggyback with infomaniak.com

        • pja1y

          > Is this still possible? Are your emails getting delivered?

          Mine are. Although it probably helps to have a static IP with a 25 year long clean history.

          Are there very occasional glitches? Sure. But I've seen ISPs drop everything from GMail on the floor for no obvious reason. I've seen GMail drop GMail email before. Same for every other large email provider.

          To date I haven't seen any reason strong enough to push me to switch to a centralised email host. That day may yet come of course.

        • A1kmm1y

          I self-host my email, and have not really had problems delivering normal quantities of personal email (except a bit of pain for Microsoft to accept mail in the first place, but it can be sorted quickly) - as long as you do DMARC / DKIM / SPF.

          I've never heard of t-online before or tried to send an email there to my knowledge... if one provider I've never heard of would refuse to accept my mail if I ever sent something to them, that's more of a them problem than a me problem - but it certainly isn't the norm for other providers.

        • hggh1y

          > Is this still possible? Are your emails getting delivered?

          Yes and yes (if DMARC/DKIM/SPF configured correctly).

        • johnklos1y

          > Good luck trying to get your email deliverd to t-online or something.

          People who say it cannot (or should not) be done should not interrupt those who are doing it.

          The dismissiveness is likely why you are downvoted, I'm guessing. The suggestion that because it's hard for you and therefore you're surprised others are doing it isn't a good look.

          Self hosting email isn't that hard, and there are many solutions for all sorts of self hosting issues. That's a topic for another discussion, though.

          • Beijinger1y

            "Self hosting email isn't that hard". Self hosting is super easy. Getting your emails delivered is hard. And I am not even talking SPAM folder here (see t-online example).

            Smart comment from reddit:

            "The problem with selfhosting email, unlike selfhosting services like Jellyfin or Nextcloud, is that you rely on other people's servers to play ball with you, but they often don't. Or they play for a while and then suddenly decide not to without telling you. It's unpredictable and we selfhosters don't have enough control over that."

            This describes it pretty well.

        • cherryteastain1y

          I fo it too and can deliver to gmail/office365 etc addresses no problem.

        • gsich1y

          yes and yes.

          Selfhost does not imply residential IP.

    • nurettin1y

      Don't worry, they are usually Russian/Chinese ips scanning for 5 year old php exploits. I've been exposing ports to the internet for decades with no issues. Always block ssh password and keep software relatively up to date. If you are very paranoid, make a vps beacon and remotely tunnel ports from your lab to it. That way you only expose the beacon.

      • zelphirkalt1y

        I wonder, what is the issue with authenticating by password. If you choose a password of lets say 64 random chars, shouldn't it be pretty safe? Or is there something in the password method itself, that is inherently weak?

        • denton-scratch1y

          > Or is there something in the password method itself, that is inherently weak?

          Your 64-character high-entropy password might be safe; other users on your system might baulk at memorising/typing in 64 random chars, and choose a less-secure password instead. With SSH keys, that can't happen.

        • cess111y

          Sure, they probably won't crack that, but there are other things to consider as well. A sshd on IPv4 port 22 that accepts password auth attracts attention, and you'll spend CPU cycles constantly checking credentials from very large database dumps that float around. In my experience it leads to more log noise too, it seems many bots will discard your IP and stop pestering it if passwords aren't accepted.

          So in practice you'll probably also use something like fail2ban, firewall rules that only allow connections from certain IP blocks, things like that.

        • a_dabbler1y

          The first benefit is some bots won't bother testing passwords as the SSH error message tells them the server doesn't use password auth. The second benefit is if your server is compromised it's quite easy for a rootkit to hijack SSH and steal your password when you login (and then abuse that on other servers you use it), the same is not true with a key and it is much harder for a rootkit to abuse as long as you only use the key on your local machine (there are strong protections against SSH handshake MITM attacks afaik)

        • Hendrikto1y

          > Or is there something in the password method itself, that is inherently weak?

          You have to send your password/hash. With PKC, your private key never leaves your device. It can even live on a separate security key. All you ever send are signed messages, never your key.

        • KAMSPioneer1y

          There are still advantages to public key auth. Sibling comment mentioned resource use, but also consider ease of use: are you setting a random 64-character password on every machine that has SSH server installed? Would it not be easier to generate one ed25519 keypair, apply a reasonable passphrase (and/or use disk encryption), and then you have secure auth on all your machines without a password manager?

          If you're _not_ setting unique 64-character passwords per server, then you should consider what happens if your super strong password is discovered -- an attacker would have access to all your boxes. Compromising a key is harder than compromising a password.

    • kristopolous1y

      I've been doing it for 25 years. It's fine.

      • Hendrikto1y

        ”Works for me.“ does not really answer the question.

        Having a 25 year history might be why your mail gets delivered, while many people trying to self-host have constant and unpredictable deliverability issues.

        • kristopolous1y

          It's more an advocacy against security paranoia.

          You will always get automated attacks, constantly. But they're almost all doing stuff like trying to exploit a 12 year old bug in Wordpress or IIS.

          They're about as sophisticated as any other scammer on the net.

    • spc4761y

      I checked the logs for May for one website I run---65% of failed requests were for PHP scripts (mostly Wordpress). I don't run PHP so I don't worry. The rest of the requests were bots that can't parse HTML [1] and other weird requests. I've been running a webserver, SMTP, SSH and DNS for over 25 years and only once had an issue due to an inside job [2] twenty years ago (hard to protect against those).

      [1] https://boston.conman.org/2019/07/09.1

      [2] https://boston.conman.org/2004/09/19.1

    • aadhavans1y

      Out of curiosity, what are the ramifications of exposing ports 80 and 443? Can these ports even be 'hacked'?

      It doesn't seem terribly unsafe to me, especially if you're serving static pages.

      • koito171y

        In my experience, most of the noise on my web server are bots with spoofed iPhone or Google Chrome user-agents. I see three kinds of traffic patterns.

        1. bogus /wp-login.php requests, or endpoints of presumably insecure wordpress plugins. These bots are pretty dumb and do it non-stop, even if the server constantly responds with a 404

        2. testing recent Apache vulnerabilities by POST-ing to something like /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh . Even if your web server clearly communicates that it's not Apache, the bots still insist on testing Apache vulnerabilities. They also occasionally test vulnerabilities that exist in ancient Nginx versions.

        3. less common, but bots that exist to scrape something from the internet. I remember two years ago seeing a bot whose sole purpose was to document as many registered, valid domain names as possible (I found out about this since they linked a website explaining who they were in their user-agent string)

        Overall, I would say the background noise of HTTP servers is tame compared to what you see for SMTP servers and, to some extent, SSH servers. I happen to also self-host e-mail; logs record failed login attempts about every second. They always pick a username like "admin" or "adm". There's also people who try using your SMTP server as a relay for spam.

        • fpoling1y

          For me the biggest source of noise in logs for a small site is the referrer spam. At some point like 12 years ago I enabled webalizer stats with a public link to the stats page. Soon I had to deal with massive amount of bot requests with http referrer pointing to porn and farmacy ads. That has not stopped after the public link was removed and the stats has started to use a public spam database. And the spam is still there after 12 years.

        • DEADMINCE1y

          > testing recent Apache vulnerabilities by POST-ing to something like /cgi-bin/.%2e/.%2e/.%2e/.%2e/bin/sh .

          Are they really recent vulns though?

        • aadhavans1y

          Gotcha, thanks for the detailed response. I've seen the WordPress login attempts in my own web server logs, and that seems to be corroborated in your comment.

        • hyperman11y

          I've added a /wp-login.php and friends that firewall-blocks the IP of the requester for a week. It greatly cuts down the bot noise.

          • immibis1y

            My competing site can have <img src="https://yourdomain/wp-login.php"> and customers won't be able to view your site after that. Thanks for the free customers!

            • sweetjuly1y

              Yep :) The real trick is to not be vulnerable to known issues, and then mitigate post-compromise like crazy on the off chance you get patch gapped or (very unlikely) zero dayed.

              Blocking IP addresses is extremely silly, especially in an IPv6 world where attacker can easily get access to gigantic numbers of addresses in hard to identify ways (there's no source of truth for what IPv6 range corresponds to one blockable "customer". Some get /56s, others get /48s, etc.). It's security theater which may well just break your service for real users.

          • Beijinger1y

            Can you post the script?

            Obviously I assume you don't run wp. I think wordfence does something similiar.

            • DEADMINCE1y

              It's probably just an nginx fail2ban jail or something that looks for the wp pattern.

      • ozim1y

        99.9999% of issues on 80/443 are apps run on the server not webserver itself.

        It is applications that you run on web server that are exploited.

        So serving static pages is safest thing you can do.

      • chipdart1y

        > Out of curiosity, what are the ramifications of exposing ports 80 and 443? Can these ports even be 'hacked'?

        These are the ports usually employed to serve HTTP and HTTPS traffic, which mean public-facing servers.

        Having a server listening to those ports is the precondition to have web servers running specific types of services, some of which have known vulnerabilities that can be and are exploited.

      • ValtteriL1y

        Ports can't be hacked but the application listening on them can ;)

        You can have vulnerabilities on the server software and its configuration even if you are serving only static content. This should be unlikely if you use up-to-date battle-tested software like nginx without making crazy config changes.

        If you serve dynamic content, that may also have vulnerabilities that hackers can exploit.

    • chipdart1y

      > I once tried hosting a web server at home by exposing ports 80 and 443 to the Internet. Hours later I reviewed the logs, thousands of attempts to hack into my lil Linux server. It spooked me to say the least, so I switched to using cloudflare tunnels instead.

      Isn't this hypothetical risk mitigated or outright eliminated by using stateless apps and periodically redeploying them in the spirit of cattle?

      • metadat1y

        Depends, If they get into the stateless app and hoist that to penetrate into other stuff in your network, they might be able to install an APT.

        • chipdart1y

          > (...) they might be able to install an APT.

          As you're periodically doing clean redeployments, that's not a concern isn't it?

          • immibis1y

            Clean deployments of your entire home network?

    • INTPenis1y

      I noticed earlier this year while deploying a CoreOS VPS with terraform that sometimes you'd get an interesting IP that would receive incoming HTTP requests for interesting domains such as theguardian.com. I of course destroyed and re-deployed the VPS several times so the interesting IPs are lost to me, but it might be worth running a HTTP honeypot as well as an SSH one.

    • DEADMINCE1y

      The traffic doesn't matter if you are sure your setup is secure. Key auth only for SSH, reverse proxy in front of your actual web server and use secured containers or VMs for each service. Throw in fail2an or crowdsec and that's more than enough for a little home linux server.

    • JackSlateur1y

      Every things on the internet is doing exactly this "dangerous things", with the exact same means you have at your disposal.

      Exposing a service is not dangerous.

      It is the same thing when you go to the sub and many people ask you for money : they keep asking, but that will not lead you to your bank account.

      So you have log, this is not an issue, this is not something to be scared of or even cared of.

      Just ignore them, as they are worthless and part of the v4 internet.

    • mikhmha1y

      Yeah this is what keeps me away from self-hosting public facing stuff. To me its like opening a new pipe into your home that is open to the whole world. And I'm too carefree to get the settings down right. So I avoid it all with complete process isolation. Don't shit where you sleep!

      • sureglymop1y

        But couldn't, you, within your home, separate it from everything else? I don't see how it's any more dangerous really.

        • mikhmha1y

          I should clarify. When I mean self host it’s for public facing applications that generate revenue. It involves some transaction in currency?value? between the user. Once money is involved you become a target. I don’t want anything that could be traced to my physical address. I told you I’m careless, I’ll eventually slip up on installing the patches or configuring something right.

          Public facing like serving some static webpages or blog, text content. Yeah do it.

        • Nux1y

          Obviously you need to know how and if you don't then it's always going to look very daunting.

  • tanepiper1y

    We run internal sites that are on the public facing web - the logs from Akamai are a daily list of mostly the same requests to find unsecured Wordpress and MySQL installs, .cgi and php files and paths like "..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../..%C0%AF../etc/profile"

    In 24 hours theres anywhere from 7000-9000 log events just from the CDN

  • JZL0031y

    How do people feel about using docker as a way of avoiding 0 day vulnerability

    It's all for personal use and maybe I'm just cosplaying as a sysadmin but I have apache proxy-pass ing to sets of docker containers. So as long as apache and ssh are kept up to date (on nixos), even if all my services are 0 day'd, they have to also escape the docker containment

  • reincoder1y

    If anyone is looking to run IP metadata based reports on their honeypot, I can suggest IPinfo's CLI (https://github.com/ipinfo/cli). Here is my summary report from Fail2Ban:

    ``` Summary - Total 1490 - Unique 153 - Anycast 0 - Bogon 0 - Mobile 52 - VPN 91 - Proxy 12 - Hosting 1003 - Tor 0 - Relay 0

    Top ASNs - AS132203 Tencent Building, Kejizhongyi Avenue 409 (27.4%) - AS14061 DigitalOcean, LLC 148 (9.9%) - AS135377 UCLOUD INFORMATION TECHNOLOGY (HK) LIMITED 72 (4.8%) - AS16276 OVH SAS 72 (4.8%) - AS206264 Amarutu Technology Ltd 44 (3.0%)

    Top Usage Types - Hosting 955 (64.1%) - ISP 418 (28.1%) - Business 49 (3.3%)

    Top Routes - 43.134.64.0/18 (AS132203) 48 (3.2%) - 183.81.169.0/24 (AS206264) 44 (3.0%) - 43.156.192.0/18 (AS132203) 36 (2.4%) - 43.130.0.0/18 (AS132203) 36 (2.4%) - 43.134.0.0/18 (AS132203) 36 (2.4%)

    Top Countries - United States 276 (18.5%) - Singapore 221 (14.8%) - China 141 (9.5%) - France 88 (5.9%) - Japan 86 (5.8%)

    Top Cities - Singapore, Singapore, SG 221 (14.8%) - Santa Clara, California, US 100 (6.7%) - Hong Kong, Hong Kong, HK 80 (5.4%) - Tokyo, Tokyo, JP 61 (4.1%) - Amsterdam, North Holland, NL 50 (3.4%)

    Top Regions - Singapore, SG 221 (14.8%) - California, US 134 (9.0%) - Tokyo, JP 86 (5.8%) - Hong Kong, HK 80 (5.4%) - New Jersey, US 70 (4.7%)

    Top Carriers - Africell 12 (0.8%) - Claro 12 (0.8%) - Vivo 12 (0.8%) - WINDTRE 12 (0.8%) - Telekom 4 (0.3%)

    Top Privacy Services - TunnelBear 36 (2.4%) - Best Proxy Switcher 12 (0.8%)

    Top Domains - ovh.net 60 (4.0%) - googleusercontent.com 24 (1.6%) - prod-infinitum.com.mx 16 (1.1%) - poneytelecom.eu 12 (0.8%) ```

    Here is the command I used:

    ``` cat /var/log/fail2ban.log | ipinfo grepip -o | ipinfo summarize ```

    The CLI is free to use. You can also do `bulk` enrichment.

    ``` cat /var/log/fail2ban.log | ipinfo grepip -o | ipinfo bulk -c > fail2ban_ips.csv ```

    Disclaimer: I work for IPinfo. However, the CLI is free to use, and the bulk feature will usually work within your free tier limits. Ping me if you have any questions

  • ProllyInfamous1y

    I somehow found myself in charge of a computer lab two decades ago... and idiotically set up admin controls via SSH.

    The entire lab was down for almost a week [immediately hacked], and then I suddenly moved a few states away.

  • simple101y

    Why is 345gs5662d34 the 2nd most tried username after root? Bizarre.

  • charles_f1y

    I opened my personal server's 22 to the world because I screwed up my vpn config a couple weeks ago. I just had a look at the auth log and closed it again. It is non-stop.

  • microbass1y

    A perfect example of why one should use SSH over a mesh network like Tailscale, and don't expose over the public internet. No attack surface means no attack.

    • stanac1y

      I love TS just for this reason. All ports are locked and ssh-ing is possible only via TS. And for public facing web apps I open only 80 and 443.

      Does anyone have any experience with CF tunnels on free account? Is it actually free for smaller apps with less than 1TB of traffic per month? I was wondering about switching to CF tunnel which would mean I could also close 80 and 443 ports and block China (because I read somewhere that most of DDOS attacks come from Chinese locale botnets).

      • microbass1y

        For some additional peace of mind, you could also use something like Authentik in front of your web apps, so you don't expose the apps themselves, only Authentik. You can then use the IDP of your choice within Authentik for authentication.

        • stanac1y

          Thanks, I was thinking about small but public project.

      • andylynch1y

        Yes, CF tunnels are $0 for very small users. I have this, as do many others, as a reverse proxy for stuff like Home Assistant and it works great.

        • stanac1y

          Thank you, I'll have to try them

  • efilife1y

    I was always wondering, how do you guys geoblock entire countries/ip ranges? Do you just use free ip lists you find on the internet?

  • lithiumii1y

    My new VPS got an SSH attempt in 5 minutes after I purchased it. I'm now in the progress of running a similar honeypot experiment.

    • cess111y

      If you push it you can scan the entirety of IPv4 in about five minutes.

  • jcynix1y

    I've been running self-hosted servers for the last 25+ years without an incident and its less complicated than it might seem if you learn a bit about securing unix-based systems (ok, I already had 10+ years of server admin knowhow for various systems, but anyway, it's not rocket science ;-). Yes, an hour or so after you connect any machine to the Internet, you'll see attempts to "talk" to your server. So don't wait to set up basic security. But it actually has never been so easy to "just give it a try" (see below), with all the virtual offerings today. So here's a short/raw sketch of basic things you'd need to do:

    1. 25+ years ago I used http://easyfwgen.morizot.net/ to generate an iptables based local firewall. Still works fine (then and now tweaking some things) and allows only certain ports too be accessed at all. I just open email, ssh and a web server.

    The generator is well documented and still works, although it would be nice to see an updated version to newer firewall software like pf.

    2. server configs:

    edit /etc/hosts.deny --> restrict all by default

      ALL: ALL
    
    edit /etc/hosts.allow --> allow your service providers networks, e.g.

      sshd: .t-dialin.net
      sshd: .dip0.t-ipconnect.de
    
    So you can connect to your machine for further setup, but not the whole world.

    3. set up sshd:

    edit /etc/ssh/sshd.config

      # allow key-based access only
      PasswordAuthentication no
    
    Maybe change sshd's port (reduces log file entries) but don't forget to allow this port in your iptables setup and your /etc/hosts.allow

    People have opinions an key-based access, I know. But my private and public key is stored in various secure locations, including my phone (password safe) and I can access my server even from my Android phone or tables via Termux.

    4. set up email (I suggest postfix as an MTA):

    configure restrictions in /etc/postfix/main.cf, e.g.

      # restrictions in the context of the RCPT TO command
      smtpd_recipient_restrictions =
            reject_invalid_hostname,
            reject_non_fqdn_hostname,
            reject_non_fqdn_sender,
            reject_non_fqdn_recipient,
            check_sender_access hash:/etc/postfix/sender_access,
            reject_unknown_sender_domain,
            reject_unknown_recipient_domain,
            permit_mynetworks,
            reject_unauth_destination,
            [...]
    
      # restrictions for clients connecting
      smtpd_client_restrictions =
            reject_unauth_destination,
            check_client_access hash:/etc/postfix/access_client,
            reject_unknown_client,
            reject_unauth_pipelining
    
    This heavily reduces the amount of spam you'll see. I add greylisting too, as this even nowadays reduces even more unwanted traffic. Combine that with spamassassin if you like. This setup gives me maybe one spam per day reaching my inbox (actually the spam subfolder).

    5. Learn by doing (not just reading stuff on the Internets ;-), that is, set up a machine, e.g.

    If you'd like to experiment a bit, take a look at Hetzner's unexpensive cloud servers, these are easy to set up (incl. a virtual firewall in front of it) and take down after some experiments of a failure. You can do this in Hetzner's web interface, even if you misconfigure your server to be unaccessible. Cf.

    https://docs.hetzner.com/cloud/servers/overview/

    Tip: Hetzner's web interface allows you to pre-define an ssh key which they'll install automatically on your new machine (but they leave password login enabled, so change that asap).

    Disclaimer: I'm just a happy customer, no other relation. And it might be as easy to do this with Digital Ocean, which have some nice tutorials too, for example on the set up of a web server:

    https://www.digitalocean.com/community/tutorials/how-to-inst...

    Last but not least No Starch Press overs some nice books like "How Linux Works" or "The Linux Command Line" (if you're not sure about that) or even "Linux Firewalls: Attack Detection and Response" ...

    You learn most by trying.

    I'm now heading for the beach to enjoy some offline adventures and will answer questions later if needed.

  • chickenfish1y

    I guess may the compromised host was probably also use same weak password as it's Brute force other host.

  • throw1567542281y

    My website backend APIs get repeated attempts at javascript prototype injection, all day, every day.

  • braza1y

    (Long shot) I really would like to USA a spare machine for web serving a Jupyter Notebook server, but I did not found a single resource that blocks everyone except a single IP or something like this. Supper annoying to pay some cloud providers to have a resource that I already have.

  • e401y

    We use port knocking and haven’t had a single hack attempt in many years.

  • pompompurin1y

    How did he expose his honeypots and make the bots aware of his existence?

    • themoonisachees1y

      If your server has something that listens on port 22, you just have to wait for like 5 minutes

  • msephton1y

    I wanted to read more about the interesting part!

  • slt20211y

    dont ever run publicly exposed production SSH. If there is vulnerability in your version of ssh, you risk getting pwned.

  • figassis1y

    Most of this nonsense disappeared when I adopted wireguard and later Tailscale.

  • 1y
    [deleted]
  • RecycledEle1y

    I am amazed we have not yet said "Hands off!" and coordinated physical interventions against the scum who attack our electronic brains.

    Is it so hard to kick in the doors of those whose IP addresses are used to try to hack honeypots?

    This lack of action is why I oppose all law enforcement. Until they do their jobs, they do not need to be paid.

  • glub1030111y

    [dead]