I don't mean to support or refuse the author's main points or analysis, but you might like to know that the Chrome team is currently working towards shipping the Topics API. I have strong opinions about it but I will try not to editorialize.

My high-level understanding is that they're going to run an ML model over your browsing history (locally on your device) to build a list of "topics" that you care about. Sites you browse can use the Topics API to pull a set of these interests from the browser to show you "relevant" ads. Mozilla has taken a negative position against this standard.

https://privacysandbox.com/proposals/topics/

https://github.com/mozilla/standards-positions/issues/622

When I started having this problem logging into a certain credit card co.'s website beginning with about Firefox 105.0.2 on Fedora 38, I was told by their apparently outsourced customer service that I had to use Chrome, which I don't have installed there and couldn't try. Yeah, they wanted me to use LogMeIn so they could fix the problem, too. Right.

Firefox on Android was still working, though, loathe as I am to put passwords of any significance on my phone. Doesn't directly address your question, which I'd like to know the answer to as well.

A third browser... like what? Chrome and Firefox are all that exist now, unless you have access to a Mac with Safari.

I've had sporadic issues with Firefox not working on work-related sites one day when the previous day it worked just fine.

I have ublock, privacy badger, decentraleyes, canvas blocker, facebook disconnect, and duckduckgo privacy essentials installed.

I would go through and disable each extension in order to see if it was the cause of the issue, and so far, every single time it has been duckduckgo privacy essentials that is breaking websites for me.

I think I should remove it at this point, but who knows? Maybe it's protecting me from something that I don't see.

With Firefox you can toggle some settings that will make much harder to generate useful fingerprints. That's already a massive privacy difference.

Why would chrome give that information away? That's Google's most valuable resource.

https://privacytests.org/ shows some good data what each browser lets through/exposes for websites.

Chrome will indeed divulge more information than other browsers but only on the condition that you have opted-in for such collection.

“The Chrome User Experience Report (CrUX) provides user experience metrics for how real-world Chrome users experience popular destinations on the web. This data is automatically collected by Chrome from users who have opted in, . . .”

Taken from https://web.dev/crux-and-rum-differences/

You're conflating a downside of using Chrome and the reason they think Cloudflare blocked them.

Blocking bots in the first place should not be acceptable since bots only act on behalf of humans. What should be blocked is abusive behavior that actually impacts the site - a single one off GET to what should be a static page should never be blocked, yet that's what CF does.

Furthermore, all bots worth their salt as far as threats go enable js and do everything they can to appear like a normal browser.

> The moment the Cloudflare dictatorship becomes less benevolent…[]

In my eyes they have already done that. ICYMI I highly suggest checking out their response and subsequent blog post around the Kiwifarms incident.

That whole debacle was enough to prove to me they learned nothing and are going to continue down this path. I migrated web services and closed my account with them shortly after that whole thing.

Cloudflare routinely ignores abuse reports for its network and takes no responsibility for the utter garbage being carried across their network. It’s almost comical how they so desperately cling to the claim that they are “just a dumb pipe” on one side of the house and on the other a “serious security vendor” who is “protecting the web” while blocking out users simply for the “crime” of trying to preserve their privacy.

If they wanted to convince me they had the web’s best interest at heart they wouldn’t host half the sites they do. They would actually respond to abuse reports and take abusive websites offline rather than wait for it to hurt their bottom line and reputation before taking action but they don’t.

Website owners can just stop using cloudflare though…

funny enough... I called out Cloudflare for the pariah it is, and got downvoted and flagged

Well, Chromium is quite popular with the security conscious on Linux. At least it was when I was using ArchLinux, they had some good custom build script versions.

> Also, fingerprinting isn't always "bad" -- any business who takes credit cards online, wants to try to exclude people who will commit fraud (because they might have done it before.) Preventing fingerprinting, means you prevent certain anti-fraud, which means that you see higher prices and more friction doing commerce online, which also affects your experience. The connection is just much less direct.

By the same argument you could say it should be fine for a physical store to refuse service to anyone who they get a bad feeling about or don't want to serve. But if you permit that then you're immediately opening the door to racism etc., which we consider socially unacceptable. It should be the same for websites too - I bet all these browser fingerprinting techniques just happen to mean better service for people who can afford the latest iphone.

Tracking is establishing your identity. Try using a private mode Firefox via a VPN. Half of the web is completely unusable. You get put in unsolvable catchpa hell as punishment for being anonymous.

That's implausible. Using finger printing for fraud detection would only catch someone using different cards on the same machine. Once a card is deemed stolen it stops working so it's unnecessary for that scenario. That doesn't even go into fake fingerprinting some browsers/plugins.

The price is the highest the market will pay. Increasing that price means few customers lower revenue. Fraud is a cost to the business they must pay out of profits because if they tried to increase prices demand would drop.

> business who takes credit cards online, wants to try to exclude people who will commit fraud

How bad is it nowadays? Can't you just enforce 3DS2?

Famous last words: "There are more important things to worry about than privacy".

If you've read history (and maybe you have, or not) privacy is a human right. When privacy goes away, then everything else goes away. Ask anyone over 60 in Germany or Romania (that was not WITH the army or the Police/Security services) and they will tell you how nice life is without privacy.

But hey, sure, 1) privacy doesn't matter, 2) you got nothing to hide, etc etc.

Ads are used to manipulate people into doing things they would not otherwise do, which very much affects "real world stuff". Mostly into wasting money on useless crap, but also worse. What ads uBlock can block is limited by Chrome. What "ads" disguised as content you are shown is affected by what information Google collects and lets other people collect. The internet isn't a nerd safe space anymore - what goes on here often affects real people.

I sympathize with your frustration, but you also have to admit that Cloudflare is tasked with an impossible problem: from a sea of requests, identify those that are coming from robots that are disguised as humans.

So there is no perfect solution. You can't use strong identity because a user can share their identity with a robot. You have to use a crapy heuristic that only works most of the time (or tell site owners it's an application layer problem and use this SASS solution to solve the problem).

I mean you admitted that you run a crawler. Cloudflare has detected that you run a crawler and has wants you to prove that you're human to access sites on their network. It actually sounds like their product worked.

In any event, there should probably be better regulation around how this blocking is handled so that users aren't being unjustly blocked. If you want to run a crawler, how do you do it ethically so that you aren't targeted and your traffic blocked? If Cloudflare blocks you from accessing one site should that block extend across their whole network? How long should it last? How do you appeal the block if Cloudflare's heuristics falsely block you? If you're in a life and death situation and need immediate access to medical information and Cloudflare unjustly blocks your access and it causes harm, who's at fault? Etc.

You're being a little dramatic. It's incredibly unlikely that millions of innocent users have been blocked, and unless you have data to the contrary you shouldn't make such a claim.

You know what else is harmful to the concept of the open internet? The enormous malicious botnets and other endemic problems that require a solution like CloudFlare.

Right. It feels silly to point out specific things when just about everything about these verification checks deployed by every megacorp throws you into a universe of suffering.

If there's a place to start, it would be with eliminating the infinite challenge loops. Bad enough that IP blocks get outright blocked. Bad enough that I have to decide whether or not that blurred sliver of the edge of the wheel+shadow constitutes being part of the bicycle. Not to mention the humanitarian betrayal of the absolute highest form to farm the free human labor to train AI models when they are simply trying to browse the $#@%ing internet.

You're going by the specified, designed use cases of those technologies.

Every spec is a three-edged sword: the spec, the intent of the spec, and the use of the spec in the wild.

In practice, Cloudflare does a pretty good job on far-more-than average of gluing together some heuristics in an unspec'd way to filter traffic. It sucks because you can't plan around it, but that's rather the point because the malicious actors are trying to plan around it also.

(ETA: Hacker News rate-limited this post. In theory, I could have set up a sock-puppet to try and work around that, but then they would catch that too and I'd be out two accounts. So I just waited out the limit. Measure and counter-measure. ;) ).

> Let's be honest here. Your service has likely caused millions of people harm who one day to the other are suddenly blocked from half the WWW

If this was true, Cloudflare wouldn't be a good product used by a lot of sites.

And even when it doesn't block you completely, it delays website loading, makes you jump through frustrating captchas, etc.

It's probably third in the list of frustrating web behaviors in the past couple of years (behind GDPR popups and registration/paywalls that seem to have gotten much worse recently).

And somehow there are some sites that I get CF delay walls on every time I visit.

This feature is utterly broken for a good web experience; it pushes users away from sites which use it.

Every time that "checking your browser" page comes up for a legitimate user should be considered a failure. Sure, it can maybe happen a few times in a thousand, but the feature is utterly broken if it comes up every time I visit the same site from the same browser not in private mode.

Your alt solution is what?Everyone should build their shit to handle millions TB/s of DoS traffic?

This is a very nasty comment. I was wondering if I could find some things that could lead to an exception.

But I'm pretty sure that millions of users aren't using stuff like w3m pager ( https://news.ycombinator.com/item?id=34175754 )

We're all technical here, we are the edge cases. We use exotic software / combos. Let's not get carried away here

The PM of cloudflare uses Firefox, I sometimes use Firefox and I don't notice any difference ( concerning this use-case at least).

If you want help, perhaps describe the actual use-case that is blocking you to him. He shared his email.

- country

- software ( VPN, ... )

- browser

- OS

- traceid

- ...

Either way, buying shady proxies as you mentioned is already a warning flag.

While using Firefox is not :)

I use a VPN 99% of the time and I definitely see more cloudflare on my systems (as opposed to when I'm bored and surfing in the library) and see the "checking" screen. It does seem to be a lot better than last year though when it seemed I was getting a captcha every time I turned around

Same for me. Although i have ublock and canvas block and I have 3rd party cookies blocked.

2-3x per day i get some sort of "click here if you're a human" thing from cloudflare.

Same for me

I've definitely seen this from time to time. I used to work for an ISP and we would occasionally, in the office, get "Your system is sending too many automated requests" from Google. Usually one of our customers had gone off the rails with some sort of amateur scraping, but this was always a pain to debug. I think we talked with Google's NOC and just had our rate limit increased or something like that.

Do you see the challenge and then you're able to pass? Or does the challenge loop forever?

Would you be able to send me a rayID of a failed challenge so I can take a loop? It sounds like you can use https://gitlab.com/users/sign_in to generate one.

You can either reply in the comments with the ID (no PII), or email me at amartinetti at cloudflare.com and I'd love to dig into it.

We're building Turnstile because we want to make challenges a better system than CAPTCHA. It sounds like for you it's worse, and we want to fix that.

> Just a challenge over and over.

It must be intentional. Not unlike the endless loop of frustratingly slow-fading reCAPTCHA challenges that don't go anywhere. The user gives up after some time, but doesn't see any explicit error or page blocking their access. I imagine it must be quite effective.

There’s a patent for that. The catchpa loop is basically a honeypot for bots… and privacy-conscious legitimate folks.

Such a classic and incredibly annoying SaaS PM move. Pinky-promise that you mean well, pretend to be invested in the issue, ask customers to supply evidence and say you'll look into it, followed by radio silence and no follow up whatsoever.

Incidentally, another Cloudflare PM for Pages asked me to do the same thing--I shared my account ID, the request, the problem, timestamps, etc...never heard back ever, request went straight into the void.

Here's some loop samples;

- Gitlab; Ray ID: 7f3961b4ec46c443

- Zabbix; Ray ID: 7f39624d982bc32e

- NameMC; Ray ID: 7f3962e68d251871

- Camelcamelcamel; Ray ID: 7f3962eb9cbb421f

Easily can recreate at least the never ending loop by flipping on ublock origin's 3rd party scripts and 3rd party frame blocking, which matches their recommended medium settings.

https://gitlab.com/users/sign_in 7f3cc1d59acd31e5

https://steamdb.info/login/ 7f3cc161bf85dbd9

https://www.zabbix.com/forum/ works

https://casetext.com/ works

https://namemc.com/login 7f3cc182b8c20ccf

https://spinroot.com/spin/whatispin.html works

https://camelcamelcamel.com/ 7f3cc171f96d2b9b

Here's a handful:

- 7f395b5ddfe43a54

- 7f395ca09bfa3a54

- 7f395d8afaf73a54

- 7f395f075e33690d

- 7f396102afef35fd

I also cannot access my VPS provider when using firefox.

ray id 7f3a169d4e630306

I previously had the same problem with ungoogled-chromium as well (regular chromium worked), but I guess it works now after 2-3 loops.

all from Opera Mini:

- https://gitlab.com/users/sign_in 7f3e45c3cebfb90f

- https://steamdb.info/login/ 7f3e4a04bf7a0e39

- https://www.zabbix.com/forum/ 7f3e4b681f8f1cc6

- https://casetext.com/

7f3e4cab4af40b05

- https://namemc.com/login 7f3e4debdf6cb7f1

- https://spinroot.com/ loads normally, no delay or blocking

- https://camelcamelcamel.com/ loads normally, no delay or blocking

Adammartinetti, I appreciate your interest in doing this, but would love to hear that CF maintains a giant white board in the developer area with the name of every TLS 1.3 web browser known to mankind (the same data on a Group Policy-enforced internal home page would be even better), to reinforce the idea that it takes more than Google to make the world go round.

Personally, I'll add myself to the list of people who think you've created a game you can never win, and thus shouldn't be playing.

gitlab 7f39759e1abe1bce

casetext 7f39762f693733e4

steam 7f397694995aa3b7

all over firefox

- Gitlab: 7f39707d0fa023af

- Zabbix: 7f3970eabe8ff196

- SteamDB: 7f396f534b0400d2

- Casetext: (works)

- NameMC: 7f3971a01a22d5a8

- Spinroot: (works)

- Camelcamelcamel: (works)

Sadly, the fact a given site works for you or me is no guarantee it works for someone else.

These bot detection systems tend to use all manner of imprecise statistical heuristics and weird fingerprinting.

Perhaps AegirLeet has a graphics card that a popular web scraper pretends to have. Maybe they're in a suspicious timezone. Maybe they've installed a font usually only found on a different operating system. Maybe I'm never blocked because I have an excellent IP reputation, due to regular visits to approved websites.

that’s because websites have evolved into web apps.

All browsers so far have come and gone in popularity. Even when it seemed unimaginable.

This would be great if it didn’t have any downsides. China has a system like that: QR code to login everywhere. Everything is linked to your phone number which is given after taking a picture of you and official ID.

We are gonna have to live in a slightly bot-rich society to keep this at bay.

It starts with browser control. And then, ends with needing human verification to ssh into a server that you own. Let’s just build better security.

The problem isn't that the hack only works in Chrome, it's that the system being proposed is inherently terrible regardless of how it's implemented.

There is no such thing as a reliable standard for browsers to verify that users are human that does not harm the open web or threaten user autonomy and accessibility. Every single accessibility standard and user choice about extensions and access is abusable by malicious actors, and every security measure to block abuse of automated scraping or access also blocks valid use cases.

Making it a web standard won't change that fact.

Yes, an open standard that any browser could use to prove human interaction would be great. It's also impossible, of course; all attempts so far lock in specific software or hardware stacks and then pretend that bots can't use those systems, guaranteeing both false negatives and false positives.

They're booing, but you know you're right. ;)

Cloudflare controls about 19% of websites. Of the websites that use a CDN, 80% of them use Cloudflare [0] (in a note at the bottom of link).

[0] https://community.cloudflare.com/t/statistically-speaking-wh...

Maybe some kind of public nuisance law applies then? Akamai knows their place and doesn't tell me to go pound sand when I'm trying to read somebody's obituary. Incidentally:

legacy.com

Ray ID: 7f3e7bad3afbb731

That's the difference to me.

And who defines is an allowed crawler? Is Google the only allowed company to crawl the web? Isn't that the definition of monopoly?

Could anybody still create a new search engine nowadays?

Website owner complains elsewhere in this topic of blocked users (country-specific):

https://news.ycombinator.com/item?id=37050774

Hah, you know, most crawlers were fine. The only one that actively DDOSed websites was fucking Yandex. It doesn't respect robots.txt and it will actively fight against any rate limits by spawning connections on new IPs the moment one is blocked

> Everyone forgets that websites become almost unusable because of crawlers and bots.

No one "forgets" this because it isn't true.

I don't agree. Maybe don't run WordPress that makes 100+ dB queries before first page load on some cheap $2 vps.

There are all kinds of tools that you can easily deal with bots and the large DDOS your ISP can handle for you if you are willing to pay for it.

Cloudflare controls about 19% of websites. Of the websites that use a CDN, 80% of them use Cloudflare [0] (in a note at the bottom of link).

[0] https://community.cloudflare.com/t/statistically-speaking-wh...

They [effectively] do prevent you from moving - they're guilty of vendor lock-in and running a passive, public man-in-the-middle attack platform (how many folks are using Cloudflare for DNS - either directly, or because their ISP is?)

Keep in mind that a bot will be picking some permutation of its stock library of UA+metrics info, and generating truly-random values for other more continuously-valued parameters (e.g. timing between actions), to try to find a combination that satisfies a backend integrity-check.

A "try again" just means "you haven't succeeded yet." If that's all you get, you're getting zero bits of new information — so you can't do anything other than to assume it was your timing that looked weird, and keep trying. (And you might be dealing with even more noise, e.g. trying to have the bot calibrate itself toward a very low human-tuned request rate limit, where above-rate-limit responses look no different than integrity-fail "try again" responses.)

Suddenly getting a (maybe permanent) hard-fail, meanwhile, means that you said something the integrity-checker really didn't like.

Presuming you have a lot of IP addresses to send requests from, you can then do many experiments to bisect the difference between a hard-fail and soft-fail, and use that to blacklist values from your UA+metrics library. It's free entropy!

While I'm inclined to agree, it's an old rule-of-thumb to give a potential attacker as little information as possible so they have to do the legwork to get un-broken.

(I yearn for a world where auth challenge failures give proper error messages so I can figure out why my regular, human-used authentication channels aren't working).

Heuristics are about optimizing between false positives and false negatives.

Many headless-browser stealth techniques involve rotating between the signatures and reflected metrics of real — but niche and/or ancient — User-Agents. (For some reason, the developers of these stealth systems think that variety beats commonality. Maybe it makes sense if they're specifically trying to overcome Apache mod_security's signature-based UA blocking or something.)

It turns out that when you actually see one of these UAs in your server logs, it's far more (99.99%) likely to be a stealthed bot that picked that UA out of a bag, than it is to be an actual niche/ancient UA.

In the case of the niche UAs, this is a tragedy of the commons.

In the case of the ancient UAs, though, there's no downside to blocking them entirely — because if the traffic is going through Cloudflare at all, then you're already requiring of the client a minimum version of TLS that the real old UAs can't even speak. So the only things actually saying they're that old device — but managing to get through an HTTP request at all — are stealthed bots.

This is the tech equivalent of "we don't serve your kind here", and limits any attempt at competition.

Why shouldn't a store get to decide the $protected_class of which customers it will do business with?

Long term, I can imagine this becoming a corollary to Net Neutrality. Whenever that finally becomes a thing, the next step could be a law/rule that "public web sites need to be accessible by the public". Not that developers need to test their work in dozens of different browsers, but that they can't actively choose their customers.

In principle? Of course. I mean I remember blocking Yandex bots from hammering some e-commerce site on shoestring budget.

But each person developing a web scrapping bot realizes at most after a week that being honest with User-agent has negative impact on how well it works, and changing it to existing browser takes literally seconds.

No need to simplify so much that only false dichotomies remain. My brain might be wired up differently but its capacity for nuance and reason is fully intact :)

No one is forcing the website owners to sign up with Cloudflare to enable this service with these aggressive configurations, and yet I understand why they would even just pre-emptively. It's cheap and effective, there's no denying that.

It is Cloudflare Inc. (66,59USD, +23.57USD/54.79% YTD), however, that architected the solution, markets it as a service, and controls it as a core part of their (i.e. everyone's) internet architecture.

As a serviceprovider they could be better at informing their customers of these unintentional side-effects and how they impact otherwise innocent visitors, but whose mental disorders/impairments cause them to be flagged for and having to undergo additional verification steps disproportionately more than others, likely due to some atypical behavioural patterns they show and their often adjusted hardsoftware setups producing an unconventional signature.

Some modifications to the system could probably be made on the architectural level too. We can get people in wheelchairs to the top of the empire state building, surely we can also find a solution that allows us to enjoy the benefits of these protective measures without wrecking the web's inclusivity and accessibility this much every time the measures need to be stepped up.

Am I asking for too much, what do you think?

  "There must be some way out of here," said the joker to the thief
  "There’s too much confusion, I can’t get no relief"

Wow, that doesn't sound like a terrible idea!

Which is honestly surprising in this area where it feels like privacy, anonymity and human verification are incompatible with each other.

I am trying to minimise my time wasted by websites, which is hard to balance with privacy, one other one is the repetitive consent forms (if you don't retain cookies, it's a never ending process). I think consent forms and human verification are the 2 biggest human time wasters.

Cloudflare is "helping" more than enough already, but thanks for the suggestions.

Besides, those solutions have far too much in common with blackmail/extortion to my liking. Either you continue to suffer this structural harassment, or hand over all your bits and maybe in specific cases suffer slightly less! :)

Pay us with your information to make your problem go away.

This should be illegal.

Time to switch from PayPal to FedNow. FedNow is run by banks and the Fed, which are regulated as to whom they can refuse to server.

You know I knew there was going to be something I really didn’t like about passkeys, and here it is

Does Microsoft have much control over Windows Hello's key attestation? It's not clear to me how they could pull that off, other than just relying on TPM attestations which are easy to obtain as long as you can buy a TPM that works with your motherboard.

Egypt do block VPNs in much more aggressive way than Cloudflare. Also this is my first time to hear that cloudflare is blocked in Egypt. People will even complain that they cannot connect to their cooperate VPNs.

Blocking cloudflare ip addresses means that half of the internet wouldn't be accessible from Egypt. its closw to blocking port 443 because some people use DNS over https.

disclaimer: I'm Egyptian living in the US.

Also, the scope of PATs is vastly different than WEI. Think of PATs as a "probably a human" signal that mostly replaces the need for CAPTCHAs.

https://blog.cloudflare.com/how-to-enable-private-access-tok...

Of course not, but it is less likely to trigger manual recaptcha on the site you're trying to visit in Guest mode.

They obviously don't do it this way and implement more roundabout ways because it's not as simple or straightforward.

Last time I checked when I hotspotted my T-Mobile (US) phone, hotspot clients got only an IPv6 address, with 6to4 [edit: no, NAT64] translation used to reach IPv4 addresses.

This breaks OpenVPN, which insists on both endpoints being one or the other.

> You're basically saying this behavior is acceptable and should be considered normal and should be expected to become the norm. If you think IPv6 is mostly pointless, I think you're unaware of the fact that a significant majority of phones already use IPv6 most of the time they're on cellular.

Can you point to where I suggested that? Or did you misread the comment?

I would imagine they have a list of the default / max DHCPv6 blocks that ISPs hand out. They are generally public knowledge so it would be easy for them to say Comcast hands out /56 so when blocking they block at that level for the ASN.

MAC is a weird thing, it is important for ethernet to work efficiently. But in actuality it only travels to about next router from machine...