To be fair, I find it hilarious that every browser includes the “Mozilla” string in its user agent, dating from the late 90’s. As much as it pains me to say it, Google may be right here: the header is at best vestigial.
I think it is weird that they still do; does anyone bother checking that part when sniffing anymore? I’d be surprised if anyone has for the last fifteen years.
I know there are webmasters that use its presence to distinguish between bots (which typically don’t have it) and browsers (which usually do). It’s a heuristic, but it’s actually really good.
Looks like a configuration error from the Web server or app. Maybe they just tell Nginx or their app to deny anything which is not a browser, forgetting to handle special cases like RSS.
Looking at the code it was actually a request to SquareSpace, and the poison seemed to be mention of “Twisted”. Best guess they are trying to ban Scrapy which uses Twisted internally.
I’ve also seen CDNs reject requests when the User-Agent string contains “python” or lacks “Mozilla”. I guess lying is just part of HTTP these days.
Perfect is the enemy of the good. Anyone might come into my house and rob me, but if someone knocks on my door and tells me they’re going to rob me, I’m still not going to let them in just because they asked permission.
GitHub uses User Agent sniffing. I set my User Agent to “Firefox” (general.useragent.override) and some features on the site no longer work and GitHub complains that it doesn’t support old browsers.
This seems like a fine plan, and other browsers have tried/proposed/supported this in the past. But because this is Google, I have to ask: what other advantages could this give to a browser-building search-providing privacy-invading ad broker?
(Actual question. I’m not coming up with anything myself.)
It would force/encourage browser vendors and web servers to support the Client Hints mechanism, which is a much more sane way of requesting adapted content….and which Google will no doubt embrace and extend.
This is the lens through which I try to view Google’s actions as well.
Is it possible that Google thinks that the loss of UA information will make it harder for their competitors to fingerprint users, but that their penetration makes it less of a problem to them?
I find it curious that the Blink team at Google takes this action in order to prevent various other teams at Google from doing harmful user-agent sniffing to block browsers they don’t like. Google certainly isn’t the only ones, but they’re some of the biggest user-agent sniffing abusers.
FWIW, I think it’s a good step, nobody needs to know I’m on Ubuntu Linux using X11 on an x86_64 CPU running Firefox 74 with Gecko 20100101. At most, the Firefox/74 part is relevant, but even that has limited value.
They still want to know that. The mail contains a link to the proposed “user agent client hints” RFC, which splits the user agent into multiple more standardized headers the server has to request, making “user-agent sniffing” more effective.
Oh. That’s sad. I read through a bit of the RFC now, and yeah, I don’t see why corporations wouldn’t just ask for everything and have slightly more reliable fingerprinting while still blocking browsers they don’t like. I don’t see how the proposed replacement isn’t also “an abundant source of compatibility issues … resulting in browsers lying about themselves … and sites (including Google properties) being broken in some browsers for no good reason”.
What possible use case could a website have for knowing whether I’m on ARM or Risc-V or x86 or x86_64 other than fingerprinting? How is it responsible to let the server ask for the exact model of device you’re using?
The spec even contains wording like “To set the Sec-CH-Platform header for a request, given a request (r), user agents MUST: […] Let value be a Structured Header object whose value is the user agent’s platform brand and version”, so there’s not even any space for a browser to offer an anti-fingerprinting setting and still claim to be compliant.
I have a use case for it. I’ve a server which users run on a intranet (typically either just an access point, or a mobile phone hotspot), with web browsers running on random personal tablets/mobile devices. Given that the users are generally not technical, they’d probably be able to identify a connected device as “iPad” versus “Samsung S10” if I can show that in the web app (or at least ask around to figure out whose device it is), but will not be able to do much with e.g an IP address.
Obviously pretty niche. I have more secure solutions planned for this, however I’d like to keep the low barrier to entry that knowing the hardware type from user agent provides in addition to those.
What possible use case could a website have for knowing whether I’m on ARM or Risc-V or x86 or x86_64 other than fingerprinting?
Benchmarking and profiling. If your site performance starts tanking on one kind of processor on phones in the Philippines, you probably want to know that to see what you can do about it.
Additionally, you can build a website with a certain performance budget when you know what your market minimally has. See the Steam Hardware and Software Survey for an example of this in the desktop videogame world.
Finally, if you generally know what kinds of devices your customers are using, you can buy a bunch of those for your QA lab to make sure users are getting good real-world performance.
Any site that offers you/administrators a “login history” view benefits from somewhat accurate information. Knowing the CPU type or window system probably doesn’t help much, but knowing it’s Firefox on Ubuntu combined with a location lookup from your IP is certainly a reasonable description to identify if it’s you or someone else using the account.
There are terms I’d certainly like sites to know if I’m using a minority browser or a minority platform, though. Yes, there are downsides because of the risk of fingerprinting, but it’s good to remind sites that people like me exist.
Though the audience here will play the world’s tiniest violin regarding for those affected the technical impact aspect may be of interest.
The version numbering is useful low-hanging-fruit method in the ad-tech industry to catch fraud. A lot of bad actors use either just old browsers[1] or skew browser usage ratios; though of course most ‘fraud’ detection methods are native and just assume anything older than two major releases is fraud and ignore details such as LTS releases.
[1] persuade the user to install a ‘useful’ tool and it sits as a background task burning ads or as a replacement for the users regular browser (never updated)
There is a draft https://user-agent.globalcode.info/ (probably not used by anyone :-) that suggest not using User-Agent string at all or encode it in form of URI (thus machine readable).
User Agent sniffing still used by pages to find out which processor architecture and operating system the user is running when offering downloads. I don’t know any other way of finding out that information.
Operating word here is upon request. I’ve never bothered my wife with it, other then by demonstration, doing it on my own phone/computer, but she now painfully clicks on “Manage Choices” or whatever on the GDPR popus and tells sites she doesn’t want tracking and personalized ads. If the site asks her “are you on an iPhone or on your Mac right now, she might click through. If they ask “do you also have this graphics card and did you last update your laptop a week ago” and stuff, she’ll stop and say “hey I don’t wanna go to this stupid meme site and give them all of that.
To be fair, I find it hilarious that every browser includes the “Mozilla” string in its user agent, dating from the late 90’s. As much as it pains me to say it, Google may be right here: the header is at best vestigial.
I think it is weird that they still do; does anyone bother checking that part when sniffing anymore? I’d be surprised if anyone has for the last fifteen years.
I know there are webmasters that use its presence to distinguish between bots (which typically don’t have it) and browsers (which usually do). It’s a heuristic, but it’s actually really good.
I had to change my feed reader’s user-agent to lie because of this. It’s nonsensical, of course — RSS and Atom feeds are made for bots!
Looks like a configuration error from the Web server or app. Maybe they just tell Nginx or their app to deny anything which is not a browser, forgetting to handle special cases like RSS.
Looking at the code it was actually a request to SquareSpace, and the poison seemed to be mention of “Twisted”. Best guess they are trying to ban Scrapy which uses Twisted internally.
I’ve also seen CDNs reject requests when the User-Agent string contains “python” or lacks “Mozilla”. I guess lying is just part of HTTP these days.
between polite bots and (browsers and evil bots) <3
Perfect is the enemy of the good. Anyone might come into my house and rob me, but if someone knocks on my door and tells me they’re going to rob me, I’m still not going to let them in just because they asked permission.
If they’ll say it in a certain way, and they will act in a certain way, you will be thankful for them for the opportunity for them to rob you.
Well, not you in particular, but people in general.
indeed, but you don’t check for the Mozilla thing there!
GitHub uses User Agent sniffing. I set my User Agent to “Firefox” (
general.useragent.override) and some features on the site no longer work and GitHub complains that it doesn’t support old browsers.This seems like a fine plan, and other browsers have tried/proposed/supported this in the past. But because this is Google, I have to ask: what other advantages could this give to a browser-building search-providing privacy-invading ad broker?
(Actual question. I’m not coming up with anything myself.)
It would force/encourage browser vendors and web servers to support the Client Hints mechanism, which is a much more sane way of requesting adapted content….and which Google will no doubt embrace and extend.
This is the lens through which I try to view Google’s actions as well.
Is it possible that Google thinks that the loss of UA information will make it harder for their competitors to fingerprint users, but that their penetration makes it less of a problem to them?
I find it curious that the Blink team at Google takes this action in order to prevent various other teams at Google from doing harmful user-agent sniffing to block browsers they don’t like. Google certainly isn’t the only ones, but they’re some of the biggest user-agent sniffing abusers.
FWIW, I think it’s a good step, nobody needs to know I’m on Ubuntu Linux using X11 on an x86_64 CPU running Firefox 74 with Gecko 20100101. At most, the
Firefox/74part is relevant, but even that has limited value.They still want to know that. The mail contains a link to the proposed “user agent client hints” RFC, which splits the user agent into multiple more standardized headers the server has to request, making “user-agent sniffing” more effective.
Oh. That’s sad. I read through a bit of the RFC now, and yeah, I don’t see why corporations wouldn’t just ask for everything and have slightly more reliable fingerprinting while still blocking browsers they don’t like. I don’t see how the proposed replacement isn’t also “an abundant source of compatibility issues … resulting in browsers lying about themselves … and sites (including Google properties) being broken in some browsers for no good reason”.
What possible use case could a website have for knowing whether I’m on ARM or Risc-V or x86 or x86_64 other than fingerprinting? How is it responsible to let the server ask for the exact model of device you’re using?
The spec even contains wording like “To set the Sec-CH-Platform header for a request, given a request (r), user agents MUST: […] Let value be a Structured Header object whose value is the user agent’s platform brand and version”, so there’s not even any space for a browser to offer an anti-fingerprinting setting and still claim to be compliant.
Software download links.
… Okay, I’ve got nothing. At least the W3C has the presence of mind to ask the same question. This is literally “Issue 1” in the spec.
I have a use case for it. I’ve a server which users run on a intranet (typically either just an access point, or a mobile phone hotspot), with web browsers running on random personal tablets/mobile devices. Given that the users are generally not technical, they’d probably be able to identify a connected device as “iPad” versus “Samsung S10” if I can show that in the web app (or at least ask around to figure out whose device it is), but will not be able to do much with e.g an IP address.
Obviously pretty niche. I have more secure solutions planned for this, however I’d like to keep the low barrier to entry that knowing the hardware type from user agent provides in addition to those.
Benchmarking and profiling. If your site performance starts tanking on one kind of processor on phones in the Philippines, you probably want to know that to see what you can do about it.
Additionally, you can build a website with a certain performance budget when you know what your market minimally has. See the Steam Hardware and Software Survey for an example of this in the desktop videogame world.
Finally, if you generally know what kinds of devices your customers are using, you can buy a bunch of those for your QA lab to make sure users are getting good real-world performance.
Amusingly, this date is a static string — it is already frozen for compatibility reasons.
Any site that offers you/administrators a “login history” view benefits from somewhat accurate information. Knowing the CPU type or window system probably doesn’t help much, but knowing it’s Firefox on Ubuntu combined with a location lookup from your IP is certainly a reasonable description to identify if it’s you or someone else using the account.
There are terms I’d certainly like sites to know if I’m using a minority browser or a minority platform, though. Yes, there are downsides because of the risk of fingerprinting, but it’s good to remind sites that people like me exist.
Though the audience here will play the world’s tiniest violin regarding for those affected the technical impact aspect may be of interest.
The version numbering is useful low-hanging-fruit method in the ad-tech industry to catch fraud. A lot of bad actors use either just old browsers[1] or skew browser usage ratios; though of course most ‘fraud’ detection methods are native and just assume anything older than two major releases is fraud and ignore details such as LTS releases.
[1] persuade the user to install a ‘useful’ tool and it sits as a background task burning ads or as a replacement for the users regular browser (never updated)
Seems like a Big Deal. Other browsers following suit: https://twitter.com/dassurma/status/1217082509839781888?s=21
Looking forward to not being served mobile pages on RPi etc just because the UA contains “armv7” or “aarch64”.
My bank does that, my router does that, and some forums (not this one), with the result of a uselessly dumbed down UI.
This seems reasonable. I expect the bigger impact will be on little used, minimally maintained web tools like scrapers.
There is a draft https://user-agent.globalcode.info/ (probably not used by anyone :-) that suggest not using User-Agent string at all or encode it in form of URI (thus machine readable).
User Agent sniffing still used by pages to find out which processor architecture and operating system the user is running when offering downloads. I don’t know any other way of finding out that information.
It looks like User Agent Client Hints would still allow for that upon request.
Operating word here is upon request. I’ve never bothered my wife with it, other then by demonstration, doing it on my own phone/computer, but she now painfully clicks on “Manage Choices” or whatever on the GDPR popus and tells sites she doesn’t want tracking and personalized ads. If the site asks her “are you on an iPhone or on your Mac right now, she might click through. If they ask “do you also have this graphics card and did you last update your laptop a week ago” and stuff, she’ll stop and say “hey I don’t wanna go to this stupid meme site and give them all of that.