Fallen Tech Blog

According to the HTTP/1.1 specification (RFC 2616, section 14.43) ( http://www.w3.org/ ):

14.43 User-Agent

The User-Agent request-header field contains information about the user agent originating the request. This is for statistical purposes, the tracing of protocol violations, and automated recognition of user agents for the sake of tailoring responses to avoid particular user agent limitations. User agents SHOULD include this field with requests. The field can contain multiple product tokens (section 3.8) and comments identifying the agent and any subproducts which form a significant part of the user agent. By convention, the product tokens are listed in order of their significance for identifying the application.

_{(emphasis by me)}

Despite this quite clear definition, a LOT of websites use the User-Agent in an attempt to identify what capabilities a browser has and deliver it to them. Some might say that Microsoft's usage (abuse) of the User-Agent was the cause of the mess of all this we have today when they made Internet Explorer identify itself as "Mozilla" when it clearly was not. When you think about it though, they most likely did this in order to OVERCOME these already incorrect uses of the User-Agent by web "developers" (as this mozilla page from '98 ( http://www-archive.mozilla.org/ ) also implies).

I use the following true, correct and legal User-Agent for my most used browser, uzbl ( http://uzbl.org/ ):

Uzbl/aa8c2e459cd035f13144c21400f8db1c47a15a36 (X11; U; Linux i686; en-US) Webkit/1.1.6 (A browser built upon the highly renowned web standards compliant and secure Webkit rendering engine that is the heart of several open source web browsers, including Uzbl, Midori and Android Webbrowser)

(the "aa8c2e459cd035f13144c21400f8db1c47a15a36" is actually the git commit (source code version if you will))

User-Agent sniffing is SO bad, that my initial User-Agent (shown below) led to too many sites simply not displaying AT ALL (errors, blank page, etc):

Uzbl/aa8c2e459cd035f13144c21400f8db1c47a15a36 (X11; U; Linux i686; en-US; A browser built upon the highly renowned web standards compliant and secure Webkit rendering engine that is the heart of several open source web browsers, including Uzbl, Midori and Android Webbrowser)


So, whenever I come across a page that I notice does a User-Agent sniff (for example they say "we are sorry, you need to be using IE or Firefox to view this page") I contact the webmaster and inform them of their error. This leads to also no change and in most cases no response.

In the event a response IS received after I contact them, it is usually in the form of "We are looking into this" or something similar to the hilarious "If you tell us your User-Agent, we can add it to the list of working User-Agents".

I have not been hugely affected by this and in the event that a site really doesn't work properly due to this, I will simply go elsewhere and "blacklist" that site (after trying to help them of course).

I have noticed very recently however, a HUGE difference in pages I visit. An extreme number of sites have seemingly started User-Agent sniffing all starting around the same time. As time moved on, I noticed it was largely (but not only) wordpress ( http://wordpress.org/ ) blogs. So what was the symptom of these "sniffing fails"? "Mobile" versions of sites. I visit Amazon ( http://amazon.com/ ) and am presented with a rather plain page:

This is quite different compaired to the standard amazon page:

This page is displayed by the EXACT same browser, passing all the same header variables with the acception of the User-Agent, where it was changed to:

Mozilla/5.0 (X11; U; Linux i686; en-au; rv:1.8.1) Gecko/20061010 Firefox/2.0


The effect is the same with all those Wordpress blogs. So why's it happening I thought, until I realised, they are all sniffing for 1 simple word... "Android". When they detect a User-Agent containing "Android", they go into this "Mobile" version.

I emailed Amazon to let them know, but got the standard "1-click" reply that's rather common with Customer Support these days:

Hello,

Thank you for your comments about using Amazon Anywhere with your mobile device. In addition to our large selection, one of the benefits we try very hard to offer our customers is convenience. I'm very sorry for the inconvenience you experienced while shopping at our store with your device.
...

Of course, they didn't actually READ my comments judging by the "... using Amazon Anywhere with your mobile device ..."

The funniest thing about this is, that even without any adblock or noscript plugins in my browser, I actually get a rather lean browsing experience as a result of this

Perhaps the solution would be to build a mobile device called "Mozilla"