Category Archives: Categorizr

HRWD – Hybrid Responsive Web Design

As the amount of web capable devices grows, a one size fits all website will no longer suffice. Responsive web design seems to be the most popular solution to achieving device agnostic web design, but there are issues we still need to resolve when creating a responsive website.

Many great minds have dissected and critiqued responsive web design, and it’s merits as a solution to creating sites for the mobile web. I’ve tried to compile a general list of issues/concerns people have had about responsive web design. As we continue to experiment and find ways to adapt to the growing landscape of the web, it’s important to communicate with one another and review our processes and techniques. What is true today, may change tomorrow.

Issue with Responsive Web Design

Back in in August 2010, Jason Grigsby published an article, CSS Media Query for Mobile is Fool’s Gold.

Some of the main points from the article are:

  • Media queries add more code to create the “mobile” version of the site
  • Media queries ignore the mobile context
  • Media queries do not optimize HTML or JavaScript
Jason also brought up many good points about images:
  • How letting the browser scale images is a bad idea
  • Full size images = unnecessarily large files to download
  • Using media queries to deliver different images doesn’t work

Jason concludes that he believes responsive design still rocks, though it’s important to understand that media queries are not a silver bullet when creating a mobile site.

James Pearce, another expert in the realm of mobile replied to this article,

+1 for context… the most important point. If there’s a media query that will turn an airline’s desktop site (flight booking, credit card promotions, photos of reclining seats, chairman’s statements etc) into its corresponding mobile site (flight time checker, on-line check-in, QR-code boarding pass etc), then I would love to see it.

James later wrote another article about issues related to responsive web design titled, Not a mobile web, merely a 320px-wide one in which he discusses issues with responsive web design.

  • Media queries only affect the styling once the markup and other resources have reached the browser
  • Media queries are not supported by most of the world’s web browsers.

The main issue James gets into is how responsive web design, as it’s currently being done, is not suitable to handle context. It also does little to handle performance optimization. James goes on to say that “…it’s quite likely that they [the mobile user] really deserve different content and services altogether – or, at least, a differently prioritized version of the default desktop experience.

While it’s true, mobile users may want different content or services altogether. Studies have also shown that users love content, no matter what device they are on. To help determine if a separate mobile site is necessary, one should research their user base. Look at your site statistics and analytics. This will give you the information you need to decided if you can offer a differently prioritized version of your desktop site, or if you need to create a custom experience tailored specially for individual user groups, be it a native app or web app. Though if you do decide to go with separate mobile site, I believe the concepts from responsive web design; fluid grids, fluid images, and media queries will still be beneficial to your separate mobile site/web app.

A word on context

Context is a funny thing, and everyone has their opinion on what the mobile web is, and thus, what mobile context is. One of this best articles I’ve come across on this topic is by Jason Rhodes entitled, More Responsive Design, Please.

Stephanie Rieger also has some great thoughts on mobile context. The Trouble With Context and Mobile User Don’t Do That

Almost a year later after Grigsby published his article regarding concerns with media queries, there was another article published on Web Design Shock about issues with responsive web design titled, 11 Reasons Why Responsive Design Isn’t That Cool!

Form these three articles we can see there are some common issues people have with how responsive web design has been done so far.

  • Media queries ignore the mobile context
  • Media queries add more code to create the “mobile” version of the site
  • CSS media query is a bad choice for multiple devices
  • Media queries do not optimize HTML or JavaScript or remove unnecessary code
  • Mobile needs are different from desktop needs
  • Issues with handling images on varying devices

It’s time we to try and solve these issues

I agree with points raised. I believe each new technique that comes out needs to be reviewed and critiqued. This is how we move the web forward. Along with offering critique and reviewing our processes, we need to seek solutions.

The first part of the puzzle is solved by thinking Mobile First

If you have not picked up Luke Wroblewski’s book, Mobile First, I highly recommend it. Going mobile first with our responsive websites allows us to solve some of the issues people have raised about responsive web design.
  • Media queries ignore the mobile context
    By starting a project with a mobile first mindset, we are now focusing on mobile from the beginning, thus acknowledging mobile context and the needs of mobile users. This also allows us new potential for interactions between us and our users, based on new capabilities. It also helps us prepare for the growing market of varying mobile devices.
  • Media queries add more code to create the “mobile” version of the site
    By building our site for mobile devices first, we can progressively enhance our base version of the site by conditionally loading in assets/resources for other more capable devices as opposed to trying to scale down our desktop sites to be more suitable for mobiles. This type of thinking is tricky though, and requires us to really re-examine a lot of our common practices that we have become accustom to. I believe the many benefits of going mobile first out weigh the trouble involved in rethinking the mobile web.
  • CSS media queries are a bad choice for multiple devices
    This issue stems from the idea of starting with a desktop site and using media queries to scale down to a mobile layout. This results in mobile devices that don’t support media queries getting the desktop styles. Though by going mobile first with our media queries, we can easily create styles for devices that don’t understand media queries, and then use media queries to progressively enhance our layout.
Bryan Rieger, puts this nicely,
The absence of a media query is in fact, the first media query“.
-Rethinking the Mobile Web

The second part of the puzzle is Adaptation

When it comes to adaptation, Bryan and Stephanie Rieger are leading the charge on what works and what doesn’t. I would highly reccomend looking through Stephanie’s slides on Pragmatic Responsive Design and Bryan’s slides on Adaptation: Why Responsive Design Actually Begins On The Server
Server side adaptation allows us:
  • Optimize the pay-load from the server to the user, for example, add or remove markup / scripts / styles for different device groups
  • The ability to alternate content appropriate for context
Client side adaptation allows us:
  • Ability to test for capabilities – Feature Detection – Modernizr
  • Ability to modify the layout as screen real estate increases - Media Queries – Responsive Web Design – JavaScript solutions

What tool do we use for server side adaptation?

In Riegers’ slides, they suggest using a device database to handle the server side logic for websites adaptation. Device Atlas, and WURFL are great resources as well as Handset Detection. However, each of these options come with a price, literally. If you go with the cloud version of Device Atlas, it’s $399/year per website or $3,990/year for 50 websites. Their enterprise prices are not listed, I don’t even want to know what those cost. WURFL has recently updated their licensing terms. If you plan to use WURFL on a commercial project, you need to buy a license. Handset Detection seems to be the most reasonably priced solution at $13/month for 100,000 detections. They also have a free option with 10,000 detections per month.

What other options do we have?

Categorizr - A modern device detection script

Categorizr is modern device detection script I wrote to help handle simple device profiling. You can read more about Categorizr here. The concept is that by basing device detection on the idea of mobile first, detection not only becomes easier, it’s also more accurate. If your curious how Categorizr handles user agents, you can view test results here from over 11,000 user agents, most of them are from the WURFL database or check your own device and see what category it falls into.

So now we have all the tools, what is hybrid responsive web design?

Hybrid responsive web design is not a new concept, and there is more than one way someone could create a hybrid responsive website. Though the general idea behind HRWD is that it pairs the concepts of responsive web design with the concepts of server side device detection. It builds on top of the concepts we know are true for responsive web design, and adds concepts from traditional mobile device detection.

After looking through Riegers’ slides from last year’s Breaking Development conference and reading Luke Wroblewski’s post, RESS: Responsive Design + Server Side Components, I really started to get interested in experimenting with this concept. Yiibu.com and The Filament Group’s website are two sites that also really got me thinking in this direction. The first step required an accurate way to handle device detection. The way device detection has traditionally been done was flawed in my mind, as it was not very future friendly, though this is the nature of the beast when you get into user agent sniffing. I wanted to find a solution that would be free for anyone who wanted to use it, and would provide accurate results and this is what led me to develop Categorizr. With Categorizr, you can group devices into specific categories and handle various adaptations from one device group to another on the server. I’ve found though, that mobile devices are where most of the server-side adaptation is needed. Adaptation for tablets and smartTVs can mainly be based on the desktop version and handled client side.

Issues HRWD solves

  • Media queries do not optimize HTML or JavaScript or remove unnecessary code
    This is a problem many people have had with responsive web design. Media queries are a client side technology based for styling, thus they cannot remove unnecessary code, optimize HTML or JavaScript. To do this, we need to do it on the server side. With Categorizr, you can detect what type of device the user is on, and give them only the markup / scripts / styles they need.
  • Mobile needs are different from desktop needs
    To quote James Pearce again, “…it’s quite likely that they [the mobile user] really deserve … a differently prioritized version of the default desktop experience.We can use Categorizr to dectect the type of device the user is on, be it mobile, tablet, desktop, or smartTV. From there we can handle priority changes on the server, whether that means shifting around markup, or loading different content/resources altogher, before sending it down to the user.
  • Issues with handling images on varying devices
    This is something I would not recommend using Categorizr for. While it does a great job of figuring out what type of device your on, it won’t get you specifics. Since there is a lot of variety in mobile screen sizes, I would recommend looking at Sencha.io Src or Adaptive Images. Currently there is discussion going on about creating a new HTML element to handle images for varying devices based on media query logic.
  • More code, more to update
    I honestly don’t understand the reasoning behind this issue. To me, you either don’t do anything and stick with your desktop websites, blissfully ignorant to the changing market, or you adapt. Hybrid responsive web design with Categorizr is actually fairly simple. If you want to support multiple devices, there is going to be more code. Though I’ll explain how we can support a wide array of devices, while still maintaining a code-base that is easy to manage and update.

Issues that still need to be solved.

Hybrid Responsive Web Design does not solve all the issues. There are still many issues we will need to tackle as the web evolves on to more devices. We are all in this together, and for us to move forward, we need to review our processes, offer critique, and help push each other to find solutions. I’m hope you’ll offer your feedback and critique on this approach so we can continue to move the web forward.

One of the biggest issues that still remains relates to content management systems. Our craft is evolving, we need the tools we use to evolve also. Though, this a topic I’ll save to tackle another day.

Mobile First HRWD Development

So, you’ve read this far. Take a break, relax, I know that was a lot to digest. I’m planning on writing up a more detailed walkthrough of mobile first HRWD development and was considering leaving this concept for another post, though while I have your attention, I figured I’d breifly cover some concepts of a mobile first hybrid responsive web design development. Try saying that ten times fast.

First, we’ll be starting our project with a mobile first mindset. As I’ve mentioned, this helps solve a lot of issues when creating a responsive site. There are 4 areas I want to quickly go over; the Sever-Side Component, how to handle HTML, how to handle  CSS, how to handle JavaSript, and how to handle images.

Server-Side Component

In this example I’ll be explaining how to use Categorizr to handle our server side adaptation. Catgorizr is PHP based and fairly easy to use, even if you have no programming knowledge. To alter various parts of your site, be it your CSS, JavaScript, page components, or other markup, all you need is a simple PHP if statement.

<?php if (mobile()) { ?>
<h1>This heading will only show up if the device is categorized as mobile device</h1>
<?php } ?>
<?php if (!desktop()) { ?>
<h1>This heading will only show up if the device is not categorized as desktop device</h1>
<?php } ?>

With this, we can keep all of our markup for each device category together in one file. This makes maintenance of a hybrid responsive website rather easy. This works great if your mobile site will be similar to your desktop site and you just want to re-prioritize your mobile site. However, if your mobile site is going to be quite a bit different than your desktop site, I would suggest using Categorizr Redirect and creating a separate mobile site.

HTML

The HTML you craft will vary from project to project. Sketching and prototyping will be your best friends as you move back and forth between design and development. For tablets and smartTVs, I would suggest using the same HTML markup as your desktop site and letting media queries handle layout changes. If you need to add specific CSS, JavaScript, or other elements solely to your tablet or smartTV version of your page, you can simply write an if statement.

<?php if (tablet()) { ?>
<h1>This heading will only show up if the device is categorized as tablet device</h1>
<?php } ?>

Be sure to be considerate of users who want to change what there device is categorized as. Add a link to your page so if users are on your mobile site, and want to view the desktop version, they can. To do this with Categorizr, all you need to do is create an anchor link and set the href value to a PHP GET variable with view set to what you want it to be. Below is an example of a link you would add to your mobile site to switch users to being categorized as a desktop.

<a href="?view=desktop">View Desktop Site</a>

When a user clicks on this link, it will reload the page, then Categorizr will pick up that the user now wants to be categorized as a desktop device. Categorizr stores this in a session so the user can move from page to page and still be recognized as a desktop device.

CSS

With CSS, we’ll first create styles for basic devices such as feature phones, legacy phones, and other devices that do not understand media queries. In this file I usually add my HTML reset, basic formatting for typography, link styles, and any other basic styles needed for these devices. I’ll usually name this stylesheet base.css though you can name it however you please.

Next, we’ll create a stylesheet for mobile devices. I name this file mobile-styles.css. In this file, we’ll set up our media queries for mobile devices. These will cascade off of the styles set in the base.css file.

Lastly, we’ll create another stylesheet that will handle our layouts for tablets, desktops, and smartTVs. I typically name this file main-styles.css. Again, this stylesheet will cascade off base.css, as well as mobile-styles.css

In development, I like to keep the files separate, though on your production site, I would concatenate them into one file. Then conditionally load them based on what device category comes to your site.

If mobile, load mobile.css.
mobile.css = base.css + mobile-styles.css 

Use this for feature phones / smartphones / and other mobile devices.

If not mobile, load main.css
main.css = base.css + mobile-styles.css + main-styles.css

Use this for tablets / desktops / smartTVs

By concatenating the stylesheets, you’ll only serve 1 CSS file which will help keep your website fast.

JavaScript

The JavaScript you use on your site will vary from site to site, however one file I would encourage you to include on your sites is Modernizr. Categorizr is great at what it does, though it only gets us some of the information we typically want to know. By pairing Categorizr with Modernizr, you can accurately detect device types, and their capabilities.

Just as we did with the stylesheets, I would recommend concatenating your JavaScript where applicapable. The HTML5 Boiler Plate has a good example of file strutures for JS.

Images

As I mentioned earlier, I would recommend using Sencha.io Src or Adaptive Images as a solution to providing images for varying devices.These are the best solutions I’ve seen so far for creating responsive images. Matt Wilcox, the developer behind Adaptive Images and Jason Grigsby are both members of the W3C Responsive Images Community Group. I’m excited to see what solution this community group comes up with as a native way to handle responsive images.

Other Components

There are many other components one could add to a HRWD development, such as appropriately caching files, gzipping files, using HTML5 AppCache, minifying files along with many other techniques. This is a topic I’ll go into more detail another day.

Is this the silver bullet we’ve been waiting for?

Nope. It’s one possible technique you can add to your toolbox. Hybrid responsive web design does not aim to create solutions for the mobile web, the desktop web, the tablet web, the smartTV web. It aims to create solutions for the responsive web. Each project should be evaluated to see which techniques will be the most benefical, not only to your clients, but also to their users.

Thoughts on Windows 8 device detection

Back at the start of February, news came out that Windows Phone 8 will based on the same Windows NT core that Microsoft will use on their Windows 8 desktops. Without having any Windows 8 devices to test, I can only make some general assumptions, though I’m going to guess that Microsoft will now likely include Windows NT in the user agent string on their portfolio of Windows 8 devices be it, mobile phones, tablets, or desktops.

What does this mean for user agent based device detection?

Categorizr works more or less by detecting the OS of the device rather than the browser in most cases. One of the primary checks in Categorizr is to look for Windows NT in the user agent string. This has been a great way to detect Windows based desktop devices, though there are small handful of Windows Phone 6.5 devices that also include Windows NT in the user agent. In these few Windows Phone 6.5 user agents, the word Mobile is also included, so it’s easy to catch these and categorize them correctly as mobile phones.

Since Windows Phone 6.5, Microsoft has included “Windows Phone” in the user agent.

User agent from HTC Radar 4G:
Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; HTC; Radar 4G)

User agent from Samsung Omnia 7
Mozilla/4.0 (compatible; MSIE 7.0; Windows Phone OS 7.0; Trident/3.1; IEMobile/7.0; SAMSUNG; GT-i8700)

User agent from HTC HD2
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; HTC_HD2_T8585; Windows Phone 6.5)

Based on this information, I’m going to guess that even if Microsoft includes a variation of Windows NT in the user agent on their new Windows 8 Phones, as long as they continue with the pattern of including “Windows Phone” in the string too, it should be possible to differentiate between Windows 8 mobile phones, and Windows 8 desktops. Differentiating between Windows 8 tablets and Windows 8 desktops get a bit more interesting.

Where is the line between Windows 8 tablets and Windows 8 desktops?

As Microsoft continues to move towards merging the traditional desktop experience and the tablet experience with Windows 8, I’m curious if they will offer a way to differentiate between the two types of devices via the user agent string, similar to what they’ve done with other Windows tablet PCs?

Windows 8 Tablet

Windows 8 Tablet

Add a keyboard, dock, and the fact that the user will now be stationary and this device falls more inline with what we consider desktops.

Windows 8 tablet with keyboard and dock

As you can see, the line between desktops and tablets is growing smaller, just as the line between mobile phones and tablets is growing smaller with the release of the the Samsung Galaxy Note, the upcoming LG Optimus Vu. What this means is that the ability to determine what category a device falls into based solely in its size will be less accurate. Thus, it’s important to have manufactures add device category keywords to their user agent strings so we can continue to categorize devices correctly to handle various contexts.

Microsoft is no stranger to this concept and is aware that developers like to differentiate between the tablets and desktops. Here’s a small quote from an article on Microsoft’s Dev Center website regarding how to Determing Wheter a PC is a Tablet PC.

In Windows Vista, the user-agent string reported by Internet Explorer includes “Tablet PC 2.0″ if, according to GetSystemMetrics(SM_TABLETPC), the device is a Tablet PC.
In Windows XP Tablet PC Edition 2005, the user-agent string includes Tablet PC 1.7. The user-agent string looks something like the following:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; Tablet PC 2.0)

Metro in the wild

I’ve come across some user agents Paulo Morgado collected from IE10, which runs on Windows 8.

User agent from Internet Explorer 10, running in the Metro UI:
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Win64; x64; Trident/6.0)

User agent from Internet Explorer 10, running in the Classic Desktop UI:
Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; WOW64; Trident/6.0)

I’m not sure if Paulo was on a desktop, or tablet when he got these, though one thing Paulo notes is that Metro Internet Explorer is always a 64bit application on 64bit Windows 8 based on the user agents he’s found so far. I’d much rather check for the words Tablet PC or even Metro in the user agent string when Windows 8 is in Metro mode than Win64 and x64.

It is important to note that the above IE10 user agents are from dev builds, so there is a chance that when Microsoft starts selling Windows 8 devices, that may make changes to the user agents above.

Final Thoughts

On my sites, the tablet version of the site is based on the desktop version and most of the adaptation for tablets happens on the client side. Though it is nice to be able to differentiate between the two device categories on the server if need be. My hopes is that Microsoft continues provide ways for web developers to differentiate between tablets and PCs through keywords in user agent. Though we will just have to wait until Windows 8 is released to know for sure.

All in all, I look forward to seeing what Microsoft does with Windows 8 and the Metro UI.

A new standard for mobile and tablet browser user agents

With the release of Firefox 10.0, many new features having been added. Though one of the features added in Firefox 10.0 that I have not seen get much attention is on the mobile version of Firefox, Fennec. This new feature allows the user the ability to choose their own user agent from a set of predefined user agent strings. This is a huge plus in terms of user experience and general usability, now users on Android tablets can finally have a user agent that differentiates them users on Android phones. You can access this feature by clicking the menu button on your phone while Firefox is open and then clicking on preferences.

Fennec 10.0 menu barOn the Preferences page you’ll now find the option to change your user agent.
Fennec user agent switcherBelow is a picture of the user agent options in Fennec 10.0
Fennec 10.0 user agent optionsPre-Firefox/Fennec 10.0, users on Android tablets that were using Firefox Mobile had the same user agent as users on Android phones.

Firefox Mobile 9.0 user agent on Android phone - Samsung Galaxy S – Epic 4G
Mozilla/5.0 (Android; Linux armv7l; rv:9.0) Gecko/20111208 Firefox/9.0 Fennec/9.0

Firefox Mobile 9.0 user agent on Android Tablet - Asus EEE Pad Transformer
Mozilla/5.0 (Android; Linux armv7l; rv:9.0) Gecko/20111208 Firefox/9.0 Fennec/9.0

Without the ability to change the user agent, this meant that there was no way for device detection scripts that look at the user agent to differentiate users on Android phone from user on an Android tablet if they were using Firefox.

A new standard
This raises the question, should this feature be standard in all mobile browser on Android that can be used on phones and/or tablets, and possible soon Google enabled TVs? The need for browser vendors to include keywords for us to detect like mobile and tablet will only grow as we see more and more web capable devices emerge. Fornately, most of the browsers you can download today on the Android Market, let you choose your user agent, either a mobile one, a tablet one, or a desktop one, some even let you type in your own. I’m glad to see Firefox finally add this to their mobile browser.

There are some other popular mobile browsers that still lack this feature though. Opera Mini is the one the first comes to mind. The user agent of Opera Mini is the same on Android phones as it is on Android tablets.

Opera Mini user agent on Android phone - Samsung Galaxy S – Epic 4G
Opera/9.80 (Android; Opera Mini/6.5.27452/26.1235; U; en) Presto/2.8.119 Version/10.54

Opera Mini user agent on Android Tablet - Asus EEE Pad Transformer
Opera/9.80 (Android; Opera Mini/6.5.27452/26.1283; U; en) Presto/2.8.119 Version/10.54

The lack of ability to change user agents in Opera  Mini is a bit odd, because Opera Mobile does have this feature.

Opera Mobile default user agent on Android phone - Samsung Galaxy S – Epic 4G
Opera/9.80 (Android 2.3.5; Linux; Opera Mobi/ADR-1111021320; U; en) Presto/2.9.201 Version/11.50

Opera Mobile default user agent on Android Tablet - Asus EEE Pad Transformer
Opera/9.80 (Android 3.2.1; Linux; Opera Tablet/ADR-1111101157; U; en) Presto/2.9.201 Version/11.50

Notice the addition of the keyword “tablet” to the user agent string. This is very helpful in recognizing users who are on a tablet using Opera Mobile.

Opera Mobile user agent on Android phone set to desktop mode - Samsung Galaxy S – Epic 4G
Opera/9.80 (X11; Linux zbov; U; en) Presto/2.9.201 Version/11.50

What will happen to browser vendors that do not add this feature?
Bolt Browser, though not as popular as Opera Mini was another browser that did not allow users to change to user agent. When I tested on my phone, Bolt’s user agent was:

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; BOLT/2.800) AppleWebKit/534.6 (KHTML, like Gecko) Version/5.0 Safari/534.6.3

Last month, Bolt Browser was discontinued. According to BitStream,

The free BOLT mobile browsing service has been discontinued. Unfortunately, the economic circumstances prevented us from running a free service going forward. We apologize for any inconvenience and thank you for your loyalty and support.

While I won’t goes as far as saying this is why Bolt Browser was discontinued. The Bolt browser lacked many other features common on modern mobile browsers such as lack of accurate support for media queries and lack of support for correct javascript browser width detection, though I’m sure the ability to not change the user agent was not helping them gain any new users either.

Without browser vendors giving us anything in the user agent string to determine what type of device the user is on, it becomes more difficult to categorize devices correctly. Incorrectly detecting the device a user is on, only leads to a frustrated user. This is why I strongly recommend adding a link to your site allow users to change how their device is categorized.

Looking to the future.
Google recommends that if you want to detect a Android tablet to look for the words “Android” and the absence of the word “Mobile”. While this works for the most part, if you really want your device to be recognized as a tablet, include the word “tablet” in the user agent. The list of web capable devices will only grow. For device detection to continue to work, we need standards we can rely on.

Odd Opera based user agents

Odd Opera based user agents

In testing Categorizr against data from WURFL, I’ve come across some odd Opera based user agents on older Windows phones. What I’ve seen so far in looking at different user agents is that both Opera Mini and Opera Mobile will have either Opera Mini or Opera Mobi in the user agent somewhere, so it’s easy to catch those, yet in the list from WURFL, none of the odd Opera based user agents have these keywords.

The user agents in question all have Windows NT in the user agent string. Initially when I was coding Categorizr, all of these select devices were incorrectly categorized as desktops, due to them having Windows NT in the UA string, which is one of the key terms for detecting Windows based desktops. I’ve since updated Categorizr to recognize these devices as mobiles. However, I’d really like to get some more information on these user agents if anyone is familiar with them. Below is an example of one of the user agents from the HTC HD2:

HTC_HD2_T8585 Opera/9.7 (Windows NT 5.1; U; en) 

I’ve posted a question on Opera’s forum for mobile phones and other devices and hopefully will get some insight on these odd user agents. I’d like to figure out if they belong to Opera Mini or Opera Mobile. In searching around for some more information [1], [2], [3] on one of the main devices in the list, the HTC HD2, it seems like the user agents are from Opera Mobile, so I assume that the rest of the odd Opera based user agents coming from Opera Mobile too, based on how similar they all are. Though I still cannot figure out why Windows NT is included in the ua string, unless it’s purposely there to spoof device detection and have the device recognized as desktop?

If this is the case, then Categorizr detecting them as desktops would be the correct result. I know on my Android phone, in Opera Mobile 11.50, the user can change their user agent from the default mobile one, to a default desktop user agent. This makes me wonder if these odd Opera user agents are coming from Opera Mobile set to desktop mode on these older Windows phones, or if these were actually the default mobile user agent for Opera Mobile on these select devices?

For the time being, these devices are all categorized as mobiles, though if you have information they should be categorized differently, let me know.

TV category added to Categorizr

Catgorizr - Now detects smartTVs, gaming consoles, and other TV based devicesAs technology progresses, we are now seeing smart TVs and other devices that allow us to browse the web on our TVs. Adding a category to detect these devices to Categorizr is something I’ve debated about. Thanks to some feedback from Eric Kim and Toni, I’ve decided to go ahead add add this new category.

As I set out to gather information on how to detect these devices based on their user agent string, I found that there was not a lot of information currently available on the topic.

There are 3 main issues I ran into when looking for data on web capable TV detection.

  • Where do I get a list of all the web capable TVs / devices that should be recognized as TVs
  • Once I have this information, how to I determine if one can actually browse the internet on this device
  • What’s the most accurate way to detect the user agent string of TV based devices that can browse the web.
The conclusion I have come to is, that for smartTV detection to be as accurate as possible, I’ll need to get feedback from people knowledgeable with these devices.
If you have any information you can share about these devices, please let me know in the comments below.

Where do I  get a list of of web capable TVs / devices that should be recognized as TVs
I’ll admit, I’m more in tune with new Android phones and tablets and Apple devices, so my knowledge of web capable TVs is limited. Wikipedia seems to have the most information that I could find on which devices I need to look at to build an accurate check for TVs in Categorizr. Based on what I found on Wikipedia, I put together a list of known smartTV devices, TV Set top boxs, and TV based gaming consoles.


  • LG ‘s ”SMART TV” branded NetCast Entertainment Access devices
  • Panasonic Viera Cast (by Panasonic), new platform is Viera Connect
  • Philips NetTV
  • Samsung Smart TVs
  • Sharp Aquos Net+
  • Toshiba Places
  • AppleTV
  • GoogleTV
  • Roku
  • Boxee
  • Playstation 3
  • Nintendo Wii
  • Xbox 360

If you know of any other devices, let me know in the comments below.

How to I determine if one can actually browse the internet on these devices

Now that I have a list of devices that seem to be good candidates to be categorized as TVs I need to figure out if users can actually browse the web on these device or if they just allow the user access to internet based apps like Netflixs, Hulu, Youtube, etc. For this, the only way I know how to get this information is to get feedback from actual users. I’ve searched the web to get some information on a few of the devices, though feedback from people that actually have these devices on if one can actually browse the web on them is more valuable in my eyes.

What I’ve been able to determine so far is that the following devices can browse the web:

  • Philips Net TVs
  • Samsung Smart TVs
  • Android powered Google TV
  • Playstation 3
  • Nintendo Wii
  • Boxee

These devices only have access to internet based apps and cannot browse the web:

  • Sharp Aquos Net+
  • Xbox 360
  • AppleTV
  • Roku

I’ve added keywords to catch these user agents in case there are any updates to these devices allow the user to browse the web on them in the future. While you can hack some of the above devices to browse the net, I have no information on what the user agents are for devices once they have been hacked.

Unknown if users can browse the web with these or not:

  • LG ‘s ”SMART TV” branded NetCast Entertainment Access devices
  • Toshiba Places
  • Panasonic Viera Cast (by Panasonic), new platform is Viera Connect

What about desktop devices hooked up to a tv?

Unfornately, I don’t see a way to detect this case. It would be similar to switching your monitor that your PC is hooked up to. Your still on the same device, you’re just displaying the video on a different screen. I did come across Kylo. A browser built specifically for desktop/laptop devices that are hooked up to tvs. If you are using Kylo when you desktop/laptop computer is hooked up to your TV, Categorizr will recognize the device as a TV.

What’s the most accurate way to detect the user agent string of web capable TV based devices.

With tablet devices, detection is fairly easy. For the most part you can look for the word tablet, or follow Google’s recommendation of looking for the word Android and the absence of the word Mobile. Though with TVs, there is no one keyword to catch the majority web capable TVs. Some have smartTV in the ua string, while others don’t, some use CE-HTML while others don’t. Since there is no common keyword, the best I have come up with is to look for a set of multiple common keywords amoung the devices.

Checking for the following keywords seems provides the most accurate results in detecting smartTVs, and TV based devices, with out causing false positives for other categories.

GoogleTV, SmartTV, NetCast, NETTV, AppleTV, boxee, Roku, DLNADOC, CE-HTML, Kylo, Playstation 3, Wii, and Xbox 

If you know of any clashes between these keywords and other user agent strings that should not be categorized as TVs let me know. So far in testing these against the user agents I’ve collected, results seem great. Though web capable TV’s are fairly new and seem to be in a growing market. I’m sure this list of keywords will need to be audited and updated from time to time.

What to do now

Now that we can categorize TVs, we can optimize specifically for them. TV’s provide a unique user expediences that offer different capabilities and constraints from other devices. I believe there is an untapped market for developers who understand how to create web experiences on these devices.

Google has put together some great documentation on how to develop websites for TVs.

One key point I would stress is to optimize your desktop site for TV’s, rather build a new site when developing your site for use on TVs.

Below is more information on this topic from Google’s guide to developing websites for TVs.

Build the TV version on your existing site/domain

In most cases, it’s better to build the TV optimized version of your content on your existing site (i.e. build on example.com rather than buy a new domain or TLD)

  • Simplifies maintenance for administrators
  • Keeps the brand consistent for users (they only need to remember one URL)
  • Strengthens indexing signals, such as PageRank, to the main site (rather than diluting signals between a TV and desktop version)

Lastly, TV based user agents are a moving target, like all user agents. As the smartTV market grows, Categorizr will need to be checked to make sure it’s handling new devices correctly. If you know of any issues with this device detection category or have any information you’d like to add, let me know in the comments below.

Known Issues with Categorizr

No device detection script will always be 100% accurate. With new devices coming out everyday and some sub-par user agents from days ago, the truth is, there will always be some inaccuracy in device detection. It’s important to audit your device detection script from time to time to see how accurate your results are. I’ve cross-checked Categorizr with data from WURFL along with checking some user agents from useragentstring.com, http://www.zytrax.com/tech/web/mobile_ids.html, and a few other places across the web. Though it’s impossible to collect every user agent to test, I feel like I’ve got a decent amount collected and I am pleased with the results I’ve gotten from cross-checking Categorizr. Though I did find some oddities. Below are issues I’ve discovered with Categorizr. While it looks like there may be a lot of issues judging by the length of this post, most of this post is about the issues I came across in testing Categorizr and how they have been solved.

Issues with Desktop Detection.

There a select few older Windows Mobile devices that when using Opera Mobile, were incorrectly categorized as being desktops. This issue has now been resolved and these devices are now detected as mobiles. You can read more on the issue here: Odd Opera based user agents.

Issues with TV Detection.

A category for TVs has been added to Categorizr. Smart TVs and TV based gaming consoles like the Nintendo Wii and Playstation 3 are now categorized as TVs. A few of the SmartTV’s categorized as a TV include Google’s Android powered TVs, Philips Net TVs, and Samsungs Smart TVs. Results so far for detecting smart TVs have been great, though as this is still fairly a new market, there may be some smart TVs or other TV based devices that slip through the device detection.

If you know of devices that are not categorized correctly, let me know and I’ll see what I can do to update Categorizr.

Issues with Tablet Detection.

Tablets are fairly easy to check for because most have them have the word tablet in some shape or form somewhere in the user agent string. I cannot stress how nice this is. I’d like to see this become a more standardized practice for tablet manufactures and browser vendors.

While checking for the word tablet in the user agent works for most tablets, Android tablets do not include the word tablet in their stock user agent. Google recommends to check for the word Android and the absence of the word mobile in the user agent when checking to see if a Android device is a tablet. This actually works rather well for the most part, but there are some Android Tablets, mainly 7 inch, pre Android 3.0 devices that fall through the cracks with this method.

The Samsung Galaxy Tab 7, Dell Streak 7, Zenithink zt180, ViewSonic ViewPad7, Huawei IDEOS S7, Coby MID7015, Archos A101IT, Archos A70BHT, Nextbook Next2, Pandigital Nova, and Pandigital SuperNova are all pre Android 3.0 tablets that do not get detected by Google’s recommended way to detect tablets.

Instead, to catch these pre Android 3.0 tablets, Categorizr checks for the actual device ID. Thankfully, Google’s recommendation for detecting tablet devices works on all Andriod 3.0+ devices so far, so the list of tablets that Categorizr needs to actually check for the device ID should not be growing, though if you do come across a Android tablet you think is being incorrectly categorized as a mobile, let me know in the comments below.

In checking against WURFL, I noticed some oddities as far as mobile devices being categorized as tablets. The HTC Magic, Sony Ericsson X10i, HTC X06HT, Samsung SC-02B, and the Vodafone 945 were categorized as tablets. In each case, the user agent has the word Android and not Mobile in it. I believe these are cases where the user agent has been modified from the stock user agent.

For example, the HTC Magic user agent recognized as a tablet was:
Mozilla/5.0 (Linux; U; Android 1.5; sv-se; HTC Magic Build/CUPCAKE) AppleWebKit/528.5+ (KHTML, like Gecko)

Here is another user agent for the HTC Magic, in this case, Categorizr correctly recognizes it as a mobile instead of a tablet.
Mozilla/5.0 (Linux; U; Android 1.5; da-dk; HTC Magic Build/CUPCAKE) AppleWebKit/528.5+ (KHTML, like Gecko) Version/3.1.2 Mobile Safari/525.20.1

To catch these oddities, Categorizr manually checks for the device ID on these select few mobiles to make sure they get categorized as mobile, not tablet.

Firefox Mobile/Fennec pre 10.0
Another cause of false positives happens with users who are on Firefox Mobile on tablets, prior to Firefox Mobile 10.0. Firefox just recently launched Firefox Mobile 10.0, which has a new feature that allows users to change their user agent. Before 10.0, the user agent on Android tablets was no different than on Android mobile phones.

Firefox Mobile 8.0 and 9.0 user agent on Android phone
Mozilla/5.0 (Android; Linux armv7l; rv:8.0) Gecko/20111104 Firefox/8.0 Fennec/8.0
Mozilla/5.0 (Android; Linux armv7l; rv:9.0) Gecko/20111208 Firefox/9.0 Fennec/9.0

Firefox Mobile 8.0 and 9.0 on Android Tablet
Mozilla/5.0 (Android; Linux armv7l; rv:8.0) Gecko/20111104 Firefox/8.0 Fennec/8.0
Mozilla/5.0 (Android; Linux armv7l; rv:9.0) Gecko/20111208 Firefox/9.0 Fennec/9.0

The ability to choose your user agent in Firefox Mobile 10.0 is a huge plus for users. Categorizr correctly categorizes all of the new possible user agents introduced in Firefox Mobile 10.0.

Opera Mini on Android Tablets
This false positive is similar to pre Firefox 10.0. As you can see below, there is nothing to differentiate Opera Mini on an Android tablet from an Android mobile phone.

Opera Mini on Android Tablet
Opera/9.80 (Android; Opera Mini/6.5.27452/26.1283; U; en) Presto/2.8.119 Version/10.54

Opera Mini on Android Phone
Opera/9.80 (Android; Opera Mini/6.5.27452/26.1235; U; en) Presto/2.8.119 Version/10.54

Opera Mobile has a feature where users can change their user agent. It would be nice to see this feature added to Opera Mini. My thoughts are if a user wants to use Opera Mini on their Android tablet, they might be trying to keep their data usage low. In this case, categorizing Android tablets with Opera Mini as mobiles would help in keeping their data usage low. I’ll admit, I’d like to differentiate between the two, but with nothing to differentiate the two in the user agent strings, it will take Opera to fix this false positive.

Whats an iProd?

Another oddity I found in looking at user agents from WURFL, was this little gem.
Mozilla/5.0 (iProd; U; CPU iPhone OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko)

Notice the iProd. This really isn’t an issue, just an oddiety I noticed when comparing against user agents in WURFL, so thats why the word iProd is in Categorizr.

Other edge cases/false positives.

The Nokia N800 causes some false positives if you are not using the stock browser. Of the 102 user agents I checked for the N800 over at Handset Detection, all of them were categorized as mobile. While the Nokia N800 may have been marketed as an “internet tablet” when it was announced in 2007, to me, the device falls more inline with other mobile devices Categorizr recognizes, so having the N800 categorized as a mobile is okay with me.

However, I did noticed some false positives when I came across 4 unique user agents from Zytrax which seem to be from thrid-party browsers on the N800 rather then the stock browser. These user agents were categorized as desktops.

  • Mozilla/5.0 (X11; U; Linux armv6l; en-us) AppleWebKit/528.5+ (KHTML, like Gecko, Safari/528.5+) tear -  Tear 0.3 (Beta) on Nokia N800 under Maemo.
  • Mozilla/5.0 (X11; U; Linux armv6l; en-US; rv: 1.9.1a2pre) Gecko/20080813221937 Prism/0.9.1 -  Prism on a Nokia N800 tablet under OS2008
  • Links (2.1pre31; Linux 2.6.21-omap1 armv6l; x) - Links 2.1 preview 31 on a Nokia N800 tablet under OS2008.
  • Mozilla/5.0 (X11; U; Linux armv6l; en-us) AppleWebKit/528.5+ (KHTML, like Gecko, Safari/528.5+) midori -  Midori on Nokia n800 tablet device.

I must admit, I’m not too familar with third-party Nokia mobile browsers, so if anyone has more information about these, I’d be glad to hear about it. The issue I have is that I am unsure if I can set up keywords to catch, Tear, Prism, Links, and Midori and always assume that if those are in the user agent, then it’s a mobile device.

In looking more into Tear, it does look like it would be safe to set up a check to catch user agents with Tear in it as mobile. If you know otherwise, let me know in the comments below.

As far as the N800 user agents with Prism and Links in them, I couldn’t find much information on them. So for the time being, I’ve set up a check in Categorizr to recognize user agents that make it through the tablet detection, but have Prism or Links in the user agent as mobile rather then desktop.

The oddest N800 is the last one, from Midori. It seems the Midori web browser can be used on desktops as-well as mobiles. So setting up a check to recognize user agents with Midori as mobile would create false positives for desktop users that are using Midori.

Now these are small edge cases in my opinion, and are why I consider it important to add a link on your site to allow the user to choose their view. If a user is on a mobile device and is incorrectly categorized as a desktop or vice-versa, it helps increase the user experience tremendously to allow them to correct this false categorization, by allowing them the option to decide which version of the site they want to view.

Issues with search engine bots, spiders, and crawlers.

One issue I had not thought of originally when working on Categorizr, was how to handle bots, spiders, and crawlers. In going with the mobile first approach when detecting devices, it does help reduce the amount of devices one has to check for. Though there are plenty of bots, spiders, and crawlers that need to be categorized as desktop. Lucklily, most can be caught with a few simple keywords. I’ve added a check in Categorizr that recognizes all major search engine bots, spiders, and crawler user agents as desktops, along with a majority of other miscellaneous bots, spiders and crawlers I’ve collected from useragentstring.com. You can view the results of how Categorizr handles search engine bots, spiders and crawlers. Most all of bots, spiders, and crawlers are recognized as desktop, except for a few miscellaneous ones which are categorized as mobile.

In looking more into the user agents which are not categorized as desktop, most of them are either no longer in use/no longer supported, are content/link scrappers or other non-preferred crawlers, or just not popular enough to merit a check to recognize them as desktop bots, spiders, or crawlers.

If you a developer/user of one of these alternative search engines that are recognized as mobile and feel that your user agent should be grouped with other desktop bots, spiders, and crawlers, let me know. I believe Categorizr handles bots, spiders, and crawlers rather well, though I’m open to feedback on this issue.

Conclusion

To wrap up, there are some user agents that are going to be falsely identified.
Without keeping a list of every device user agent, this is inevitable. Even with tools like WURFL and Device Atlas, there are going to be false positives. This is why it’s important to have an option for users to choose their view of the site, be it the mobile version, desktop version, or whichever version you decide.

Categorizr is fairly new and I’m sure there may be more false positives that come up as this technique is used more and more. Though I feel confident in the results I’ve seen in testing Categorizr so far. If you run into any other issues with Categorizr or come across a user agent that you feel is categorized incorrectly, please let me now in the comments below.

Categorizr – A modern device detection script

Categorizr - A modern device detection script

Categorizr is a modern device detection script that helps categorize different web capable devices into 4 categories, desktop, tv, tablet, or mobile. It take’s traditional device detection and flips it on it’s end, assuming devices are mobile first, unless otherwise detected as tablet or desktop. Categorizr was created by Brett Jankord and is based on an my earlier device detection script.

You can download Categorizr from github here: https://github.com/bjankord/Categorizr

Developers can include Categorizr on their site to detect if a device is a desktop, tv, tablet, or mobile phone. With these 4 categories, you can choose what you want to do when a user accesses your website from varying devices. Originally the script was written with the idea in mind of using it as a solution to creating a hybrid responsive web site. Though you could also use it for traditional mobile redirects.

What makes Categorizr different?

Instead of assuming the device is a desktop, and detecting mobile and tablet device user agents, Categorizr is a mobile first based device detection. It assumes the device is mobile and sets up checks to see if it’s a desktop or tablet. Desktops are fairly easy to detect, the user agents are known, and are not changing anytime soon. Tablets are also pretty easy to detect.

By assuming devices are mobile from the beginning, Categorizr aims to be more future friendly. When new phones come out, you don’t need to worry if their new user agent is in your device detection script since devices are assumed mobile from the start.

Categorizr test results.

I’ve set up a page where you can view a cross-check of Categorizr against over 10,500+ user agents. The cross-check includes data from WURFL along with some user agents from http://goo.gl/sRA2X, http://goo.gl/p0dhR, and a few other places across the web. I’ve also cross-checked Categorizr on all the devices I could get my hands on at Best Buy, Verizon, AT&T, T-Mobile, and Sprint.

You can test your own devices and other user agents against Categorizr here. If you come across any devices you think are categorized incorrectly, let me know in the comments below, I’ll see what I can do to catch them in the script.

So far in testing Categorizr, I’ve seen really positive results, though with any kind of device detection where you’re checking user agents, there is going to be some chance of error. In the 10,500+ user agents tested so far, I did come across some issues. You can read about them here. Known issues with Categorizr. I would strongly encourage developers to add a link to let the their user decide what view they want, be it the mobile version, desktop version, or whatever other version you decide.This will help ensure if initial detection is incorrect, the user can still get to an optimized version suited for them.

Updated – 2.6.12
A category for TVs has been added to Categorizr. Smart TVs and TV based gaming consoles like the Nintendo Wii and Playstation 3 are now categorized as TVs. SmartTV’s categorized as a TV include Google’s Android powered TVs, Philips Net TVs, and Samsungs Smart TVs. I have been unable to get any information about the user agents from LG’s smart TVs so I’m unsure what category they would fall into. http://goo.gl/ZSDUc

If you know of other smart TVs that are not categorized correctly, let me know and I’ll see what I can do to update Categorizr.

Updated – 2.29.12
Categorizr
is ready to download.

You can download Categorizr from github here: https://github.com/bjankord/Categorizr

It comes in two flavors, Categorizr and Categorizr-Redirect.
If you looking to create a hybrid responsive wed design, go with Categorizr. If your looking for more traditional redirect, go with Categorizr Redirect.

A WordPress plugin is in the works.

If your using Cateogrizr on a bigger site, let me know. I’d love to know how/where below are using it. Comments, concerns, questions, let me know in the comments below.

Device detection, flipped on it’s end

A word on Device Detection

Device detection has been a one of the main solutions for many mobile websites. Usually it works like so; check the user agent string to see if the device is a mobile device, if so, do mobile stuff, if not keep doing desktop stuff. One of the issues with this is the possibility for new mobile phones/web enabled devices to fall through the cracks of the device detection script. If for whatever reason, your new phone/tablet isn’t in the device database or user agent filter check, you get the desktop experience, for better or for worse.
*Due to poorly engineered mobile websites and highly capable smartphone browsers, some users actually prefer viewing the desktop version of a site as opposed to the mobile on their phone.

However, if we assume you are on a mobile phone from the beginning, things get a lot easier. We only have to detect if you are not on a mobile phone, by checking against well known PC user agent strings. Below is the script I’ve put together so far. Much thanks to Rob Mason for the idea for this code.

/**
* PHP Device Detection Script Version 1.0
* Copyright © 2011
* Brett Jankord - http://www.brettjankord.com/2011/11/11/device-detection-flipped-on-its-head/
* Big thanks to Rob Manson and http://mob-labs.com for their work on
* the Not-Device Detection strategy:
* http://smartmobtoolkit.wordpress.com/2009/01/26/not-device-detection-javascript-perl-and-php-code/
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Lesser General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Lesser General Public License for more details.
* You should have received a copy of the GNU General Public License
* and GNU Lesser General Public License
* along with this program. If not, see http://www.gnu.org/licenses/.
**/

<?php
$ua = $_SERVER['HTTP_USER_AGENT'];
$screen = $_GET["screen"];

if ($screen == "pc") {
    /* This assumes you have set screen=pc in your query string */
    $user = "pc";
} else if ($screen == "tablet") {
    /* This assumes you have set screen=tablet in your query string */
    $user = "tablet";
} else if ($screen == "mobile") {
    /* This assumes you have set screen=mobile in your query string */
    $user = "mobile";
} else if (
    (preg_match('/iPad/i', $ua)) || (preg_match('/tablet/i', $ua))
)
{
    /* This user agent is  on a tablet */
    $user = "tablet";
} else if (
    (preg_match('/Linux/i', $ua)) && (preg_match('/Android/i', $ua)) && (!preg_match('/mobile/i', $ua))
)
{
    /* This user agent is  on a tablet */
    $user = "tablet";
} else if (
    (preg_match('/Win/i', $ua)) && (!preg_match('/Windows.Phone/i', $ua))  && (!preg_match('/Mobile/i', $ua))
)
{
    /* This user agent is Windows PC */
    $user = "pc";
}  else if (
    (preg_match('/Mac.OS/i', $ua)) && (!preg_match('/.*iP(hone|od)(;|\s).*$/i', $ua))
)
{
    /* This user agent is  Mac PC */
    $user = "pc";
} else if (
    (preg_match('/Linux/i', $ua)) && (!preg_match('/Android/i', $ua)) && (!preg_match('/mobile/i', $ua))
)
{
    /* This user agent is  a Linux PC */
    $user = "pc";
} else if (
    (preg_match('/Solaris/i', $ua))
)
{
    /* This user agent is  a Solaris PC */
    $user = "pc";
} else if (
    (preg_match('/BSD/i', $ua))
)
{
    /* This user agent is  a BSD PC */
    $user = "pc";
} else {
    /* Otherwise assume it's a mobile */
    $user = "mobile";
}
?>

The benefits of this method are that there are not a whole bunch of devices we need to check against. Also, there won’t be a whole boatload of new desktop devices soon, so this solution seems fairly stable. This solution covers my user-base extremely well, yet if you need more information on the devices coming to your website, such as device capabilities, I would suggest checking out WURFL.

EDIT: This script has now evolved into Categorizr. Please read here for more info.