User Experience Research

How iOS 7 stacks up:
Smartphone OS User Experience Shootout

Major Points

User experience of tablets and smartphones is one of the most important aspects of the overall perceived value of the device.

The arrival of iOS 7 will have a profound impact on the market for  connected digital devices

Pfeiffer Consulting has conducted in-depth user experience benchmarks of the 5 major mobile operating systems currently in use: iOS 7, iOS 6, Android, Windows Phone 8 and Blackberry 10 


Pfeiffer Consulting


Download the complete benchmark report (release 1.1):
How iOS 7 Stacks Up: Smartphone OS User Experience Shootout (PDF)

Related Research:
Google vs. Apple vs. Amazon: The 2013 App Store Maturity Shootout (PDF)

(More information on this report is here)

 

Why is it that the arrival of iOS 7 is necessarily a momentous event for the smartphone market? Simple: Unlike any other operating system out there, it will be in the hands of millions or tens of millions of users within a few days after its launch. 

And that will make it a force to be reckoned with.

The question is, of course:
How good is it really?

How iOS 7 stacks up: Smartphone OS User Experience Shootout

Introducing: The 2013 Mobile OS User Experience Benchmarks

Whether we like it or not, smartphones have become a software game. Take any recent top-of-the-line smartphone, and you are likely to get a well-designed, fast, pleasant to use bit of hardware: fluid operation, responsive interaction, fast graphics. The difference of user experience, therefore, stems almost exclusively from the operating system, the user interface design, the application integration, the overall coherence.

This research project compares the five major mobile operating systems in use today: iOS 7, iOS 6, Android, Windows Phone 8, and Blackberry 10, and rates them in terms of user experience. 

We do not look at features, we do not compare cutting-edge options and gadgets, we only look at aspects that have a direct impact on the day-to-day user experience of an average, non-technical user.

The aspects we have surveyed and rated are the following:
cognitive load, efficiency, customization, as  well as user experience friction. Based on the results from these benchmarks we have then established an overall Mobile Operating System User Experience Index presented at the end of this document.

The benchmarks are based on the Pfeiffer Consulting Methodology for User Experience Quantification. You can find detailed information on the subject by following this link.

Key Results

 mobileoshomescreens2013

The combined results of the four different benchmarks and evaluations give iOS 7 a clear advantage in terms of of overall user experience, taking into account the context defined for these benchmarks: 
day-to-day user experience of an average, non-technical user. For details and methodology please refer to the complete report.

Overall Mobile Operating System User Experience Index

  

Individual Benchmark Results - Cognitive Load

Cognitive Load Index of Major Mobile Operating Systems

Cognitive load—the sum of elements you need to get familiar with in order to use a device spontaneously and intuitively—is one of the key aspects of user experience for a non-technical user. For this benchmark we counted the number of apps/widgets as well as other icons and user interface elements a default installation of the operating system contains. Differences between operating systems are significant. For a detailed  comparison of cognitive load elements of mobile operating systems please check the full report.

iOS7 Cognitive Load Benchmarks

 Individual Benchmark Results - Efficiency and Integration

Efficiency Rating of Major Mobile Operating Systems

Smartphone user experience depends in large part on the easy and efficient integration of different key features and services.
For this benchmark we analyzed access to key settings, integration with notifications, multitasking, and camera access, among others. While there is a core set of integration functionality all mobile operating systems share, there are significant and surprising differences between the five major players in the market.

iOS7 Efficiency Benchmarks

Individual Benchmark Results - Customization

Customization Rating of Major Mobile Operating Systems

Consumer-level customization is one of the key user experience aspects of connected digital devices. Nevertheless, the current crop of mobile operating systems differs quite significantly in terms of customization, ranging from the almost dizzying granularity of Android’s options, to the starkness of Windows Phone 8, which allows hardly any user-level customization.

iOS7 Customization Benchmarks

Individual Benchmark Results - User Experience Friction (UXF)

User Experience Friction (UXF) of Major Mobile Operating Systems

UXF is the bad stuff, the aspects of a device that can annoy you in a niggling way, or, in extreme cases, drive you crazy. Basically, UXF occurs whenever a device does not do what you expect it to do - or lacks a key feature that should be available. For this survey we took only the most obvious UXF factors into account, and rated them. For a detailed account of the UXF factors, as well as the rating method, see the full report.

iOS7 User Experience Friction Benchmarks

 

Andreas Pfeiffer,  October 2013

  • http://pfeifferconsulting.com Andreas Pfeiffer

    So?

  • Suneet Sood

    In order to better understand the results of the survey, I looked at the methodology.
    Of course, the ideal way of measuring user experience is as follows: give an Apple smartphone to 100 persons, an Android to another 100, etc. Ensure that all volunteers are of approximately similar backgrounds, e.g. educational, and that they are all smartphone-naive. To get some crude figures, ask them to rate the smartphones on a scale of 1-10 according to previously defined criteria. To get more objective information, call them back and ask them to perform some defined tasks, and analyze the data.
    I note that your method is strikingly different. You haven’t given the smartphones to a single user. Instead, you have identified certain criteria of usability. (Your criteria are logically conceived, I agree, but only represent opinions of the research team.) The criteria have not been collected from published literature. For example, you say “User experience is dependent on tangible as well as intangible aspects”. This sounds logical, and may be true, but I ask, where is the evidence that it is actually true? Has some research shown that tangible and intabgible aspects influence user experience? If so, which one has greater influence? You claim that the number of megapixels of the built-in camera will have little or no noticeable repercussions on the overall user experience, nor on the quality of the images the built-in camera produces, for that matter. Again, this may be true, but are you sure? Has some research shown that the megapixels have little repercussions on the user experience? Similarly, you have presumed, without providing published evidence, that the Kindle’s on-off switch is a cause for user friction. Is there proof that smartphone users are “average”? Is there proof that the majority of users purchase tablets to “provide access to the vast library of apps”? On what published research have you provided the weightages of 20% for hardware nd 50% for device universe?
    In my opinion your approach to the problem is appropriate: you correctly define characteristics to be measured and then go ahead and do the measurements. Unfortunately, your research design lacks validity: it does not measure what it claims to measure.

    • http://pfeifferconsulting.com Andreas Pfeiffer

      Thank you for your thoughtful comments. I understand your concerns, but I think there is a basic misunderstanding (as from most people who have criticised the research) as to what it was we had tried to achieve. We weren’t out to quanitify the reaction of users; what we were trying to do was to quantify empirically surveyable aspects. And that is because I really believe we need to extend our ways of discussing technology – not only collect more market research data.

      So here is what we set out to do: to empirically describe differences between different operating systems and devices, and the to find a way of quantifying it. Cognitive load is a good example: you cannot argue that it exists, and you cannot argue that our way of measuring it is unsound.
      Lets talk about user experience friction: again, we took an empirical approach: what are the friction elements that occur? Can we observe and describe them, and if we cannot measure them, can we rate them in a coherent way? And that’s what we have done. The methodology of quantification is not ideal, but it is in my view significantly more useful in quantifying a particular friction point than having 52 users say “I hate that”

      Yes, it would be nice to be able collect data from users, but if you have any experience with market research (we have done a lot of it over the years) you will also be aware of the immense difficulties and pitfalls of that approach. Can you compare results from a european user to those from Asia or Latin America? Can you compare reactions from women to those of men? Of different age groups? Of people of different socio-cultural backgrounds? Or do you have to conduct a research that has sufficient sample size in all these categories? (which is of course totally impossible)

      I see our research as a first step in what I consider a very important direction. It is not perfect, and it does not pretend to be. It is intended to get a discussion going, and to introduce new concepts into our ways of analyzing technology. And I do think notions like cognitive load and user experience friction push the envelope of what we can talk about.

      But I would be happy to extend and grow our conceptual framework beyond what it is today. As I said, it’s a first step.

    • Andreas Pfeiffer

      Thank you for your thoughtful comments. I understand your concerns, but I think there is a basic misunderstanding (as from most people who have criticised the research) as to what it was we had tried to achieve. We weren’t out to quanitify the reaction of users; what we were trying to do was to quantify empirically surveyable aspects. And that is because I really believe we need to extend our ways of discussing technology – not only collect more market research data.

      So here is what we set out to do: to empirically describe differences between different operating systems and devices, and the to find a way of quantifying it. Cognitive load is a good example: you cannot argue that it exists, and you cannot argue that our way of measuring it is unsound.
      Lets talk about user experience friction: again, we took an empirical approach: what are the friction elements that occur? Can we observe and describe them, and if we cannot measure them, can we rate them in a coherent way? And that’s what we have done. The methodology of quantification is not ideal, but it is in my view significantly more useful in quantifying a particular friction point than having 52 users say “I hate that”

      Yes, it would be nice to be able collect data from users, but if you have any experience with market research (we have done a lot of it over the years) you will also be aware of the immense difficulties and pitfalls of that approach. Can you compare results from a european user to those from Asia or Latin America? Can you compare reactions from women to those of men? Of different age groups? Of people of different socio-cultural backgrounds? Or do you have to conduct a research that has sufficient sample size in all these categories? (which is of course totally impossible)

      I see our research as a first step in what I consider a very important direction. It is not perfect, and it does not pretend to be. It is intended to get a discussion going, and to introduce new concepts into our ways of analyzing technology. And I do think notions like cognitive load and user experience friction push the envelope of what we can talk about.

      But I would be happy to extend and grow our conceptual framework beyond what it is today. As I said, it’s a first step.

      • Suneet Sood

        I have no experience with market research. My field is medicine, in which I do have research experience. Then again, the principles of research should remain the same. First, research should be new, true, important, feasible. Your project largely fulfills these criteria. Next, the methods should be clear and reproducible; you are on target so far. Further, I think your research design is based purely on logic, and does not seem to be twisted to try to favor any one OS. That’s great.
        Now research goes in two directions. The conventional way in medicine is from clinical experience to science. Clinical experience shows that aspirin relieves headache, so researchers go back to the lab and study the effect of acetylsalicylic acid on the pain pathway. In your field you might find that users prefer Blackberry’s mail features to Apple’s, then go back to the lab to dissect Blackberry’s mail software properties.
        The other direction is “bench to bedside”. Researchers could study the surface of a cell, discover that a nut-shaped receptor causes cell division, and hypothesize that a chemical molecule shaped like a bolt will fit this receptor and cause cell division. They would then validate their hypothesis by testing bolt-shaped molecules on cells, to confirm that multiplication actually occurs.

        It seems to me that your project resembles the latter type of research. It starts with completely logical hypotheses, but it needs validation in the field.
        I agree with you, of course, that your research is a first step in an important direction. Near-perfection will come with time.

        • Andreas Pfeiffer

          Your comments made me realize that one of the most important aspects of what we are trying to achieve is to expand the concepts and notions that we use when we discuss digital devices. The reason is that it is blatant for me that the way we describe these devices is woefully removed from the actual perceived value and user experience. So we wind up with devices that look similar, and have similar specifications, but provide completely different user experience – yet we don’t know how to discuss and quantify this.

          Several years ago I worked on a market research project about total cost of ownership of Mac and Windows computers in publishing. During the research interviews, a significant number of respondents using both platforms stated strongly that the mac was more fluid. It was something they clearly perceived, but we thought could not be quantified. And the the notion of “user experience friction” occurred to me (the term did not exits at the time, but has been adopted widely in the User Interface community since. ANd we also found ways of measuring it (you can read more here should you be interested http://www.pfeifferreport.com/v2/wp-content/uploads/2013/05/UIF_Rep.pdf )

          The point being that once we had thought of the concept of user interface friction, describing it and measuring it became easier. But we could never have gotten that notion through simple market research.

          So this is what we are trying to achieve for tablets and smartphones: change the way in which we discuss them. “Cognitive load” “user experience friction” are essential notions. There are others, but I won’t go into detail here.

          But as I said before, these are all small first steps in what I hope will evolve considerably over time.

  • http://www.facebook.com/profile.php?id=1375644377 Dave Bucci

    There’s no details on how you weigh or combine your various scores. The starkest negative mark for Android is your “cognitive load” concept. What is the overall score’s sensitivity to the results in that category?

    I do also question your methodology for cognitive load … for a non-technical user, the cognitive load of using the home screens for the basic functions they do should be the most important measure, not a raw count of how many apps ship by default on the device. Even settings changes has to be 2ndary to how easy/difficult it is to do the basic, core functions the average non-technical user does on the phone, using the home screens, where such a user spends the vast majority of his/her time.

    • Andreas Pfeiffer

      Thanks for pointing this out — you are absolutely right we should have specified the overall rating system (and we will do so in our next update to the report.)

      Each category had the same weight in the overall score, i.e. 25%. However, since there is an obvious disproportion between the cognitive load score, and the others, the absolute result of the cognitive load score were scaled by 50%, which brought them within the same range as the other results,

      We assumed an ideal score of 100 for each category (which currently no operating is even close to reaching). Cognitive Load and User Experience Friction were input as negative numbers, i.e. deducted from the ideal score, or, if you will, as result that diminishes the overall score. The final score is the average of the results for each category.

      This means that Android, for instance, with a cognitive load count of 162 had a final cognitive load result of 19 (i.e. 100 – 162/2)

      I hope that makes things clearer.

      BTW, we have just uploaded an updated version of the report, which contains full details of the efficiency and customization aspects taken into account for the rating. You can download it here: http://www.pfeifferreport.com/v2/wp-content/uploads/2013/09/iOS7-User-Experience-Shootout.pdf

      • http://www.facebook.com/profile.php?id=1375644377 Dave Bucci

        Ok, that’s starting to help – to clarify, the cognitive load results then were:

        – Android: 19
        – iOS7: 80
        – iOS6: 84
        – WP8: 80
        – BB10: 73.5

        You then normalized to 25 by dividing by 4, correct? With resulting contributions to the overall score of Android 4.75, iOS7 20, iOS6 21, WP8 20, and BB10 18.375. Is that correct?

        So Android’s cognitive load evaluation lowered its overall score, compared to iOS7, by 15.25 points … is that all correct?

        If so, that emphasizes even more that the quality of your evaluative criteria for cognitive load is absolutely critical – since if that category were removed, Android and iOS6/iOS7 would be judged as effectively the same, by your other criteria.

        You didn’t address my contention that counting apps and UI elements doesn’t seem like a very high quality way to evaluate cognitive load. (and on the other hand, you don’t seem to dock iOS* in the customization category anywhere near as much as would be reasonable by any common definition of customization).

        Not being snarky – just trying to understand and dampen any harmonic down to the best possible study results. I do tradeoff studies for a living, and am very aware of how the evaluative criteria end up controlling the results.

        • Andreas Pfeiffer

          Thank you for your thoughtful comments!

          The cognitive load count you calculated is correct.

          Each group of results contributed 25% of the overall score. Android is better than iOS in Customization, and both are equal in Efficiency. (We have posted an updated report with detailed lists of the specific features and a description of the methodology at the top of this page). Androids UXF score is almost twice as high as iOS, so the two factors that account for the difference between iOS and Android are Cognitive load and UXF.

          I agree that the cognitive load measure we are using is barebones, and that much more needs to be done. That being said, you cannot argue that the elements we took into account do not exist – only that we should have taken into account more aspects.

          Based on the other angles of cognitive load analysis that we have looked at over the past year, however, it is pretty clear to me that Android would have fared if anything worse had we included other elements: the interaction tree of Android is far more complex than iOS, and so are basic interaction procedures that need to be acquired to use Android. (Just take eliminating apps as an example). There are many other examples. The main problem being that Google is trying to turn Android into a fully fledged operating system that can compete with its desktop rivals, while Apple is trying to stay within the device realm. Both approaches have their merits, but Google’s yields a more complex system. (the presence of both apps and widgets in Android is a good example for that tendency.)

          As I said in an earlier comment, we see this as a first step. A version 1.0 if you like. I think it is essential to start introducing these concepts into the way we analyze technology. For me it is the beginning of a discussion that needs to evolve over time. A lot more needs to be done, and we are delighted about all constructive comments.

          • http://www.facebook.com/profile.php?id=1375644377 Dave Bucci

            No argument, Android has a higher cognitive load, and understood that evaluative criteria evolve. Maybe a cognitive load measurement based on common tasks – with weighting of those tasks by importance – would be the right balance (though it starts to infringe on your UX criteria, possibly). To use your example – deleting an app is harder, but also less common and less important for the userbase you posit as your target for the evaluation. Is it harder to make a phone call on Android? Possibly, but not by much – and that would have to be a higher priority use case. To use it to find information? Probably easier, and that would again be a somewhat higher priority use case than deleting an app. etc. etc.

            No need to reply, i can wait for v2 of whatever you’re working!

          • Andreas Pfeiffer

            Tablets are much easier than phones. Tablets have no obvious primary use, and you can outline a coherent context much more easily.
            All of the aspects we covered need to be expanded upon significantly , but as soon as you try to do that there are so many things to take into account that it becomes really convoluted: gender bias, age-bias, social background…

  • p3ngwin

    2 versions of iOS V’s just one for each other platform ?

    can i enter 2 people as a single team against other competitors in the Olympics, etc to give myself twice the chance of winning ?

    Even then, Samsung smartphones beat Apple in customer satisfaction in of all places the USA, ahhahahaha !

    http://www.tgdaily.com/mobility-brief/73329-samsung-beats-apple-hah-in-legit-customer-satisfaction-study

    • Andreas Pfeiffer

      The report states what we set out to do quite clearly: to compare the newly arrived iOS 7 to the four major OS present in the market: iOS 6, Android, WP8 and BB10. In other words, we compared the newcomer to the incumbents, and iOS 6 is one of them.

      • p3ngwin

        so you compare iOS 7, with its predecessor iOS6 because it’s still a “major version” ?

        doesn’t explain why you only list one version of each other OS.

        don’t Android and the others have major previous OS versions still in use ?

        this “comparison” is fundamentally flawed.

        • Andreas Pfeiffer

          There is a basic misunderstanding of our approach. We compared a radically redesigned newcomer to the 4 major incumbents, one of which is a previous release of iOS. If the newcomer had been a radically redesigned version of Android, we would have proceeded exactly in the same fashion: comparing the newly released Android version with the most widely used older one &s well as other incumbents ( and of course we would have only taken the most widely used release of iOS into account, not several.)
          In which way is this approach (which is clearly stated in the report) “fundamentally flawed”?

          • p3ngwin

            from your own sidebar description:

            “Pfeiffer Consulting has conducted in-depth user experience benchmarks of the 5
            major mobile operating systems currently in use: iOS 7, iOS 6, Android,
            Windows Phone 8 and Blackberry 10 -”

            There are not “5 major mobile operating systems currently in use”, and there is not a single Android OS version in use, in fact there at least 3 major “versions” of android currently in use,.

            nowhere do you “clearly state” you are taking a newcomer and comparing to existing OS versions, *including* the previous version of the newcomer.

            There seems to be a basic misunderstanding of your definition of what it means to be “clear”.

            The entire “Article” is dripping with Apple bias with such examples as:

            “The arrival of iOS 7 will have a profound impact on the market for connected digital devices”

            “iOS 7 is necessarily a momentous event for the smartphone market”

            “iOS 6 has good basic customization options, and offers some accessibility features which are still missing in competing mobile OS’s”

            “Android has the longest list of customization options of all mobile OS’s surveyed, but lacks some of the accessibility options present in iOS”

            “What separates the Android user experience from iOS7 is not functionality, but feature-bloat and sloppy user interface design.”

            there are plenty more examples of such bias, but it’s clear enough this isn’t a genuine article aiming to describe the pro’s and con’s of each platform, and is instead shallow flattery masquerading as journalism.

          • Andreas Pfeiffer

            The title of the report states the content very clearly: How iOS 7 stacks up. Would I follow your logic I would have to include two or three other versions of iOS as well, as well as older versions of Blackberry (at least 2) and Windows Phone 7.

            As for the rest of your accusations: You are quoting individual lines of the report out of context, carefully avoiding the preceding or following texts that explain and validate the statement. There is no point in discussing the report on that basis.

            I am just picking the first line you single out. Her is what the whole introductory paragraph says:

            “Why is it that the arrival of iOS 7 is necessarily a momentous event for the smartphone market? Simple: Unlike any other operating system out there, it will be in the hands of millions or tens of millions of users within a few days after its launch.

            And that will make it a force to be reckoned with.

            The question is, of course: How good is it really?”

            If you had actually objectively compared the information and analysis presented in the report, you would see that all of the statements you single out are based on facts we have surveyed. If you have doubts, compare the tables at the end of the report.

          • p3ngwin

            your first sentence makes no sense.

            “We are comparing a highly anticpated newcomer to the incumbents – and
            the incumbents are the latest available versions of each major OS in the
            market, including obviously iOS 6.”

            how can iOS 6 be both “the latest available version” and also iOS 7’s predecessor ?

            This still does not explain why you would have 2 versions of iOS and only version for each of the alternative platforms.

            “Would I follow your logic I would have to include two or three other versions of iOS as well, as well as older versions of Blackberry (at least 2) and Windows Phone 7″

            It’s astonishing you think it’s unreasonable to compare an equal number of versions for the OS’s you are comparing.

            As for your comments being “taken out of context”, that’s laughable, as your full quotes change nothing about their meaning.

            your example doesn’t address the fact you are still stating as fact iOS7 is a “momentous event”, and you’re qualifying it with the idea it will be available to many people quickly.

            How does that make it any different than many other OS launches?

            Heck desktop OSes have been achieving the same for decades, and the same happens with every mobile OS, so what’s so “momentous” about iOS7’s release ?

            The answer is nothing, which is exactly the value of your rebuttal, especially as you don’t see fit to address the rest of the examples demonstrating your bias.

          • Andreas Pfeiffer

            I regret that you disagree with our approach. It is customary in technology reviews to compare a new product to the products it will be competing with: a reviewer who analyzes the latest Nexus 7, for instance, will necessarily compare it also to the previous version of the same device, as well as to the competition.

            As to the importance of iOS 7, the considerable amount of media coverage the new release of Apples mobile operating system received should give you a clear indication that there was an unusually high demand for information.

            Our report analyzes a market situation and operating system services that impact the user experience, according to a methodology that is clearly defined and documented. I am sorry that you are unhappy with them.

          • p3ngwin

            customary maybe to compare newly released products to their predecessors, but not so customary to fawn over the product with such bias.

          • Andreas Pfeiffer

            The Nielsen-Norman appraisal of iOS 7 is an interesting analysis conducted from a pure usability standpoint. Several aspects criticized by the NN report are also pointed out in our report, in the user experience friction section. On the other hand the NN report, preoccupied with structural user interface design issues only, does not provide a precise user-context for their analysis, nor does it cover the user experience aspects our report covers.

          • p3ngwin

            it also doesn’t have your bias, such as my previous comment’s references, as well as your comparison of OS “customization”, concluding iOS is only one point below Android (6/10 v’s 7/10), while Windows Phone is apparently deserved of 2/10.

          • Andreas Pfeiffer

            I would suggest objectively comparing the list of customisation and accessibility options at the end of the report instead of just quoting the overall rating. WP8 lacks several very important options, hence the low rating. All this is documented in the report.

          • p3ngwin

            you missed the point of the comparison i refer to.

            i was demonstrating how you apparently judge Windows Phone to have only 2/10 for customization, yet iPhone with it’s similarly pitiful options for user customization gets almost as much as the “dizzying granularity” Android offers.

            Apparently Android’s lack of “some customization options” compared to iOS offers makes all the difference, despite iOS having closer to Windows Phone’s level of options.

            Those “accessibility options” you mentioned must be *really* important to warrant ignoring the complete lack of customization options iOS has compared to Android “dizzying granularity”.

            Yet it’s strange how “accessibility” is quantified in your report, because it’s not apparent why “readability” is valued, yet Android’s unique “widgets”, etc aren’t even mentioned or valued at all as a form of “accessibility”.

            you mention Windows Phone’s lack of background and tile colour options, yet Android “widgets” and “live backgrounds” are worth nothing ?

            1 point, that’s all you deem the difference is in “customization” between iOS and Android ?

            So it seems a few “accessibility options” means iOS gets almost as much as Android for customization, while Windows Phone gets only 2/10.

            Demonstrating how biased your report is.

          • Andreas Pfeiffer

            Accessibility means features for handicapped people, not some strange concept we introduced. The list of features taken into account is published at the end of the report. Compare what is available on each platform.

          • p3ngwin

            none of what your pointing to is supporting your position.

            you say “accessibility” on iOS “offers some features still missing in competing OS’s”.

            So what?

            Android offered options not available on iOS too, but you didn’t mention that in the summary, but you don’t miss a beat to mention Android “lacks some of the accessibility options present in iOS”.

            Well how about that? The bias is strong with this one.

            Looking at the detailed report at the bottom reveals……

            For “Accessibility options”:

            iOS uniquely had:
            1- High contrast
            2 – Guided Access
            3 – Switch Control
            4 – Assistive touch

            While Android uniquely had:
            1 – Speak password
            2 – Colour Adjustment
            3 – Enhance web accessibility

            and for “Customization options”:

            iOS uniquely had:
            1 – customizable notification area
            2 – Change boldness type

            While Android’s customization options uniquely had:
            1 – Change font style
            2 – Customizable quick-access controls
            3 – Choose specific home page
            4 – Customizable quick-access controls
            5 – Choose specific home page

            Of course, despite Android’s customization superiority, Apple’s accessibility options were the ones deemed more valuable overall. Leaving the difference not a gaping one as it should be, but instead a single point separating the two.

            For customization you didn’t even mention the fact iOS forbids “default associations” to be changed, one of the most basic customisation features of an OS, and yet Apple’s iOS has no option for the user to change the default application used to open files or links, etc.

            E.G. Emails containing URL links will open in Apple’s own Safari, because there is no way to change that to something like Opera, Firefox, or Chrome.

            Same for maps links, etc. no option, it will open in Apple’s maps, no other option is available to the user, and horrible “cut ‘n’ paste into desired app” is the only workaround .

            you’d think that would be rather important when reviewing an OS’s “customization” feature list yes ?

            no mention of Android’s more relevant features from it’s “dizzying granularity of options”, such as the fact you CAN change default app associations, and “live backgrounds”, etc

            No mention….nothing.

            It’s obviously either not important or the reviewers are ignorant of such customization options, probably because they’re too busy fawning over the Apple product, so clearly there’s they’re worth practically the same marks for “customization”.

            it’s clear the reviewers don’t even know how to use the technology they’re paid to assess, and as such the report has no credibility at all.

            This is the last i’ll write on the matter as you’ve spoken much and yet failed to address the embarrassing deficiencies of your report. As such, the topic has lost any value it once had.

            Thank you, and goodbye.

          • Andreas Pfeiffer

            There is indeed not much point in continuing this discussion, since you steadfastly seem to avoid taking into account the context defined for the research (clearly stated on the first page of the report), without which our analysis does not make sense.

            Yes, I find accessibility options more important than the power-user features you list as missing from our analysis. Improving the comfort of use for users with disabilities is IMHO significantly more meaningful than changing which browser will open a URL. I’m surprised that anybody could view it differently.

  • gc8

    This study is biased. There’s nothing wrong with iOS7. It’s just that it caters to people who like to play “safe” and are happy with the experience. Things can go wrong with Android,and there are mostly things that go right for those who like to customize. I have both platforms. The Android platform allows you to break the rules. I prefer that platform because it gives me a better user experience. The fact that iOS has imitated Android’s control panel and the Play store certainly tells something.

    To delete an app in Android, all I have to do is long press and click Delete from the pop-up menu. The aftermarket launcher allows me to do this. An iOS user will not know about the many options you can do to an Android. I can slide to unlock to the camera or the 7 other apps. Your study states that it cannot. Simply put, the Android isn’t for people who aren’t tech savvy. Another reason why I told my parents to stay with iOS.

    In summary, it takes work to bring the Android to a superior experience. Whereas, the iOS 7 does well out of the box.

    • Andreas Pfeiffer

      Our study used a clearly defined context, that of the non-technical user, for whom, as you state yourself, iOS is better. That does not make it biased, it just means that we analyzed the platforms from a vantage point which is different from yours.

  • Andrew Watson

    I notice that you’ve analysed Samsung’s Touchwiz UX as “Android” and there’s a few things I would like to point out. Firstly, while I concede that the majority of “Android” devices today have the Touchwiz UX, I would like to point out that not all Android devices are made by Samsung, and that Touchwiz is vastly different in many ways to the “vanilla” or “stock” Android UX which is found on other devices (Google Nexus devices, ASUS, Motorola). I’d also like to point out that the Touchwiz UX is not in representative of the UX on all Android devices and is significantly different from how the makers of Android (Google) would like Android to be.

    Was this a deliberate decision to analyse Touchwiz specifically and not “vanilla” Android? And would you be open to the possibility of analysing “vanilla” Android on a nexus device? I notice that a number of the UXF issues with “Android” in this report are specific to Touchwiz, and are not present in “vanilla” Android (access to camera, redundancy of apps and widgets, advertising on homescreens, feedback when failing to place a widget/app). Additionally, the majority of the “default user interface elements” contributing to the high “cognitive load” for the Samsung device are again Touchwiz specific, and not present in “vanilla” Android (my Nexus 4 device came initially with 28 apps, and 30 widgets, much less than the 104 in the Touchwiz device analysed).

    One more thing (and this is more of a curiosity) I’m wondering how you’re defining “unintuitive” in the UXF section. I personally find that dragging an item to a Trashcan icon (which has been used in Operating Systems for >15 years) much more intuitive compared to clicking and holding, then tapping a small ‘-‘ icon to remove something, but it completely comes down to preference, and neither would be more or less “intuitive” than the other.

    • Andreas Pfeiffer

      As I said in my comment above, Samsung has such overwhelming market share that focusing on this vendors implementation seems coherent from the perspective of market reality for users. The situation is completely different for tablets, where there are many sellers with comparable marketshare. in our tablet report we cover 3 different Android devices: The google Nexus (stock android), the Samsung Galaxy Tab 2 (Samsung’s Android) and the Kindle Fire HD (again a heavily customized version of Android).

      The other problem is that it is not clear to me which vendor with any reasonable international marketshare actually ships real stock Android, since most of them customize Android to some extent. Trying to cover the reality of Android smartphones would mean including a large number of vendors, Google, Motorola, Sony, HTC, etc. etc. some of which are not even yet available around the world, or produce different versions for different markets.

      In short, it would be interesting to do such a report, but it would differ quite significantly from the one we set out todo.

      As for your question about erasing files: Whether we like it or not, the basic interaction paradigms for modern smartphones have been pioneered by the original iPhone. that’s where users learnt to press an icon for a few seconds, and to be presented with a cue to erasing it (an x on iOS and BB10, a crossed pin on WP8) Android requires a different action, dragging the app to a dustbin, which i a user interface convention from the personal computer. Since it involves additional steps than the established method we judged it less intuitive. Had we performed efficiency benchmarking for this operation, it would have shown that Android is much slower performing that operation.

      I hope that answers your question.

      • Jonathan Berry

        While you were careful to label the charts as “Android (Samsung)” (thank you for that) your article still gives the impression you are judging Android as a whole. I understand your reasoning as Samsung does have very large market share among Android phones, but it seems reasonable you could have included a Nexus device. This does two things: 1. It allows you to review the most current version of Android and 2. It represents Google’s vision for Android.

        As for removing apps, I find your arguments unconvincing. Many people are coming to smart phones for the first time and going to Android. Not everyone has used an iPhone while most people have used a computer. Also, I don’t see how Android’s method of uninstall takes any more steps than iOS’s. In Android you: 1. Find the app in the app drawer, 2. Long press it, 3. Drag it to uninstall (this is vanilla Android, I’m assuming Samsung’s TouchWiz does the same/similar). This maps very closely to iOS: 1. Find the app (slightly different since iOS doesn’t have an app drawer/home screen separation). 2. Long press it (same), 3. Tap the uninstall button (seems very similar to me). So, I count the same number of steps. And if you find dragging the icon to the uninstall button too laborious, you can just “fling” the icon upwards to initiate and uninstall. What “additional steps” are you talking about?

        • Andreas Pfeiffer

          Intellectually I completely agree, and for the tablets report we did exactly that. In that market place, where there are several equallly sstrong players, it’s the only way to go. Doing the same thing for smartphones had several problems, and one was simply that Google has not been selling the Nexus 7 in Europe for a long time – we had started working on the report several months ago before you could get the Nexus 7. We also had an HTC One for review but didn’t use it in the end. (The fragmentation of the Android market is a real problem, whichever way you take it, there is always some vendor you have overlooked whose fans get mad at you.)

          I hear what you are saying about erasing apps. There are two things I’d like to say: one is that many Android users told me they had issues with finding the way to do it at first; and the other one is that we have conducted action-based efficiency benchmarking for over a decade, testing things like menu latency and mouse precision on computer operating systems, and when you actually measure those things, you realize that the slow-down is actually much more significant then on thinks. We haven’t benchmarked erasing files on smartphones, but from my experience there is no doubt that we would find a 30-50% efficiency difference if we did. (If you are interested in that sort of stuff you can read more here :http://www.pfeifferreport.com/v2/wp-content/uploads/2013/05/UIF_Rep.pdf

  • blobsberg

    Why were there two versions of iOS in the benchmark but only Android with touchwiz was evaluated instead of stock Android? Many of the features that came with the results were part of touchwiz alone, which is a third party launcher running on top of Android. The stock launcher has a lot less cognitive load and some of the other popular launchers in the Play Store allow for much more customization. The fact that you can have different launchers alone should count big time for Android in the customization regard and it is not a criteria here for some reason…

    It might take an extra step to get these customizations done but I think these days even the non-technical average user knows how to download an app.

    • Andreas Pfeiffer

      The reports sets out to compare the newly arrived iOS 7 to the four major OS present in the market: iOS 6, Android, WP8 and BB10. In other words, we compared the newcomer to the incumbents, and iOS 6 is one of them.

      We chose Samsung’s version of Android because of Samsung’s overwhelming market share: for the majority of smartphone users, Android is what Samsung sells to them.

      The situation is completely different for tablets, where there are many sellers with comparable marketshare. If you check our report on 7 inch tablets, you will notice that we have 3 different Android devices: The google Nexus (stock android), the Samsung Galaxy Tab 2 (Samsung’s Android) and the Kindle Fire HD (again a heavily customized version of Android).

      • blobsberg

        This makes sense but still… I would speculate phones running stock android or some version pretty close to stock are higher in number (or at least comparable) than WP8 phones and Blackberry phones…

        • Andreas Pfeiffer

          This really depends which geographic zone you are looking at – China today has a 30% share of the global smartphone market, and Android is totally predominant there. If you look at recent US figures for smartphone subscription from Comscore, Apple is leading with 39%, Samsung with 22%, followed by HTC, Motorola and LG, none of which run stock Android to my knowledge. http://www.comscore.com/Insights/Press_Releases/2013/9/comScore_Reports_July_2013_U.S._Smartphone_Subscriber_Market_Share
          In Europe, the picture shifts from country to country, with Samsung leading, followed by Apple, Sony, HTC…

  • Proteous

    I think a lot of users on twitter would disagree with this study:

    https://twitter.com/search?q=ios7%20sucks&src=typd&f=realtime

  • Michael

    I have just only one question. I didn’t find the answer in your report – how many people were object of your research? And how you choose the people for the research… i’m interesting about their previous experience with different types of smaprtphones.

    • Andreas Pfeiffer

      The research project analyzed in detail the aspects of the mobile operating systems that impact user experience, based on the conceptual framework we have established, and which can be downloaded here: http://www.pfeifferreport.com/v2/essays/user-expereince-benchmarks/
      In other words, we weren’t out to quanitify the reaction of users; what we set out to do was to quantify empirically surveyable aspects of the operating systems, and to rate them according to the rating method described in the research reports.

  • Hasaan Rafique

    “Cognitive load—the sum of elements you need to get familiar with in order to use a device spontaneously and intuitively—is one of the key aspects of user experience for a non-technical user. For this benchmark we counted the number of apps/widgets as well as other icons and user interface elements a default installation of the operating system contains.”

    I disagree with the ‘cognitive load’ concept; it’s hardly appropriate. i.e. if Android has more apps/widgets/UI control elements, it naturally takes more time (for a non-technical user) to get familiar with, BUT once it does, it gets all the more better because users have many more ways to control their device intuitively.

    What should infact be a plus point for Android has been made to backfire in this survey. Also, it should be brought into consideration that not all the apps/widgets/controls have to be used anyway, which would lower the cognitive load practically.

    We’re talking about smartphones here. Greater control is essential. A simple OS that is easy to learn but offers less control is not at any advantage.

    I also disagree with the customization graph. iOS 7 is not even close to android yet, no replaceable dialers, keyboards, homescreens, lockscreens etc. The graph almost puts them parallel..

    • Andreas Pfeiffer

      Thank you for your comments. For an expert, or a technically minded user, everything you say is correct – but that is not the context we have defined for our research, as is clearly stated at the beginning of the report.

      As for cognitive load, you cannot argue that it does not exist. Elements that are presented to you are weighing on your efficiency. A program with twenty items on a menu has higher cognitive load than a program with five items. Having to go through 40 or 50 apps (not to say over a hundred widgets and apps) to find the one you want is more confusing than to be presented with a single screen of ten or twenty items. Actually, very often when we show the cognitive load comparison illustrations comparing Samsungs implementation of Android and iOS, people tell us that they understand better why they find their Android phones confusing.

      But again, we are talking about non-technical users.

      For the same reason, we did not list every single customisation option on either Android or iOS, leaving out more arcane features in both operating systems. Again a question of context: We are looking at features that a casual user can easily discover and understand. The introduction of the report is quite to the point about it: “We do not look at features, we do not compare cutting-edge options and gadgets, we only look at aspects that have a direct impact on the day-to-day user experience of an average, non-technical user.”

Back to top