The mobile app industry has reached unprecedented heights, generating $45 billion in the U.S. and $253 billion worldwide in 2023. Providing an exceptional user experience is crucial for the success of these apps. But how do we measure the effectiveness of these experiences?

We've explored the impact of perceived ease of use and usefulness on people's intention to use and recommend software products. We've also seen how those intentions predict actual behaviors, demonstrating a clear chain of influence from UX metrics to important business outcomes.

However, mobile apps require their own unique measurement approach. Until now, research has been limited by the lack of standardized questionnaires specifically designed for evaluating mobile app user experience (app user experience).

Early Mobile UX Measurement

Lusky and Boehm's 2017 review of methods used to evaluate mobile UX highlights the need for a taxonomy that categorizes methods into generic, mobile-adapted, and mobile-specific. Applying this framework to standardized questionnaires, we find numerous examples of researchers using generic UX questionnaires in research on mobile app UX.

For instance:

  • van der Heijden and Sangstad Sørensen (2003) used a standardized consumer acceptance questionnaire to measure attitudes toward mobile information services.
  • Dhir and Al-kahtani (2013) evaluated the UX of mobile augmented reality prototypes using the AttrakDiff questionnaire.
  • O'Malley et al. (2014) studied a mobile app for adolescent obesity management using the Software Usability Measurement Inventory (SUMI).
  • Kortum and Sorber (2015) assessed mobile applications for phones and tablets using the System Usability Scale (SUS).

Standardized Questionnaires for Evaluating Mobile App UX

We know of three standardized questionnaires that were developed specifically to assess the app user experience of mobile apps. The first two, the Mobile Phone Usability Questionnaire (MPUQ) and the mod-AUG scales, used classical test theory. The third, the SUPR-Qm, was developed using item response theory.

The MPUQ

The MPUQ is a multidimensional instrument with six subscales: ease of learning and use, assistance with operation and problem solving, emotional aspect and multimedia capabilities, commands and minimal memory load, efficiency and control, and typical tasks for mobile phones. Each subscale has 12 items, and the reliability of the subscales ranges from .82 to .93.

Some examples of the items are:

  • Ease of Learning and Use (LEU): "Is it easy to learn to operate this product?" or "Is discovering new features sufficiently easy?"
  • Helpfulness and Problem-Solving Capabilities (HPSC): "Is the HELP information given by this product useful?" or "Does the HELP function define aspects of the product adequately?"

The Mod-AUG Scales

Hoehle and Venkatesh developed a questionnaire based on Apple's UX guidelines, consisting of 78 items that measured six high-level constructs: application design, application utility, user interface graphics, user interface input, user interface output, and user interface structure. The reliability of the scales ranges from .75 to .85.

Examples of the items are:

  • Application Design (DES): "Overall, I think the mobile application is designed well." or "I am very satisfied with the overall design of the mobile application."
  • User Interface Graphics (INTG): "Overall, I think the graphics displayed on the mobile application are designed effectively." or "Overall, the mobile application has very good user interface graphics."

SUPR-Qm

The SUPR-Qm was developed using item response theory and consists of 20 items that measure two high-level constructs: app usability and overall satisfaction. The reliability of the scales ranges from .86 to .94.

Examples of the items are:

  • App Usability (AU): "How easy is it to perform tasks on this mobile application?" or "How well does this mobile application support your goals?"
  • Overall Satisfaction (OS): "Overall, how satisfied are you with using this mobile application?" or "Would you recommend this mobile application to others?"

By understanding the strengths and limitations of these standardized questionnaires, researchers and developers can work together to create more effective and user-friendly mobile apps that meet the needs of their users.