On applying experience sampling method to A/B testing of mobile applications: A case study

Myunghee Lee, Jeonghyun Kim

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

With the advent of mobile devices, the experience sampling method (ESM) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. Mitigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer Verlag
Pages203-210
Number of pages8
Volume9297
ISBN (Print)9783319226675
DOIs
Publication statusPublished - 2015
Event15th IFIP TC 13 International Conference on Human–Computer Interaction, INTERACT 2015 - Bamberg, Germany
Duration: 2015 Sep 142015 Sep 18

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9297
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other15th IFIP TC 13 International Conference on Human–Computer Interaction, INTERACT 2015
CountryGermany
CityBamberg
Period15/9/1415/9/18

Keywords

  • A/B testing
  • Experience sampling method (ESM)
  • Usability

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Fingerprint Dive into the research topics of 'On applying experience sampling method to A/B testing of mobile applications: A case study'. Together they form a unique fingerprint.

  • Cite this

    Lee, M., & Kim, J. (2015). On applying experience sampling method to A/B testing of mobile applications: A case study. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 9297, pp. 203-210). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9297). Springer Verlag. https://doi.org/10.1007/978-3-319-22668-2_17