On applying experience sampling method to A/B testing of mobile applications

A case study

Myunghee Lee, Jeonghyun Kim

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

With the advent of mobile devices, the experience sampling method (ESM) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. Mitigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.

Original languageEnglish
Title of host publicationLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PublisherSpringer Verlag
Pages203-210
Number of pages8
Volume9297
ISBN (Print)9783319226675
DOIs
Publication statusPublished - 2015
Event15th IFIP TC 13 International Conference on Human–Computer Interaction, INTERACT 2015 - Bamberg, Germany
Duration: 2015 Sep 142015 Sep 18

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9297
ISSN (Print)03029743
ISSN (Electronic)16113349

Other

Other15th IFIP TC 13 International Conference on Human–Computer Interaction, INTERACT 2015
CountryGermany
CityBamberg
Period15/9/1415/9/18

Fingerprint

Mobile Applications
Sampling Methods
Sampling
Testing
Unbalanced Data
User Behavior
Mobile devices
Mobile Devices
Experience
Concretes
Dependent
Evaluate
Costs
Experimental Results

Keywords

  • A/B testing
  • Experience sampling method (ESM)
  • Usability

ASJC Scopus subject areas

  • Computer Science(all)
  • Theoretical Computer Science

Cite this

Lee, M., & Kim, J. (2015). On applying experience sampling method to A/B testing of mobile applications: A case study. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 9297, pp. 203-210). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9297). Springer Verlag. https://doi.org/10.1007/978-3-319-22668-2_17

On applying experience sampling method to A/B testing of mobile applications : A case study. / Lee, Myunghee; Kim, Jeonghyun.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 9297 Springer Verlag, 2015. p. 203-210 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9297).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lee, M & Kim, J 2015, On applying experience sampling method to A/B testing of mobile applications: A case study. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 9297, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 9297, Springer Verlag, pp. 203-210, 15th IFIP TC 13 International Conference on Human–Computer Interaction, INTERACT 2015, Bamberg, Germany, 15/9/14. https://doi.org/10.1007/978-3-319-22668-2_17
Lee M, Kim J. On applying experience sampling method to A/B testing of mobile applications: A case study. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 9297. Springer Verlag. 2015. p. 203-210. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). https://doi.org/10.1007/978-3-319-22668-2_17
Lee, Myunghee ; Kim, Jeonghyun. / On applying experience sampling method to A/B testing of mobile applications : A case study. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 9297 Springer Verlag, 2015. pp. 203-210 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).
@inproceedings{31037307cd4d4190b55ae04582c2931b,
title = "On applying experience sampling method to A/B testing of mobile applications: A case study",
abstract = "With the advent of mobile devices, the experience sampling method (ESM) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. Mitigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.",
keywords = "A/B testing, Experience sampling method (ESM), Usability",
author = "Myunghee Lee and Jeonghyun Kim",
year = "2015",
doi = "10.1007/978-3-319-22668-2_17",
language = "English",
isbn = "9783319226675",
volume = "9297",
series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",
publisher = "Springer Verlag",
pages = "203--210",
booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

}

TY - GEN

T1 - On applying experience sampling method to A/B testing of mobile applications

T2 - A case study

AU - Lee, Myunghee

AU - Kim, Jeonghyun

PY - 2015

Y1 - 2015

N2 - With the advent of mobile devices, the experience sampling method (ESM) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. Mitigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.

AB - With the advent of mobile devices, the experience sampling method (ESM) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. Mitigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.

KW - A/B testing

KW - Experience sampling method (ESM)

KW - Usability

UR - http://www.scopus.com/inward/record.url?scp=84945965724&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84945965724&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-22668-2_17

DO - 10.1007/978-3-319-22668-2_17

M3 - Conference contribution

SN - 9783319226675

VL - 9297

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 203

EP - 210

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

PB - Springer Verlag

ER -