Abstract
With the advent of mobile devices, the experience sampling method (ESM) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. Mitigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.
| Original language | English |
|---|---|
| Title of host publication | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
| Publisher | Springer Verlag |
| Pages | 203-210 |
| Number of pages | 8 |
| Volume | 9297 |
| ISBN (Print) | 9783319226675 |
| DOIs | |
| Publication status | Published - 2015 |
| Event | 15th IFIP TC 13 International Conference on Human–Computer Interaction, INTERACT 2015 - Bamberg, Germany Duration: 2015 Sep 14 → 2015 Sep 18 |
Publication series
| Name | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
|---|---|
| Volume | 9297 |
| ISSN (Print) | 03029743 |
| ISSN (Electronic) | 16113349 |
Other
| Other | 15th IFIP TC 13 International Conference on Human–Computer Interaction, INTERACT 2015 |
|---|---|
| Country | Germany |
| City | Bamberg |
| Period | 15/9/14 → 15/9/18 |
Fingerprint
Keywords
- A/B testing
- Experience sampling method (ESM)
- Usability
ASJC Scopus subject areas
- Computer Science(all)
- Theoretical Computer Science
Cite this
On applying experience sampling method to A/B testing of mobile applications : A case study. / Lee, Myunghee; Kim, Jeonghyun.
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). Vol. 9297 Springer Verlag, 2015. p. 203-210 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 9297).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
}
TY - GEN
T1 - On applying experience sampling method to A/B testing of mobile applications
T2 - A case study
AU - Lee, Myunghee
AU - Kim, Jeonghyun
PY - 2015
Y1 - 2015
N2 - With the advent of mobile devices, the experience sampling method (ESM) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. Mitigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.
AB - With the advent of mobile devices, the experience sampling method (ESM) is increasingly used as a convenient and effective way to capture user behaviors of, and evaluate mobile and environment-context dependent applications. Like any field based in situ testing methods, ESM is prone to biases from unreliable and unbalanced data, especially for A/B testing situations. Mitigating such effects can in turn incur significant costs in terms of the number of participants and sessions, and prolonged experimental time. In fact, ESM has rarely been applied to A/B testing nor do existing literatures reveal its operational details and difficulties. In this paper, as a step toward establishing concrete guidelines, we describe a case study of applying ESM to evaluating two competing interfaces for a mobile application. Based on the gathered data and direct interviews with the participants, we highlight the difficulties experienced and lessons learned. In addition, we make a proposal for a new ESM in which the experimental parameters are dynamically reconfigured based on the intermediate experimental results to overcome the aforementioned difficulties.
KW - A/B testing
KW - Experience sampling method (ESM)
KW - Usability
UR - http://www.scopus.com/inward/record.url?scp=84945965724&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84945965724&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-22668-2_17
DO - 10.1007/978-3-319-22668-2_17
M3 - Conference contribution
SN - 9783319226675
VL - 9297
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 203
EP - 210
BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
PB - Springer Verlag
ER -