Mobile evaluation: what the data and the metadata told us

Evaluating mobile applications to identify usability problems presents a unique set of challenges. Not only is it difficult to capture data on an application that is inherently mobile, but generating an authentic environment of use is also problematic. This paper compares two traditional user-based approaches to evaluate a mobile system: one laboratory-based and the other in the field. These data serve as a basis for the primary focus of this study: the effectiveness of 'metadata' generated from 'rapid reflections'. These data were collected by the evaluators after each day of evaluation in order to investigate the quality of metadata in relation to the data of the two evaluation methods. The study also found that the laboratory study identified typical usability problems with the system at a more detailed level whilst the field study identified characteristic problems of mobile use. The metadata findings summarised the major findings in a useful way, but generally were less specific and reflected subjective theories of individual researchers.