Background: This paper explores the robustness of the cognitive and perceptual skills developed within a game-like instrument scanning training task. The task uses static, stylised instruments and explores transfer of training to perceptual manipulation and to a dynamic training task. Method: Participants (n=50) completed a static instrument scanning task (including static transfer), presented via a laptop under supervised conditions. They then completed a dynamic instrument scanning task to test for transfer of static training to the more realistic dynamic environment. A small number of experienced pilots (n=3) completed the dynamic task alone to ensure that the task tapped skills relevant to the real world flying domain. Results: Participants could perform reasonably accurately (median 80% correct compared with a chance level of 10%) on the static instrument scanning task after minimal training, and response times improved significantly across training trials, including static transfer trials. Performance on the dynamic task was uniformly poor even for the subset of high-performing participants (n=19) who were the most accurate on the static task. However, the experienced pilots were able to perform well on the dynamic task suggesting that this task draws on skills required in the aviation context. Conclusion: Despite superficially good performance on a game-like static instrument scanning task, there is little evidence of transfer of training to a dynamic training environment with more ecological validity. The gap between game-like training environments and real world operational tasks is likely to be substantially greater, such that the potential for training transfer is further reduced.