Frames of reference for evaluating new learning technologies

New educational technologies provide opportunities for gains in resource efficiency and in educational effectiveness. Are the anticipated benefits realised? This paper reports on an attempt to apply a frame of reference to the selection of criteria for evaluating technological innovations in higher education. Defining the types of innovation to be evaluated is an issue but establishing evaluation criteria is the major challenge. Four positions are considered. One is oriented to the stated objectives of the particular educational innovation; one to comparison with alternative educational approaches; one to the benefits and costs anticipated from a knowledge of the state of the art of learning technologies; and one to criteria developed from a particular educational and/or social theory. These four approaches to are interactive rather than exclusive zones of reference, each of which has a place in selecting criteria for evaluation of educational innovations employing learning technologies. To engage all four, exposes contests between theoretical foundations, public policy, local pragmatics and individual objectives. Evaluation in this context, then, identifies dialectical relationships and provides a transformative tool for the construction and reconstruction of technology in education. In the background are issues of causation in complex educational contexts. Attaining generalisable conclusions from the evaluation of educational technologies, however, is problematic and beyond the scope of an evaluative approach. The paper reports on an evaluation of an innovation that combined the introduction of problem-based learning with Web-based tuition in a post-graduate course conducted at an Australian university.