Criteria for selecting and evaluating digital learning tools

Rationale and scope of the study

The range of digital learning tools, from educational games and mobile apps to online lessons and AI-supported study assistants, continues to grow. Used well, this variety is a valuable opportunity to support learning; however, it also makes choosing more difficult for educators, families and educational institutions. The features highlighted in the promotion of tools often do not coincide with the qualities that matter most for learning.

This study sets out a framework of criteria that can be used in selecting and evaluating digital learning tools. Its scope covers content, applications and AI-supported tools for different stages of education, from primary school to university. The framework does not evaluate or rank specific products; it defines the questions to be asked when making a selection and the order in which they should be addressed.

The framework has been prepared with three user groups in mind: educators choosing tools for their students, families wishing to evaluate the tools their children use, and educational institutions making procurement decisions at institutional level. The criteria are defined simply enough for all three groups to use; detailed reviews requiring specialist support, in areas such as data security and accessibility, are indicated separately.

Criteria areas

The framework considers a learning tool in six areas. Rather than being scored independently and added up, the areas are assessed together; a serious shortcoming in one area is not considered to be offset by strengths in others. For example, a tool that is pedagogically strong but processes children's data for unclear purposes is excluded from selection.

• Fit with the learning objective
• Suitability for age and developmental level
• Pedagogical design and accuracy of content
• Accessibility and usability
• Data security and privacy
• Sustainability and support

Each area is made concrete through a set of evaluation questions. The questions are written to require a reasoned assessment rather than a yes or no answer. The assessor notes the basis for each answer, such as the tool's terms of use, content samples or pilot observations.

Learning, age and content criteria

Fit with the learning objective is the starting point of the evaluation. Questions should be asked about which knowledge or skills the tool aims to develop, how this aim relates to the curriculum or lesson plan, and how the student's progress is made visible. A tool whose purpose cannot be defined cannot be evaluated in terms of learning, however engaging it may be.

Age-appropriateness covers not only the difficulty level of the content but also the mode of interaction, screen time, the reward and notification system and the language used to guide learners. In tools aimed at primary school children in particular, elements that draw attention away from the learning objective, as well as advertising content, should be examined carefully. In the area of pedagogical design and content accuracy, the following questions stand out:

• Has the content been reviewed by a subject expert or an educator?
• Does the tool only tell students whether an answer is right or wrong, or does it explain the reason for the mistake?
• Do the activities offer opportunities for application and reflection beyond repeating information?
• Is human oversight in place to guard against incorrect or misleading content in AI-supported responses?

AI-supported tools require additional attention in this area. A tool that gives students direct answers may shorten the learning process and take the place of the student's own effort to think. It should therefore be examined separately whether the tool guides the student towards the answer step by step, whether the level of hints and explanations can be adjusted, and whether the educator can monitor how the tool is used.

Access, data and continuity criteria

The accessibility and usability area considers whether the tool can be used on different devices, at low connection speeds and by students with different learning needs. The key question in this area is whether alternative means of access are offered for users with visual, hearing or mobility impairments.

When the data concerned belongs to students, and especially to children, data security and privacy are an integral part of the selection decision. What data is collected, for what purpose it is used, where it is stored and who can access it must be answered clearly. Organising data access according to user role, so that the student, the family and the educator can access only the information their role requires, is a fundamental condition. Progress reports provided to families are also evaluated in this area: reports are expected to be clear, to describe progress and the next step rather than using language that labels students or compares them with others, and to explain which data they are based on.

The sustainability and support area assesses whether the tool can be used over the long term. It examines how content is updated, whether user support is available, the conditions of free access, and the portability of data if the institution stops using the tool. For tools offered free of charge, it is also important that the way the service is financed is clear; models based on advertising or data sharing should be considered together with the assessment in the data security and privacy area.

Selection and evaluation process

For the criteria to be applied consistently, the selection process also needs to be defined. The framework sets out a five-step process:

• Defining the learning need and the target user group
• Preliminary review of candidate tools across the six criteria areas
• Educator review and accessibility assessment
• Pilot use with a limited group over a defined period
• A decision on wider roll-out based on pilot findings and feedback from students, families and educators

The indicators to be monitored during pilot use are set before selection. Progress towards learning objectives, the usability of the tool, content suitability in educator review and accessibility findings are chief among these indicators. Defining the indicators in advance ensures that the outcome of the pilot rests on comparable information rather than impressions. The decisions taken at each step and the reasons for them are recorded; these records provide a starting point for future evaluations addressing the same need.

Relationship to the programmes

This framework of criteria has been prepared for use in the Digital Education and Learning Technologies Programme, coordinated by the Education and Academic Cooperation Directorate, both in evaluating tools selected from outside and in reviewing content developed within the programme. Data and infrastructure criteria for AI-supported tools are addressed together with the Türkiye-Based Artificial Intelligence and Data Infrastructure Programme. Contributions from educators, academics and educational institutions to the development of the criteria are considered within the programme's academic cooperation activities. The framework is reviewed periodically in the light of changes in curricula, new types of tools and findings from pilot implementations.