Assessment of COVID-19’s Impact on EdTech: Case Study on an Educational Platform, Architecture and Teachers’ Experience

Abstract

The education sector has been confronted with different challenges due to the situation caused by the pandemic, when families were asked to be confined at home as well as return when schools were opened. This lockdown situation presented both a challenge for the EdTech sector and for teachers and families. Consequently, this study analyzes the importance of online methodologies, usage of an educational resource example, and the impact of the lockdown. Thus, these objectives are assessed from different perspectives such as users’ consumption, technical challenges of cloud architecture and experience from teachers who have used the platform during the lockdown. In this way, to understand the challenges of Cloud architecture, the structure of the Pre-COVID-19 platform and the changes implemented to adapt to the new needs are described. An increase in schools’ subscriptions was observed when home lockdown was decreed, the differences in usage with the return to the classroom are also discussed. The research methodology entailed an assessment instrument developed for teachers. Teachers highlight the contents of Smile and Learn platform and their motivational characteristics for the students’ learning. The assessment points out the limitations that many teachers face while using these resources.

1. Introduction

This article puts forward the analysis of the use of Smile and Learn digital material in Spain during the COVID lockdown period and of the subsequent year, from three interdisciplinary areas focused on the educational area. Thus, the objectives of the study are:

• Explain the challenges produced in the EdTech sector in relation to the technological infrastructure of a digital platform. Address the architecture of the Pre-COVID-19 platform and the technical challenges that lockdown has posed;
• Analyze the use of the platform during the months of lockdown and the return to the classroom;
• Inquire into teaching experience using the digital platform through the development of an assessment instrument. This research aims to provide an assessment by teachers of the resources offered by Smile and Learn platform (see the Supplementary Materials), which were provided for free during the COVID pandemic. Therefore, the study looks at the teacher’s perspective in order to improve implementation processes in classrooms or teacher training that can be offered, as well as characteristics that would need to be improved in the resources.

This study is organized into five different sections, which follow the structure described below. Initially, Section 1.1 describes the Smile and Learn platform and its types of digital content (to contextualize the results presented). Section 1.2 describes the initial architecture on the platform before lockdown. Section 1.3 includes the state-of-the-art methodology in relation to the use and implementation of digital material in education.

Section 2 covers the research (materials and methods) that has been carried out through the educational assessment instrument (user experience assessment) and analysis of the usage of the digital platform. In addition, technical challenges that have had to be solved due to the increase in the number of users are indicated in Section 2.

This research has two hey aspects of interest. Firstly, the evaluation of the Smile and Learn materials and platform by their usage during the lockdown and back-to-school period. Secondly, we want to know teachers’ training and their opinions about digital material in general. These research results are presented in Section 3. In Section 4, the educational implications and the use of the material by the teachers are discussed. Finally, Section 5 summarizes the conclusions of this study and the main ideas to be considered in future research.

1.1. Smile and Learn Platform: Activity Types

Smile and Learn is an intelligent, digital platform in the Ed-Tech sector. The platform includes activities that cover core curriculum subjects from preschool to elementary school (3–12-year-olds). This way, the different subjects are merged facilitating the use of a single resource which can adapt itself to all areas of study. Likewise, the platform helps students to work with other aspects like competences and cognitive skills such as content language integrated learning (CLIL) [1]. As of March 2020, the platform had more than 5000 activities in five different languages (Spanish, English, Portuguese, French and Italian). These activities can be classified into the following types (Figure 1):

(a)

Quizzes: activities with the objective to evaluate the theoretical knowledge gained while working with applications of a specific study area/core subject. This way, in this kind of activity game mechanics like information, questions, answers and feedback are interconnected with learning mechanics like identifying, discovering, tasks, reflection or repetition. The pedagogical objective in combining these mechanics lies in understanding, remembering, analyzing and evaluating [2,3]. Similarly, according to Baptista et al. [4] using quizzes in learning promotes the development of strategic thinking skills like decision making (decision quality) and technical learning.

(b)

Games: more practice-based activities to work on curriculum concepts or cognitive skills. In these activities, game mechanics like behavior, information, steps, answers, progress, points or feedback are interconnected with learning mechanics like guides, identifying, exploring, experimenting, tasks, repetition, reflection or analysis. Thereby, the pedagogical objectives are to understand, remember, analyze, apply and evaluate [2,3]. The games integrated into the platform work on strategic skills like technical learning, decision quality or problem solving. These games also work on operational skills like the amount of time spent on decision making [4].

(c)

Stories: activities to promote an interest in reading through interactive resources and reading comprehension [5,6].

(d)

Theory: activities to learn basic concepts through interaction with the device.

(e)

Videos: through visual channels and with the help of theories like the Cognitive Theory of Multimedia Channels [7], core curricular theory concepts are visually displayed for learners. By these means, the intention is to achieve a meaningful learning experience using different information processing channels (visual and auditory) [8].

During the lockdown, Smile and Learn has been deployed in different proposal plans focused on all the population. On one hand, there has been an agreement with the Spanish Ministry of Education and Professional Training, and Radio Televisión Española (Spanish public service television) according to which specific Smile and Learn didactic content (videos) would be broadcast from Monday to Friday during the morning time slots. The morning show “Aprendemos En Casa” (Learning at Home) is an inclusive and educative program which aims at reaching every household, including those that lack interactive electronic devices. On the other hand, a free model of the Smile and Learn platform was offered to all those interested.

1.2. Pre-COVID-19 Architecture of Smile and Learn Platform

Most of the Smile and Learn platform has been deployed on Amazon Web Services (AWS) since its beginning. AWS is a cloud infrastructure which allows horizontal scaling of resources in order to adapt to changes in the demand or the load of the service. In particular, the production services of Smile and Learn involve the following components:

• A landing website which displays information about Smile and Learn, as well as promotions, news, etc. This is a standard development based on WordPress and was not deployed on Amazon Web Services, but in a standard (non-dedicated) hosting service;
• The Learning Analytics platform, which allows teachers and parents to see the progress of children, assign homework, etc. This application is also used by schools to manage their groups and students, and includes a management backend to automate certain tasks. The web server for this platform runs in an Elastic Cloud Compute (EC2) virtual server of type c5.2xlarge, featuring 8 virtual CPUs and 16 GB of RAM memory. Additionally, the database is located in AWS Relational Database Service (RDS) with a MySQL database management system, in a db.r5.large instance, which is optimized for memory intensive processes, and features two virtual CPUs and 16 GB of RAM;
• The bundles and contents of the Smile and Learn platform, which were stored in Simple Storage Service (S3).

Generally speaking, all of the aforementioned infrastructure is easy to scale, except for the landing website. Later in the paper we will discuss the technical challenges that this infrastructure faced during the early days of the COVID pandemic and how it was redesigned to adapt to the new load, as there are different reports from the educational community that have also detected this impact of the pandemic on digital resources, online methodologies and other challenges in the EdTech sector [9,10].

1.3. Digital Material and Online Education: State-of-the-Art

For many years, an increase in material, platforms and online resources has been observed in the education sector [11,12]. Studies such as Chauhan [13] highlight the learning possibilities of primary school students when using Information and communication technologies (ICT). This may indicate a decision to invest in implementing digital resources in the classroom. Nevertheless, the process of implementing such resources has been challenged by limiting factors. Some of these limiting factors related to technology can be the lack of devices in schools, obsolete devices or poor connectivity [12,14,15]. Among these limitations, the lack of budgets to acquire devices stands out. It may also be that these devices are outdated and not compatible with educational platforms or digital materials. Another constraint is the lack of time in the classroom. This makes it difficult to use these resources properly by following a temporarily adjusted curriculum. In addition, many of the materials are not adapted to the specific curriculum, which is a challenge for teachers interested in using these resources [16,17]. Moreover, limitations could be found in relation to teacher profiles. Therefore, it can be found that many teachers may require support materials or training for the correct use of these resources [12,18]. An additional point regarding teachers may be their attitude towards implementation [19,20,21]. In many instances, this leads to inadequate use of online methodologies or digital devices. Regarding teachers, who in many cases are motivated to implement this sort of resources, the lack of specific training and time to be able to use them in class are highlighted [22,23]. Although these digital resources can encourage active learning or can be customized to adapt to the learning rhythms of students, there is still a debate about their utility in the classroom [24,25].

Because of the global pandemic in 2020, the lockdown and the closings of schools, the educational community has been obligated to rely on these resources [26,27,28]. To be able to continue with their lessons, some teachers opted for online methodologies or e-learning. These are characterized by being a distance teaching and learning methodologies based on the use of technology devices and digital resources [29,30]. Thus, the search for resources and online teaching alternatives to continue with the training of students has been observed in the different educational levels [21,31,32]. As a result of the pandemic, teachers have been prompted to work on their skills in using these tools, as well as to reflect on the processes following the application of the online methodologies [14,31,33].

At the same time, this has signified an added engagement for the families [15]. Thereby, online teaching allows the continuity of the use of mixed methods that were already being used in the classroom. It is important to be aware of what these materials can contribute and how online methodologies can help students learn [34]. This way, this digital material can be used as complementary material to other methodologies that make use of digital resources like the flipped classroom method [35] or game-based learning [36]. The Flipped Classroom is a teaching methodology that reverses the traditional structure of work in the classroom. This methodology seeks to implement and encourage the study by students of those contents and activities of a subject that are easily studied by themselves outside the classroom through the use of ICT, such as viewing videos, lectures, etc. With the above, we seek to promote the learning and reinforcement of content, the development of exercises and practices, along with the application of skills during classroom time [37,38]. This model is being used more and more by teachers, since it helps to resolve doubts and carry out activities of greater complexity, which occupy more time or require specialized infrastructure during their classes and at the same time assist students with special educational needs or learning difficulties, thus allowing for continuous accompaniment and guidance by the teacher as described by Martínez [38] and Angelini Doffo and García-Carbonell [39]. The main objective of this methodology is to give the student a participatory role during the active learning process and collaboration between peers. Class time is freed up and the teaching-learning dynamic instead of focusing on theoretical presentation becomes a pedagogical process (suitable for all educational cycles) that seeks to encourage debate and therefore the commitment of the students themselves in their learning process based on constructivism [37,39,40].

Similarly, other methodologies and digital resources that contribute to the knowledge society have been implemented along with technologies such as gamification. In these methodologies, new scenarios are being proposed that seek to motivate, encourage and resolve concerns in students through new strategies, achieving an increased commitment to the acquisition of knowledge within technological ecosystems. Gamified activities, which use game elements, are also a learning tool to strengthen knowledge and skills to solve problems, meet challenges, work in teams and promote communication processes [41,42,43].

According to the 2030 Agenda of the United Nations, where the SDGs (Sustainable Development Goals) are embodied, one of the main problems to be solved by 2030 is the one related to education, which corresponds to the SDG 4. Quality Education: “Guaranteeing inclusive, equitable and quality education and promoting lifelong learning opportunities for all” [44]. All the experience lived during the pandemic has highlighted a critical review or reassessment of these established objectives as presented in the work of Crawford and Cifuentes-Faura [45] which can be considered for the impact it can have within the EdTech sector. Thus, this industry can contribute to objective 9 “build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation”, as discussed in the previous section in the structure of the platform analyzed.

In recent decades, great progress has been made with respect to education and its influence on poverty reduction by allowing school-age children and youth to have access to school. By 2018, around 260 million children still do not have the possibility of enrolling in an educational center or have access to technological tools, limiting their academic skills and the possibility of competing in their working lives with their peers. However, with COVID-19 and the temporary closure of schools, more than 91% of students were affected not only in terms of learning but also in terms of the benefits they obtained from attending educational centers (especially those with a higher degree of vulnerability and low economic resources), for example, with school meals, drastically changing their lifestyle and putting at risk the improvements in education worldwide [30,31].

In addition to schools and teachers, the pandemic has also affected families and students. Based on the Spanish literature in this area, dissatisfaction and negative perceptions have been detected among students at different educational levels. This has underlined the need to make education more flexible and adaptable to student-centered models [14,15,46]. The application of online methodologies has also been limited by the shortage of electronic devices at home or the lack of digital competencies. In Spain, according to data from the National Institute of Statistics [47], 19.1% of households did not have a computer and 43.2% did not have a tablet at home. These figures represent a considerable percentage of the population that could make access to education difficult. On the other hand, although many households are equipped with devices, these may not be sufficient for all family members according to the situation produced during the lockdown.

2. Materials and Methods

2.1. Participants

In order to analyze the use of the platform during the months of lockdown and the return to the classroom, data were obtained from the Smile and Learn platform, with participants being those schools that registered and accepted the platform’s privacy policy (information attached in Appendix A).

For the goal of inquiring into the teachers’ experience using the digital platform, the total participants are 70 teachers whose age range is between 31 and 62, with different professional experiences (from less than 5 years to more than 20 years). In addition, these 70 teachers are from 29 different schools, divided into two different types: public schools (n= 19 schools) and private schools (n = 10 schools).

Every teacher could participate voluntarily, securing his or her anonymity, therefore we did not ask about more personal information to prevent different cognitive biases such as social desirability [48] (C.f. ethical statements).

2.2. Measuring Instruments

The Smile and Learn platform database was used as an instrument to collect information on school usage. In addition, we have developed an ad hoc assessment instrument to evaluate all the teachers’ opinions of the Smile and Learn platform, e-learning and lockdown. The main advantage of this assessment instrument was that every item had an individual value, without item dependence and without a previous factorial structure. That is why the term “questionnaire” is avoided in favor of “assessment instrument”.

However, an exploratory factorial analysis of the items was performed to inquire into the structure of the instrument. The exploratory factorial analysis shows three statistical factors that explain 68.254% of the variance. Also, it is a reliability of 0.859 by Cronbach Alpha.

Nevertheless, due to the casuistry of this research, since it is a screening, although the reliability and validity values are good, it is necessary to cautiously analyze the psychometric values of this instrument, and it should be tested in future samples.

2.3. Procedure and Data Analysis

At first, the assessment instrument was developed without including specific personal questions trying to avoid different cognitive biases. Therefore, the assessment instrument was transferred to an online software for its faster and cheap dissemination. After that, a developed diffusion plan by e-mail was established, contacting schools aiming for their teachers’ collaboration. Schools selected for the diffusion of the assessment instrument were those that had signed up to the subscription model provided by Smile and Learn during the lockdown.

Descriptive analyzes, exploratory factorial analysis and reliability tests were carried out using SPSS v.25.0.0.1.

The data obtained from the Smile and Learn platform are analyzed using Excel v.16.64 and JASP v.0.16.3. For this study, the data have been converted to base 100 to comply with the regulations agreed with the company in its distribution of information in this study. The descriptive results are presented in graphs. A Wilcoxon signed-rank test was also performed to contrast the groups: the use during the lockdown, and the use in the same period of time the following year, once in the schools. The use of non-parametric statistics was required since the data did not follow a normal distribution.

2.4. Technical Challenges during COVID-19 Lockdown: Method for the Adaptation of the Digital Platform

This section explains the challenges faced by the platform during the lockdown in relation to the Cloud architecture structure of the digital platform. In this way, information on one of the objectives of this study to inform the educational community of the interdisciplinary nature of the challenges managed is collected.

The usage of Smile and Learn during the early days of the lockdown due to the COVID-19 pandemic grew significantly due to the different promotional actions taking place and the public realization of the importance that digital learning platforms would gain popularity during the upcoming months.

This growth had a direct impact in the technical resources supporting Smile and Learn. The Learning Analytics platform and database saw a significant increase in the number of requests, which led to a slight slowdown of common processes, although no shutdowns were registered. To improve this situation, vertical scaling of the infrastructure took place, raising the web server instance from a c5.2xlarge to a c5.4xlarge (16 virtual CPUs and 32 GB of RAM), and the database instance from a db.r5.large to a db.r5.xlarge (4 virtual CPUs and 32 GB of RAM). This increase was sufficient to accommodate the new load, and eventually (for the summer season, where usage often decreases), these resources were scaled down again.

A more noticeable issue arose with the landing page, which runs on top of a WordPress CMS and was running in a non-dedicated, inexpensive hosting service. The growth in the number of visits to this landing page led to a shutdown on Sunday, March 15th (the policy regarding the closing of schools took place on March 11th and was announced two days earlier). Because we were running on a non-dedicated hosting, we were not able to set up the infrastructure by ourselves. However, our hosting provider informed us that even though they doubled the resources assigned to our website, it was still failing, and was even affecting the performance of other customers’ websites which were deployed on the same physical server.

As a result of this, we quickly designed a plan to move this landing website to AWS cloud, designing the different components to enhance scalability and adapt costs to the demand. The resulting infrastructure, which is still working as of September 2022, is now described.

The web server containing the WordPress code was loaded in a c5.9xlarge instance (36 virtual cores, 72 GB of RAM). This was an expensive instance, costing $1.728 per hour. Moreover, it was not very satisfactory, since the service was too expensive in low-demand scenarios (such as during the night) and was performing poorly during demand peaks.

To increase performance, we moved the WordPress database to RDS. In particular, we chose Aurora Serverless, which is an Amazon product able to scale out resources automatically based on the current load of the database. Each computing unit, called ACU, costs $0.07 per hour. During low demand scenarios, the database is only running one ACU, reaching up to eight in high demand scenarios.

Although this solved the issue of the server instance performing poorly at times, it was still not the best solution since the instance was expensive, and substantially over-dimensioned for most of the hours of the day. As a result, we started an autoscaling group with a load balancer. This allows AWS to create clones of the web server when demand is high and distribute the requests between these clones, and to destroy them when demand decreases. As a result, the c5.9xlarge instance was converted into a c5.xlarge instance (costing only $0.192 per hour) and introduced into the autoscaling group.

The autoscaling group still features one problem when using WordPress: when resources are uploaded (e.g., images for a blog post), they are only uploaded to one of the servers within the scaling group. To solve this problem, we launched a storage unit in Elastic File System (EFS), which can be mounted into the different servers as one regular shared network drive would be. To speed up the load of the website, only those files within the wp-content folders were loaded into EFS, whereas the others would remain in the server disk. This solution, along with the shared database in RDS, allowed for scalability while allowing contents uploaded in one server to be seen in the others.

Finally, we started to serve the bundles and contents through CloudFront, AWS’s content delivery network (CDN), since they were initially served directly from S3. This allowed for lower latencies due to local caches being present in different regions, as well as a reduced pricing model.

A summary of this new architecture, also including the prices of the most relevant components, is included in Figure 2.

3. Results

This section first analyzes the data on the impact of the COVID-19 situation on the use of the digital platform Smile and Learn, in lockdown and one year later. In this way, information is provided on the increase in the EdTech sector that has taken place. Next, the results obtained through the assessment instrument are presented. Thus, first an evaluation is made of the material and the characteristics of the platform that teachers have been able to use in the lockdown. This is followed by an evaluation of the teachers’ skills in the use of digital resources in general. Educational implications and perspectives are discussed in the discussion section.

3.1. Analysis of the Use and New User Registrations on the Digital Platform

Concerning education centers and schools, there has been an increase in the need for digital material. Figure 3 shows the daily registrations of new users who opted for the school promotional offer as of March 11, when the school closing announcement was made official. Regarding user accounts created for schools with the COVID-19 promotional offer, we highlight the peak observed one week after the lockdown had begun. The number of school accounts starts dropping during the weeks following the lockdown official announcement.

From this growth of new school registrations in the educational platform, the need to seek solutions and adopt online teaching with digital platforms to continue with the education of students is highlighted. The growth observed in lockdown was a specific case. To check whether schools had begun to promote the use of this type of digital resources in schools, the demand by schools after the return to the classroom was analyzed. Figure 4 shows different periods during the school year (at the beginning of the school year, at Christmas, at the beginning of the year, at Easter and at the end of the school year) in which the use of these resources was implemented in schools.

The periods in which we observe an increase in subscriptions and, therefore, an implementation of these resources in the classrooms coincide with the different school trimesters. So, the implementation of the resources in the schools has taken place in periods close to bank holidays. The remarkable increase of new schools in January 2021 is due to the establishment of an institutional agreement with a Spanish regional institution, for which all public schools in said region would gain access to the Smile and Learn platform.

3.2. Preferences for Use of Types of Activities

As mentioned in the introduction, there are different types of activities within this educational digital platform: games, quizzes, theory, stories and videos. In order to understand the users’ preference, the use by type of activity during lockdown and later by the users of the platform has been analyzed. In this way, the percentage of time dedicated to the contents used by the teachers is shown in Figure 5.

It is observed that during the lockdown the preference of use of the activities by the educational centers was centered on the platform games. This use is sustained in a similar way upon returning to the classroom, whereby students invest more time using the platform in the gaming activities. Thus, the content that stands out from the platform or the most selected by teachers to use in their teaching methodologies is centered on the use of game activities. Following the use of games, videos are the second choice for students to practice core curricular content.

3.3. Comparison of Use between Lockdown and Return to Classrooms

For an in-depth analysis of the use made of the digital platform by schools and students, the data recorded on the platform on the time of use grouped into two time slots (8 a.m. to 5 p.m. & 5 p.m. to 12 a.m.) and the total sum of the time of use (in hours); during the months of March, April and May of the 2020 academic year (lockdown) and of the 2021 academic year (presential education) have been collected (Figure 6). The purpose of presenting the time slots is to distinguish between the use made of the platform between school hours and out-of-school hours.

During lockdown, use is fairly evenly distributed between the school time slot and the afternoon time slot, although the time spent from 8 a.m. to 5 p.m. is still higher. On returning to school, this use is again concentrated more on school hours.

The highest use in the 5 p.m. to 12 a.m. time frame may have been due to the use of the platform with the family or while doing homework at times other than school due to the work-life balance that families have had to maintain at lockdown. On returning to the school, the use of the platform during this period is much reduced. This may be due to the fact that teachers do not send homework through the platform as a main resource and it Is used more as teaching material in the classroom.

Using the data obtained for the time of use in hours, a Wilcoxon signed-rank test was performed to compare the use between the months of lockdown and the months corresponding to the year 2021. For the 8 a.m. to 5 p.m. time slot we obtained a significant difference (p = 0.048; W = 2647.5) with a small effect (Rank-Biserial Correlation = 0.238). In the case of the 5 p.m. to 12 a.m. time slot, significance is obtained (p < 0.001; W = 3480) with a very high effect (Rank-Biserial Correlation = 0.963). In this case, the increased use of the platform can be explained by the fact that families have been more involved in the learning process and support of their children. As a result, depending on the situation of the parents, study time slots for the day may be organized accordingly.

Likewise, among the totals of the time of usage of the platform, a significant difference was also obtained between the groups (p < 0.001; W = 4198) with a high effect (Rank-Biserial Correlation = 0.627). This difference between use during lockdown emphasized the importance of these materials in these months. On the other hand, the use in schools may be reduced due to more methodologies to be used by teachers.

3.4. Results of the Teaching Assessment Instrument

The findings of teaching assessment instrument are divided into two parts. First, the evaluation of Smile and Learn Platform by 70 teachers, and then some questions about their training and skills about ICT, which could be interesting for the new online classes due the pandemic.

3.4.1. Evaluation of Smile and Learn Platform

At first, the evaluation of the platform is shown. As it is said, every item has been created with an individual value and the original items were written in Spanish. Therefore, we cannot ensure that they will be statistically correct in English, because a literal translation would not be the best way to evaluate the same constructs.

All these items, as well as some descriptive statistics, are presented at

Table 1. Descriptive statistics about Smile and Learn Platform.

Item	Mean	Standard Deviation	Trend
1. Smile and Learn is a good educational tool	4.56	0.694	5
2. Students like Smile and Learn	4.57	0.693	5
3. The organization within each world/intelligence is intuitive to me	4.17	1.049	5
4. The organization inside each world/intelligence is intuitive to me	4.07	1.044	5
5. The applications (understood as game, quizzes, stories and videos) are easy to use	4.29	1.105	5
6. Games and stories on the platform serve as complementary materials in my classes	4.30	0.823	5
7. Videos on the platform serve me as complementary material in my classes	4.27	0.815	5
8. Generally, I believe that Smile and Learn increases the motivation of my students to learn	4.54	0.736	5
9. I’m aware of the guides of Smile and Learn	4.07	0.960	5
10. (Only if you know didactic guides) I think that they are useful	4.10	1.002	5
11. I know YouTube video tutorials	3.87	0.984	5
12. (Only if you know video tutorials) I think that they are useful	3.91	0.996	5
13. I have recommended Smile and Learn to other teachers	4.22	0.889	5
14. I have recommended Smile and Learn to my school’s families	4.36	0.874	5
15. I will continue to use Smile and Learn for the next courses	4.09	0.935	5
16. I think that in 2025 ICT will be more used than textbooks in schools	3.59	1.110	3
17. I think that in 2025 teachers will create their own material with other pedagogic resources	3.64	1.077	4

. Every item is structured by a Likert scale 1 to 5, 1 being totally disagree and 5 totally agree, meaning that 3 is neutral. It is important to reflect that every item scored more than a 3 on average.

In addition, there were three more items. The first of them asked if the schools where they teach will increase the use of ICT because of COVID-19, or if they will return to the Pre-COVID situation. At this question, 53 answered that their school will increase the implementation of ICT (75.71%) and, on the other hand, 17 think that the use of traditional teaching materials will persist (24.29%).

The other two were open questions where teachers could reflect on what they like about the platform or any other idea about it. There were some tendencies saying that the platform is easy to use, helping to motivate students with a lot of different tools and that their students can learn by playing.

3.4.2. Evaluation of ICT Training and Skills

As the last point, all the main results are shown at

Table 2. Descriptive statistics about teachers training and skills on ICT.

Item	Mean	Standard Deviation	Trend
1. I have experience using Smile and Learn	4.19	0.681	4
2. I have experience using other ICT in my classes	3.93	0.781	4
3. In the last 5 years, I think I have received enough training about ICT in the classrooms	3.70	0.869	3
4. In the last 5 years, I have used ICT in the classrooms	4.41	0.888	5
5. I think teachers are well-trained in the application of ICT	3.33	0.832	4
6. I think that I am well-trained in the application of ICT	3.63	0.967	3
7. I think that the use of ICT in the classroom improves students’ motivation	4.59	0.572	5
8. I think that the use of ICT in the classroom improves students’ learning	4.41	0.694	5
9. In the last 5 years, the school’s resources were enough to implement ICT	3.78	1.121	4
10. In the last 5 years in the school, I faced significant limitations in implementing ICT	3.15	1.350	4
11. I think that after lockdown my school will invest in more resources in ICT	3.81	1.039	4
12. The lack of time is a limiting factor for the use of new ICT in the classroom	3.89	0.934	3
13. The transition to online teaching with new ICT has been easy for me	3.74	0.984	4

. As it happened with results, they are literal translations which means that could not be the best formulation in English. Also, it is a Likert scale from 1 to 5, where 1 means totally disagree and 5 means totally agree, and 3 is neutral.

On the other hand, it was asked how many hours of training they used to take about ICT before lockdown: less than an hour per month (25.90%), 2–4 h per month (40.74%), 5–9 h (22.22%), more than 10 h (11.14%). In addition, it was asked how many hours they devoted to training in ICT after lockdown: less than an hour per month (7.40%), 2–4 h per month (18.51%), 5–9 h (14.81%), and more than 10 h (59.28%), which shows a critical change that should be discussed.

Finally, there were two open questions. First was about limitations of the use of ICTs during lockdown, where teachers agreed that the lack of resources like personal computers and internet connection are the worst problems. The last question was about the future challenges with ICTs in education and, in general, teachers think that the most remarkable problem will be the training of both families and teachers.

4. Discussion

The free subscription model during the lockdown did not include all the customization features of the platform, so this assessment focuses on aspects such as the evaluation of the Smile and Learn material usage. We also wanted to explore the teaching perspective on the general use of the ICT and their training, through this feedback can take new directions when training courses considering these results.

Thus, first of all, the opinions on the digital material of the analyzed platform are discussed. In relation to the qualitative data, high scores close to 5 stand out in the evaluation of the digital material. In this way, the upper scores coincide in assessing it as a good educational platform which students enjoy for learning, intuitive and which the teachers would recommend to use. These scores are in line with comments on the open-ended questions about the digital material developed. In general, the teachers surveyed emphasize that the contents and the motivation that this kind of material promotes for learning in their students are their favorite aspects of the platform. This coincides with studies such as Slussareff et al. [49], Lin et al. [25] or Šimandl and Novotný [12] which highlight this characteristic of engagement. The lowest scores (3.87 and 3.91) were related to the unfamiliarity with the tutorial videos of the Smile and Learn YouTube channel. Thus, this point may have arisen due to the age of the teachers or due to a need on the part of the company to promote this content or conduct training for its use. On the other hand, teachers point to a certain trend of digital change in schools by 2025.

Regarding the points to be improved upon in relation to content, the comments generally agree on the need to expand the number of activities. The increase in the number of digital activities is something that the company carries out with every update of the platform. It also mentioned the difficulty of using the platform by young children with the computer when they have to use the mouse or keyboard. That point is aligned with Revelle’s study [50] which comments that tablets or devices that allow greater interaction are easier to use for young children. It should be noted that, due to the limitations of the types of devices, the Smile and Learn platform is multi-device, that is, it can be used on computer, tablet, mobile and digital blackboards. The multi-device options are intended to reduce the gap in devices available in schools [12], which may have been affected at home during COVID-19 lockdown [44,47].

In the assessment instrument on ICT experience and skills, teachers show a positive attitude towards the use of technological resources. According to Uluyol and Şahin [20] and Krejins et al. [19], teachers who show encouragement and positive attitude about the use of digital materials will be more willing to use them and learn about their application. Likewise, they emphasize that these materials could help their students to be motivated and learn [13,24]. This could also be linked to the fact that most of the time spent on the platform’s digital materials is focused on game activities (Section 3.2). Increased use of game-based activities may be due to the fact that these activities are more motivating for learning [25,49]. However, they have found limitations in schools for the implementation of these resources and consider that more training is needed [12,31,33]. Interestingly, despite the limitations experienced by the resources or management of the pandemic [12,15,21], the attitude of teachers has promoted the personal assessment of their processes in the use of online methodologies and the way this has impacted student learning [14,15,30]. Such motivation to know the impact has also been reflected in the interest to evaluate the situation created by COVID-19, and how this use of methodologies has been perceived by the students, as discussed in the introduction (e.g., [14,15]).

Among the limitations emphasized at lockdown, teachers have commented: “Difficulty in conciliating work and family life”, “Students are not prepared”, “Teachers and parents’ lack of knowledge of online tools”, “Time”, “Lack of devices in families or by students”. “The difficulty of covering all the needs of students”. These coincide with the limitations commented on in Section 3.3. Thus, it is still necessary to address many of these limitations as commented on in the studies of Šimandl and Novotný [12] or Baek [16], or with the limitations that have also been reported in other studies in the Spanish context (e.g., [14,15,31,33]). On the other hand, our research also asks about the teachers’ perspective in relation to the future challenges of education with new technologies. The most common answers were the need for training, data protection, accessibility, adaptation to all educational needs and combination with traditional learning. One teacher underlines the need to work in the uncertainty of the future, which would require the acquisition of the skills or competencies of the 21st century. Thus, this study coincides with previous studies that also highlight the need to address these points in order to be able to make future improvements [12,13].

Upon returning to schools, it has been observed that a high demand of digital resources remains for the training of students. In addition, after the challenges overcome by teachers during the pandemic, different perspectives have been taken into consideration, such as the assessment of educational practices and the combination of hybrid models for teaching that allow the necessary flexibility to meet the learning objectives of students [14,30,46].

Although the perceptions and context of the students have been studied mainly in the university environment or secondary education in the Spanish population, it is interesting to consider the contributions of these references to further improve teaching in all sectors or groups involved in teaching [14,15]. In this way, digital material could also be improved at the business level by assessing the perception of students regarding the use of these resources.

5. Conclusions

The use of activities and the number of Smile and Learn users has increased owing to school closings and the lockdown of the families. This fact has brought about a greater demand for digital resources on behalf of teachers to continue their classes with their students. In a similar manner, this use of digital materials has continued during the return to school. On this occasion, it was more limited to the school hours and with a lower use than observed during the pandemic. Thus, during lockdown, the online methodologies that teachers could use were more limited in resources, and different methodologies for the use of these materials in schools were used again. In reference to the selected learning activities, there is a preference for game-based activities. Moreover, it would be interesting to ask ourselves whether or not there has been a change in the schedules of the students, and how such change has affected family life.

Additionally, it is worth mentioning that the COVID-19 pandemic brought significant technical challenges that needed to be faced to adapt the service to the high levels of demand [9,10]. This required some effort to scale up the cloud resources and a major redesign of the infrastructure involving the landing page, which was moved from a regular inexpensive hosting to a cloud infrastructure involving an autoscaling group with a load balancer, as well as a serverless database.

Based on the impact that COVID-19 has had on the EdTech sector [10], it is proposed to continue the analysis of digital materials. One approach that could provide solutions to the stated need for learner-centered education [15] may be to complement the design flow with research in the area of user experience. In this way, improvements and developments of these materials would be focused on the needs of the users [51]. Based on previous studies on the analysis of online methodologies and assessment of these digital resources such as Lee’s work [34], future work can focus on improving the weaknesses and limitations identified during the pandemic. Therefore, it would contribute to the SDG goals [44,45].

Concerning the section of the assessment instrument that evaluates the Smile and Learn material, it is concluded that these materials are motivating for students and useful for teachers. In fact, the request to continue increasing the material developed by the company stands out as an aspect of improvement. However, the dissemination of some of the company’s resources, such as the tutorials on the YouTube channel, should be improved. In the section of the assessment instrument on teacher training in the use of digital resources it can be highlighted that the attitude of teachers is positive towards the use of digital materials. However, they believe that there is a need to improve the training they receive as well as the resource supply of schools.

Another future study to consider would involve the resources available to families apace with the use of this material by students. This way, we could direct our approach towards the needs of the school community (families, teachers and students) in relation to the use of technology and digital educational material. Likewise, this research should be taken one step further and explore the training of the teachers who used Smile and Learn materials, along with the methodology they followed. Furthermore, it should be noted that the personalization of content makes more sense when the teacher is not present, but is lacking in that adaptation. The digital material increases the possibility of tracking learning to boost the areas where students are challenged. This task can promote the continuous assessment that teachers often use.

In summary, COVID-19 has been a challenge for different sectors in the educational field and has emphasized the need for online methodologies, although the whole process of design and use of these products can still be improved. Having information accessible to stakeholders on the perceptions and challenges in this interdisciplinary work can help contribute to not remain in the present moment and to further work for the education of future generations.