1. Introduction
Students enrolling in undergraduate degrees now are unlike any generation that has preceded them. The population cohort commonly known as Generation Z (Gen Z) were born between 1995 and 2012 [
1], and this group currently forms approximately 60% of the undergraduate university students in the UK [
2]. They were born into a world where Google and
Wikipedia already existed. They have never known a life without computers, or the internet, and do not see these technologies as tools but as an integral part of their everyday existence [
3]. They do not even view their mobile phones as being luxury items of technology, but instead consider them to be an essential part of everyday life [
4]. These students are at home with the concept of social media, and use platforms such as Snapchat and Instagram for the majority of their communication. In fact, it is a generation that has garnered significant attention for its uniqueness in comparison with previous generations [
3]. Together, these factors have a direct impact upon how we market the benefits and opportunities of higher education to them.
Simultaneously, university internationalization has become an important investment for education institutions in recent decades [
5,
6], and this has seen the higher education sector become an active player in the global marketplace [
7]. Whilst this increased number of overseas students in the UK brings much added value to ongoing issues, such as classroom diversity and tuition fee income [
7,
8], it further challenges the operational ‘norms’ of pedagogy, with student demands for more technological interaction and experiential learning becoming more common.
Accordingly, the digitization of education practice has been a key focus for the strategic goals of the UK higher education sector body known as Advance HE (and prior to this, the Higher Education Academy) for many years. Concerted effort to increase academic adoption and integration of digital, and social media, technologies to support contemporary pedagogic practice has been evident across the sector [
3]. However, whilst the landscape of higher education has embraced advancements in technology-enhanced learning (TEL), to meet the needs of the 21st-century learner, students from this generation themselves were certainly not prepared to, or expecting to, experience their undergraduate degrees entirely through online learning, which has been the primary resulting impact for them of undertaking higher education during a pandemic which forced university campuses in the UK to close for approximately 18 months.
The shift from didactic face-to-face time with students to a complete program of online learning happened literally overnight for the higher education sector, as social distancing restrictions were put into place by the government, and new terminologies, such as ‘lockdown’ and ‘the tier system’, became the norm. Educators duly faced a number of social, pedagogical, and technological challenges in this new era of learning remotely through online teaching [
9]. As a result, educational institutions continue to experience unprecedented times as they navigate students’ expectations with a range of TEL (technology enhanced learning) tools available, while balancing this with the reality of their own institutions’ ability to deliver a high-quality educational experience for students.
Given this complicated background, this research explores the impact of undergraduate student learning during the enforced move to online teaching during the global pandemic. Through the application of a pre-existing learning gain model [
10], this research explores whether the online teaching approach used has met the educational goals identified, to ensure that each student’s skill gaps have still been met, or whether the students are reporting a drop in their learning during this period as a result of being taught online.
This paper considers online teaching pedagogies and the potential role of student learning gain as an indicator of perceived teaching quality. The methods applied to this study are detailed, followed by the presentation of the results obtained. These results are then discussed in detail, and conclusions made regarding the reported differences in learning between in-person (pre-COVID-19 pandemic) teaching and subsequent online (during COVID-19 pandemic) teaching. Finally, limitations to this research study are acknowledged with recommendations for how the work can be further extended in the future.
2. Online Teaching Pedagogies
Takavarasha [
11] asserts that technology is now an integral part of everyone’s lives. The 21st-century consumer is a digitally ‘savvy’ choice maker [
12] whose expectations, motives, and experiences are socially constructed [
13]. This is evident in higher education, where we witness innovative technology-enhanced teaching and learning models, often driven by student demands for bespoke learning [
14]. Technology can positively impact student learning by offering a personalized and interactive experience, thereby enabling students to take a deeper responsibility for their educational journey [
15]. Clearly, successful teaching strategies can minimize any resulting learning anxiety [
16]. Furthermore, these creative approaches can be adopted by higher education institutions in an attempt to build synergy with the latest trends in educational technologies [
17]. The authors acknowledge that shifts in education, and the introduction of technology-enhanced learning, were ongoing before the arrival of the COVID-19 pandemic. However, the campus closures due the pandemic acted as a catalyst for demanding rapid change which might otherwise have taken a decade or more to achieve.
As a sector, we have therefore seen a shift from didactic teaching approaches in higher education to embracing creative pedagogies centred on enhancing the student learning experience [
18], with tactics such as flipped classroom (students receive teaching materials beforehand so that subsequent lecture time can be used to reinforce learning) and gamification (the use of game mechanics, such as points scoring and league tables within teaching to increase student engagement) now more commonly applied. This approach complements the characteristics of the modern-day student, who demands speed, nonlinear processing and social learning [
19], collaboration [
20], and multitasking [
21].
3. Learning Gain
With the ever-increasing costs of higher education tuition fees [
22,
23,
24], and an opposition to student loan funding support [
25], the value of higher education is often questioned by students [
26,
27] and there is growing pressure to ensure the availability of relevant metrics to support the ongoing marketization agenda [
6]. As a result, educators must consider the value and impact of their own teaching [
28,
29,
30,
31], whether it is delivered face-to-face, online, or as a hybrid of both.
The landscape of higher education has therefore experienced a shift which has seen more measurement and assessment of student learning outcomes. This reliance upon metrics forms part of a performativity agenda [
32]. Within such a metric-driven environment, the learning gains of students are now considered to be key indicators of teaching and learning excellence [
33], and whilst other metrics may only feed into the cynicism discourse (moving to a consumerist model), and the pragmatism discourse (demonstrating value to stakeholders) of teaching excellence, evaluating learning gain both supports these and also supports the aspirationalism discourse relating to the integrity of teaching enhancement undertaken [
34].
A recent office for students study concluded that existing methods for determining student learning gain need more development to enable them to account for contextual and subject factors [
35,
36]. Andrade [
37] proposes that the actual definition of learning gain should be for individual higher education institutions to determine, but Arico et al. assert that ‘learning gain is now prominent when considering the effectiveness of higher education’ [
38] (p. 249). As such, our understanding of student learning gain should help us to understand how we can maximize the effectiveness of our own teaching [
29], and how the value of our teaching may have changed with the transition from face-to-face to online teaching.
Taking these considerations into account, this study has used an alternative learning gain model first proposed by Polkinghorne et al. [
39], and subsequently detailed by Polkinghorne et al. [
40], which integrates both student explicit knowledge (subject learning that can be codified and verbalized as theory) and tacit understanding (experience and practical application of knowledge) to create a unique two-dimensional evaluation of student learning. Therefore, the model is suited perfectly for holistically assessing learning outcomes, and for informing the continuous improvement process [
41] to ensure that the quality of teaching being delivered, and the resulting student learning experience, can be understood, and then improved upon year on year.
As discussed above, the academic year 2020/2021 was significantly impacted by the global COVID-19 pandemic. With university campuses being closed and teaching cancelled or moved online, the sense of what being a student actually is was forced to evolve. Without doubt, such radical changes will have impacted the learning experience of students. The general consensus is that this impact will have been negative in nature, but how can we know this for sure, and how can we obtain a sense of the scale of this impact? In 2018, and again in 2019, the authors evaluated the learning gain of undergraduate marketing students studying on the first year of a business studies degree course at Bournemouth University [
39].
Using the model of learning gain described above, it was possible to gather an overview of how the learning on the course had evolved by 2019, following changes to teaching made at the end of 2018. The authors considered if this study was to be repeated, would it be possible to view the learning of students on the course delivered during the COVID-19 pandemic, and if so, how will this learning compare to the pre-COVID-19 pandemic learning of 2019? These questions will now be investigated in this paper to see whether a sense of scale for the change in learning can be obtained, and to identify what direction that change in learning represents.
4. Methods
This paper reports on a mono-method research study considering primary data collection supported by secondary research.
The strategy used for reviewing existing academic literature was archival. The specific archival databases accessed included Academic Search Complete, British Library EThOS, Directory of Open Access Journals, Education Source, ScienceDirect, Supplemental Index, and the Teacher Reference Center. Additional governmental sources were also considered. Due to the changing nature of higher education, priority was given to relevant peer-reviewed papers published since 2017. The search string terms used included higher education, university, learning gain, and student.
This study collected data based upon the views and opinions of participants. The collection of the primary data utilized self-reflective surveys, ensuring that the data were ordinal (ranked). Questions were asked to test the attitude of participants based upon the intended learning outcomes of the taught university module being studied (
Table 1).
Face and discriminant validity checks [
42,
43] were undertaken to ensure that the question constructs had clear distinction. The question response options and codes were designed to indicate how students perceived that their own learning had changed from studying the module, and were based upon a unipolar scale in which 0 = no change; 1 = minor improvement; 2 = moderate improvement; 3 = significant improvement; and 4 = exceptional improvement in perceived student learning.
To enable comparison with the previous data collected in 2019, participants for this study were once again level 4 (first-year undergraduate) university students at Bournemouth University studying a marketing module delivered as part of a management-based business studies degree course. This is a longitudinal study. Data collection was undertaken using the JISC online survey platform, and was based upon a self-selection purposive strategy from a heterogeneous student group with the defining characteristic of gender.
In both cases, no additional data relating to sociodemographic characteristics were captured.
5. Results
As detailed in
Table A1, data were originally collected in April 2019 from a previous cohort of students studying the marketing module. Data were collected using the online data collection tool, and the data collection itself was undertaken during an in-person taught lecture. There were a total of 47 participating students who represented 15 male students (32%) and 32 female students (68%). These were the pre-COVID-19 benchmark data that would be used for comparison with the COVID-19 period data collected during the subsequent academic year 2020/2021.
As detailed in
Table A2, new data were then collected in April 2021 representing a subsequent cohort of students studying the same marketing module. Once again, data were collected using the online data collection tool, and the data collection itself was undertaken during an online taught lecture. There were a total of 65 participating students in this group who represented 40 male students (62%) and 25 female students (38%). These were the COVID-19 period data which would be used to identify changes in learning experience when compared with the pre-COVID-19 data collected during 2019. All of the students participating in this research fall into the definition of the Generation Z age bracket.
In both data collection periods, the results obtained were divided into the following three categorical groups:
Positive Group 1—Combined positive responses for moderate, significant, and exceptional improvements in learning (labelled M + S + E);
Positive Group 2—Combined positive responses for significant and exceptional improvements in learning (labelled S + E);
Positive Group 3—Positive responses for exceptional improvement in learning only (labelled E).
In each case, the 2021 data collected were compared with the 2019 data previously collected to ascertain whether there was an increase, or a decrease, in the percentage of students reporting their learning within Positive Group 1, Positive Group 2, and/or Positive Group 3. The data from the three categorical groups were analysed separately to avoid skewing the results, and then compared against each other to provide an increasingly granular understanding of the student responses being reported. Using an alpha level of 5%, separate chi-squared tests for Positive Group 1 (degrees of freedom = 4), Positive Group 2 (degrees of freedom = 3), and Positive Group 3 (degrees of freedom = 2) were undertaken to check for differences between the expected frequency responses represented by the 2019 data and the observed frequency responses of the 2021 data.
6. Discussion
Considering the overall learning gain data collected (
Table 2), we can see that there has been a reduction in the percentage of students reporting results in Positive Group 1 (M + S + E) during the COVID-19 period, with this decrease ranging from 3% for Q2 (marketing practice) to 13% for Q5 (marketing problems). The two exceptions to this trend are Q6 (marketing solutions) with an increase during the COVID-19 period of 7% and Q3 (marketing environment) with an increase of 9%.
Positive Group 2 (S + E) overall data responses reflected a similar downturn in learning being reported during the COVID-19 period of up to 14% in the case of Q5 (marketing problems), which interestingly included Q3 (marketing environment) and Q6 (marketing solutions), which had both previously reported an increase in learning in Positive Group 1, but now report a decrease in learning in Positive Group 2. It should be noted that there is no change in learning being reported for Q7 (transferable skills).
In the case of Positive Group 3 (E), there is similarly no change in reported learning for Q7 (transferable skills). Q3 (marketing environment) once again reports a small increase in learning of 1%, but for the remaining six question areas, a downturn in learning is being reported of up to 9% in the case of Q1 (marketing principles).
Table 3 provides a useful numerical comparison of the 2019 and 2021 data collection, and a graphical comparison is described in
Figure 1 from which the changes in response patterns across the years for the various questions become more distinct. For example, apart from Q3 (+1%) and Q7 (0%), the data reveal a drop in the number of students reporting exceptional improvement in their perceived learning, with a decrease in learning of up to 9% being reported. This trend is also reflected in the significant improvement category in which a general decrease in learning is reported ranging from 1% up to 9%, the exception to this being Q1, which reports a 2% increase in reported learning.
The more interesting results appear in the moderate improvement category, in which, with the exception of a decrease in learning reported of 6% for Q7 and no change in reported learning for Q5, the results otherwise indicate a surprising increase in learning across the board ranging from +1% (Q1) to +16% for Q3 and +17% for Q6. Interpreting this unexpected development, there would appear to be a consolidation of reported learning around the moderate improvement category; that is, in comparison with the 2019 pre-COVID-19 data, the 2021 COVID-19 period data indicate a reduction in both the minor improvement and significant improvement categories, with the consequence that more students are reporting a moderate improvement in their perceived learning.
Separating out the overall data into results for female students (
Table 4) and then for male students (
Table 5) reveals interesting local variations based upon gender. Considering female students first, whilst the overall Positive Group 1 responses are largely negative with decreases in learning of up to 20% being reported for Q7 (transferable skills) and 19% for Q8 (marketing techniques), for Q3 (marketing environment) at +8% and Q6 (marketing solutions) at +5%, increases in learning are being reported during the COVID-19 period. These increases do not appear to map directly to the Positive Group 2, since although Q5 (marketing problems) indicates a 6% increase in learning, there is no corresponding increase in reported learning for Q3 and Q6. However, Q3 and Q6 do feature again in the results for Positive Group 1 with increases in learning of +6% for Q3 and +5% for Q6, respectively.
Considering male student response data in Positive Group 1, an increase in learning was reported for Q3 (marketing environment) of +13% and for Q6 (marketing solutions) of +10%. Q2 (marketing practice), Q7 (transferable skills), and Q8 (marketing techniques) all reported an increase in learning of +1%. Conversely, reductions in learning were reported for Q1 (marketing principles) and Q4/Q5 (marketing problems) of up to 22%. In the case of Positive Group 2, the only increase in learning being reported is for Q7 with an increase of 15%, and in Positive Group 3, there are no increases in learning being reported, and the decrease in learning varies from small amounts of up to 4% (Q3, Q4, and Q7) to much larger changes of up to 20% (Q1, Q2, Q5, Q6, and Q8).
Analysis of the data with respect to distance travelled (DT) and journey travelled (JT) reveals additional patterns and trends (
Table 6). Overall, the data indicates a drop in both distance travelled and journey travelled for Positive Groups 1, 2, and 3, with there being little difference in the decrease in reported learning between the three groups. In terms of Positive Groups 1 and 2, male students reported a lower decrease in journey travelled (up to 4%) learning compared with that of distance travelled (up to 12%). However, for Positive Group 3, male students reported the same 10% reduction in their perceived learning for both distance and journey travelled. In the case of female-student-orientated data, for Positive Group 3, there is no change in reported perceived learning in either the distance travelled or the journey travelled groups. Interestingly, for Positive Groups 1 and 2, a significant decrease in perceived learning occurs for journey travelled (up to 10%), but much less so for distance travelled, and with no change in learning being reported at all for distance travelled in Positive Group 1.
As described in
Figure 2, which is for Positive Group 2 results only, male students reported that distance travelled (explicit knowledge) learning was most affected for them, whereas female students reported that journey travelled (practical experience) learning was most affected.
In terms of Positive Group 2, there is a wide variation in the learning being reported when analysed by gender in respect of each individual question. Male students reported reductions in their perceived learning for Q2 (marketing practice), Q3 (marketing environment), Q4/Q5 (marketing problems), and Q8 (marketing techniques) of up to 18%, but for Q1 (marketing principles) and Q6 (marketing solutions), there was a negligible change in the reported learning. In fact, for Q7 (transferable skills), a 15% improvement in learning was reported. Female students, however, reported reductions in their perceived learning for Q1 (marketing principles), Q6 (marketing solutions), Q7 (transferable skills), and Q8 (marketing techniques) of up to 13%. For Q2 (marketing practice), Q3 (marketing environment), and Q4 (marketing problems), there was a negligible change in the reported learning, and for Q5 (marketing problems), a 6% improvement in learning was reported.
7. Conclusions
This research was undertaken in response to the need to understand students’ learning gain during the period of online teaching forced upon higher education in the UK due to the global COVID-19 pandemic of 2020/2021. It was fortuitous that the research team had already started collecting learning gain data before the pandemic, which could then be used in this study as a benchmark for comparison with the data collected during the pandemic. Without this happenstance, this study would not have been possible.
This research study is based upon 2-year groups studying the same module at the undergraduate level, the first-year group being prior to the COVID-19 pandemic (2019) and the second-year group being during the COVID-19 pandemic (2021) at which time there was considerable disruption to higher education provision in the UK as campus closure was enforced, and students were mainly studying from home. With no planned changes to the curriculum and teaching schedule during this period, the only variations that occurred were due to online delivery of the teaching itself and, of course, the fact that no two-year groups are ever exactly alike. Some natural variation between year groups is therefore always present.
In summary, the overall picture would suggest that the students’ own perception of their learning during the COVID-19 pandemic in 2021 was lower than that reported by the previous cohort of students in 2019 who were studying the same topic pre-COVID-19. This difference may be related to teaching online, but may also be related to a range of external factors, such as access to reliable technology, being able to create a suitable learning environment at home, poor health due to the pandemic, and/or mental stress caused by continued isolation.
Of particular interest is that the learning gain model used enabled the academic team to appreciate, for the first time, that in the case of the female students, the practical side of their education was more affected (journey travelled) by the change to online teaching, whereas the male students reported that the more theoretical side of their learning was more affected (distance travelled). Such gender variations need now to be confirmed by further studies, but if true, this may have significant implications for how we teach online to ensure equity and inclusion. Such a potential differentiation in learning needs to be accounted for, even in the hybrid models of university teaching currently being proposed across global higher education as we start to move through, and past, the restrictions put in place during the COVID-19 pandemic.
8. Limitations of the Research
This research only considered one taught module and compares 2-year groups on a single course studying at one university. Whilst the results are interesting and informative, the population size is too small to make generalizations from, and a wider, more expansive study is recommended to explore the implications and potential value across discipline areas and teaching delivery methods.
Furthermore, it should be noted that the gender split between the two sample groups varied with the percentage being female dropping from 68% in 2019 to 38% in 2021. Whilst this will not have any impact upon the gender-based analysis, given the differences in gender learning experiences being revealed, this change in gender split may have influenced the overall results obtained.
In addition, there are several potential equity groups within the student population of which gender is one that has been identified, but other relevant ones may include differences in learning styles and/or social, demographic, or economic background. No data on these additional dimensions were available to facilitate comparison in this study, but it is the recommendation of this research that any future studies should include them.
Due to the impromptu nature of this research study, data relating to relevant sociodemographic characteristics like this, which might have otherwise been captured in the 2019 data, were not captured. Consequently, there was limited value in collecting such data from the 2021 data as no direct comparison was possible. However, repeating the study in the future, and capturing such data, would considerably help our understanding of how student learning may vary between in-person and online delivery methods within a modern university environment.