Technology's impact on language learning: Meta-analysis on variables and effectiveness

ABSTRACT


INTRODUCTION
Global development has been significantly influenced by technology, particularly in education.The advancement of technology in the field of education has manifested in the implementation of more sophisticated learning facilities.The accessibility of technology has aided educators in enhancing their instructional quality.The statement has been delivered by Keengwe and Georgina [1] as they argue that technological advancement has brought about significant changes in teaching and learning practices.Information technology can be accepted as a medium for conducting the educational process, including aiding the teaching-learning process, which includes reference searching and information sourcing [2].
Recent science and technological development has significantly influenced the domains such as economy, society, politics, law, art, culture, and even education [3].In education, especially in language learning, the technology significantly influences the success of the language learning process J Edu & Learn ISSN: 2089-9823  Technology's impact on language learning: Meta-analysis on variables and … (Syarief Fajaruddin)

513
and results [4]- [6].The findings from a Hussain [6] study show that Information and Communication Technologies (ICT)-based learning can improve vocabulary mastery.Similarly, according to Enayati and Gilakjani [5], using computer-assisted language learning (CALL) can improve vocabulary mastery compared to the traditional method.Departing from the results of these studies, it is clear that technology can improve language learning.Several meta-analysis studies have been conducted concerning the association between technology and language learning.Zhao [7] has studied the most recent technological and language-learning advancements.In his research, he utilized journal articles published between 1997 and 2001.According to the results of his study, the literature on the effectiveness of technology use in language education is limited in four ways: the small number of systematic empirical evaluation studies, the limitation of the study to higher education and adult learners, the limitation to the general foreign language and English, and the short-term experiment with one or two learning aspects (such as grammar or vocabulary).
Grgurović et al. [8] conducted a study on the effectiveness of computer-supported language learning.The researchers utilize 37 studies published between 1970 and 2006 in academic journals in this study.These 37 articles come from three electronic databases: linguistics and language behavior abstracts (LLBA), education resources information center (IRIC), and dissertation abstracts (DA), as well as six periodicals: CALL, System, computer assisted language instruction consortium (CALICO) Journal, ReCALL, language learning and technology (LL&T), and TESOL quarterly.
On the contrary, [9]- [12] have conducted a meta-study by analyzing the influence of cellular gadgets on language learning.The study by [11] uses nine journal articles, conference proceedings, and doctoral dissertations published between 2008 and 2018.The results of their study show that the use of the cellular phone for language learning is more effective than the conventional method.Then, another study by Cho et al. [12] used 20 studies from electronic databases such as education resources information center (ERIC), EBSCOhost (academic research complete), PsycINFO, journal storage (JSTOR), and ProQuest Dissertations and Theses from 2005 until 2017.The results of their study show that the overall size effect is 0.51.Next, Taj et al. [10] study utilizes thirteen studies published between 2008 and 2015 from the electronic databases ERIC, digital libraries of the University of Jeddah, digital libraries from the University of Malaysia Pahang, and Google Scholar [10].The findings of their investigation indicate that mobile assisted language learning (MALL) has promoted English as a foreign language (EFL) instruction.Their study's aggregate size effects (d = 0.8) are the largest of all.In the meantime, Sung et al. [9] utilized 44 journal articles and doctoral dissertations published between 1993 and 2003.Their study reveals a moderate effect size of 0.55 regarding the use of mobile devices in language acquisition.These meta-analysis investigations were conducted using out-of-date sources from 1970 to 2017.In order to conduct further research on the relationship between the use of technology for language learning and the effect of potential moderating variables that influence the strength of the relationship, it is necessary to analyze the overall impact of the most recent technology on language learning.

METHOD 2.1. Literature search
The research was conducted by searching the Scopus database for articles addressing the topic of technology-based language learning published in scientific or academic journals as of March 3, 2021.Utilizing the keywords "technology-based and language learning" and "the effect of technology on language learning," a literature review was conducted.To ensure the robustness and reliability of our meta-analysis, strict inclusion and exclusion criteria were applied to the selection of studies.These criteria were established a priori to maintain consistency and minimize bias in the selection process.Table 1 provides a detailed overview of the inclusion and exclusion criteria.
Been studying has collected 291 documents.29 out of 291 documents have been collected since researchers imposed the following restrictions on these documents: i) the documents are from open access; ii) the documents are from the period of 2012-2021 (specifically on May 2021 since 201 has not ended yet so that it becomes possible the number of indexed articles may expand after May; iii) the documents are of the social subject; iv) the documents are of article type; v) the documents are in the final publication stage; vi) the documents are from the journal sources; and vii) the documents are in English.Afterwards, the researchers search the articles by using the keyword "the effect of technology on language learning."The results of the search were returned in 1,673 documents.With similar limitations, the researchers have gathered 85 documents.
Several screening criteria will be applied to the documents obtained through the literature search.First, the studies should assess the correlation between technology use and language learning and report the measurement outcomes.In addition, the investigations should report the relationship between the variables.Consequently, studies that only report the outcomes of technology use or language acquisition will be excluded.Similarly, studies that do not report the relationship between the variables will also be excluded.Second, the studies should explicitly report the sample size used.Consequently, the studies that do not report the sample size will be excluded.Third, the studies should report the technique for measuring the technology use in language learning.Hence, the studies that only report the association between technology use and language learning with a clear description of how the technology used in language learning has been measured will be excluded.Fourth, each study should report Pearson's r correlation.Thus, the studies that only report the regression model analysis reports, or the multilevel regression model analysis reports will be excluded.Fifth, the studies should use English.Sixth, the grade level of participants consists of elementary school until college level.Therefore, the studies in which participants are from the preschool level will be excluded.
The number of documents that meets the inclusion criteria is 19 main studies as shown in Figure 1.From the 19 main studies, the researchers have found 43 independent samples that will be analyzed.This figure appears to the surface because the 43 independent samples from the 19 main studies report proficiency of more than one language aspect [13]− [20].Meanwhile, other studies have only reported on one aspect of language proficiency [5], [6], [21]− [29].Technology's impact on language learning: Meta-analysis on variables and … (Syarief Fajaruddin) 515

Included samples
The 19 primary studies for our meta-analysis originated from a reputable database, as detailed in Table 2.These studies were selected to provide a comprehensive overview of the diversity of research conducted in the field of technology-based language acquisition.The inclusion criteria ensured a wide range of research contexts and methodologies, which contributed to the robustness of our analysis.To ensure the relevance and novelty of our findings within the pertinent time frame, our literature analysis covered studies published from 2012 to 2021.This allowed us to capture recent developments in the field of technologyassisted language learning.Abdulrahman et al. [22] International Journal of Language Education Universitas Negeri Makassar 3.
Mellati and Khademi [29] International Journal of Applied Linguistics and English Literature Australian International Academic Centre 5.
Farooq et al. [24] English Language Teaching Canadian Center of Science and Education 8.
Schechter et al. [13] Computers in the Schools Routledge: Taylor and Francis Group 9.
Kurt [16] Elementary Education Online Ankara University Faculty of Education Department Primary Education 12.
Kashani et al. [17] English Language Teaching Canadian Center of Science and Education 14.
Yeşilbağ and Korkmaz [18] Education and Information Technologies Springer BavaHarji et al. [20] English Language Teaching Canadian Center of Science and Education 18.
Alemi [27] International Education Studies Canadian Center of Science and Education 19.

Data analysis − Effect size calculations
For the correlation study, the Pearson coefficient of correlation (r) refers to the index of effect size from a study [30]- [33].If the individual studies are statistically significant, then the resulting effect size is assumed to use using a p-value of 0.05.In order to ensure the stabilization of the sample distribution, the r should be transformed into the Fisher's z transformation [34], [35].The formula that will be used for transforming r to z can be viewed in (1): After the mean effect size and the trust interval have been attained, these values are transformed into r.In order to calculate the mean effect, the researchers use the random effect model.The random effect model is selected because the researchers assume that the effect size of the different studies probably comes from the diverse populations, and the diverse populations have their respective sample distribution [35], [36].The diversity of the experimental settings from each study (grade level, country or region, and gender) will be fitter in the analysis if the random effect model is used [37].In addition, the meta-study analysis data analysis will use Jeffrey's amazing statistics program (JASP) free software.

Moderator analyses
In order to identify the variation among the results of the different studies, the researchers also conduct the heterogeneity test (Q test).If the Q statistics are significant, it can be concluded that each study does not come from the common population.In other words, the significant Q statistics show that the mean effect size of each component in the moderating variable is significantly different, and, therefore, the moderating variable analysis can be potentially conducted.In the study, the five moderating variables (grade level, region, measure of self-control, measure of academic achievement, and sample size) are analyzed using the analysis of variance (ANOVA)-like models.For the ANOVA-like models, the researchers report the within-group effect means (weighted r), the 95% confidence intervals (CI), the within-group variability (Qw), and the between-group heterogeneity (Qb).The significant Qb statistics show that the mean effect size between the components in the moderating variables is significantly different.
Next, the moderating variable publication year is analyzed by using Pearson's correlation test.The analysis is conducted in order to identify the association between the publication year and the effect size.The researchers report the coefficient of correlation r and the 95% confidence intervals within the analysis.The significant r correlation shows a significant relationship between the publication year and the effect size.All the moderating variables are analyzed using assistance from the JASP free software.

Evaluation of publication bias
The meta-analysis study uses three approaches in exploring the publication bias: funnel plot, Egger's test, and fail-safe N. The funnel plot is used to display all effect sizes clearly.If the pattern that has been shaped is symmetrical, then the pattern will indicate that there is not any biased publication [36].Then, Egger's test is a linear regression method used to test the symmetrically of the funnel plot [38].The fail-safe N estimates the number of studies with statistically insignificant results (unpublished data) required for the mean effect size to become statistically insignificant [39].

Results
The researchers collected data from 19 main studies (scientific articles) that met the predefined inclusion criteria.In total, 43 independent samples were obtained, and these samples became the objects of our meta-analysis.Table 3 (see in Appendix) presents key information about these studies, including year of publication, effect size (g), standard error (SE), publication type, sample size, and grade level.Table 4 (see in Appendix) provides information on the geography and technology categories utilized in language-learning studies.It is noteworthy that a majority of these studies were conducted in Turkey (20.93%), and online media (37.21%) was found to be the most extensively used technology category for language-learning activities.Additionally, Table 5 (see in Appendix) illustrates the moderating variables analyzed in our study.These variables encompass a range of factors, including level of education, location of research, measurement of language proficiency, and type of technology used.

Findings from the main analysis
The main findings from the meta-analysis study are displayed in Table 6.The analysis results that have been conducted using the random effect model show that the mean effect size from the 43 studies is 0.562 (p < 0.001), with the interval degree of trust 95% from 0.418 until 0.706.These results imply that there has been a significant impact from the use of technology on language learning activities in comparison to traditional learning (the learning process without the use of technology).The size effect values of 0.80, 0.50, and 0.20 represent the big, moderate, and small sizes [40].Thereby, it can be concluded that the effect of technology on language learning belongs to the "Moderate" category compared to traditional learning.
The results of the heterogeneity test as show in Table 6, that the effect size of the 43 independent samples has been heterogenous (Q = 138.055,df = 42, p < 0.001).These results show that the inter-effect size variance used in the study has been very varied.Thus, these results imply the need to analyze the moderating variable to identify each moderating variable's contribution toward the inter-effect size variance differences.
Figure 1 displays the forest plot of 43 studies analyzed using the random effect model.In the forest plot, the effect size of each study is symbolized by square dots, while the horizontal line on both sides of the squared dot displays the estimates of trust interval.The forest plot shows that the effect size of the 43 studies has been quite varied, with the smallest effect size of -0.45 and the largest effect size is 2.90.Several studies have negative effect sizes, such as [15], the second study by Kashani et al. [17], the fifth study by Yeşilbağ and Korkmaz [18], and the first study by Alfaleh [28].These studies indicate that the implementation of technology in language learning activities within these studies has been proven ineffective compared to traditional language learning activities.In addition, several studies have also shown insignificant size effects [21], [22], [24].These findings indicate that several studies confirm that the impact of technology use on language learning activities is not significantly different from that of traditional language learning.However, in general, it can be seen in the forest plot that most of the studies that have been analyzed have high and significant side effects [14], [23], [29].These studies indicate that using technology in language learning is more effective than traditional learning.

Moderating variable analysis degree
The moderating variable degree consists of four categories: elementary school, junior high school, senior high school, and university.The results of the moderating variable analysis as show in Table 6, that the effect size means the score of the four degrees is not significantly different from one to another (Qb = 5.048; p = 0.168).This finding indicates that the moderating variable degree does not significantly influence the effectiveness of technology in language learning compared to traditional language learning.However, the use of technology in language learning throughout the four categories of degrees is proven to be effective compared to traditional language learning.The use of technology in language learning is most effective in senior high school (g = 0.979; p < 0.050), followed by junior high school (g = 0.558; p < 0.050), university (g = 0.536; p < 0.010), and elementary school (g = 0.484; p < 0.010).

Sample size
The study's moderating variable sample size consists of two categories: the studies with small and big sample sizes.The results of the moderating variable analysis as show in Table 6, that the effect size means score in both categories of the moderating variable sample size is not significantly different (Qb = 1.085; p = 0.297).This finding indicates that the sample size does not influence the effectiveness of technology in language learning compared to traditional language learning.Despite that, the effect size means score of the studies with small sample sizes (g = 0.605; p < 0.010) is higher than that of the studies with big sample sizes (g = 0.553; p < 0.010), the differences in the effect size mean score between the two categories of the moderating variable sample size has been confirmed insignificant.

Country
The moderating variable country consists of eight categories: Indonesia, Iran, Jordania, Malaysia, Pakistan, Turkey, and the United States of America.The results of the moderating variable analysis as show in Table 6, uncover that the effect size means the score of the studies in the eight categories has been significantly different from one to another (Qb = 52.433;p < 0.010).These findings indicate that the research locations (countries) significantly influence the effectiveness of technology in language learning compared to traditional learning.From the eight categories of the moderating variable country, the studies that have been conducted in Jordania earn the highest mean score and significant effect size of all countries (g = 1.180; p < 0.010), followed by Pakistan (g = 0.696; p < 0.010) and Iran (g = 0.649; p < 0.10).These findings show that the use of technology in language learning is effective compared to traditional learning in the three countries.On the contrary, the studies that have been conducted in Indonesia, Turkey, and the United States of America have been confirmed insignificant.Thus, the statement implies that the use of technology in language learning has been confirmed ineffective compared to traditional learning in countries other than Jordania, Pakistan, and Iran.

Skills under measurement
The study's variable moderating skills under measurement consist of seven categories: general proficiency, listening, reading, writing, vocabulary, and sign language.The results of the moderating variable analysis as show in Table 4, that the effect size means a score of the seven categories under the variable moderating skills under measurement has been significantly different from one to another (Qb = 18.308; p < 0.050).These results thus indicate that the skills under measurement influence the effectiveness of technology compared to traditional language learning.Then, from the seven categories of the variable moderating skills under measurement, technology has been found most effective in sign language (g = 0.725; p < 0.010).In addition, the use of technology in mathematical learning has also been found effective for exercising vocabulary mastery (g = 0.659; p < 0.010), writing skills (g = 0.498; p < 0.050), and general skills (g = 0.391; p < 0.010).On the contrary, to exercise other skills, such as listening, and reading, technology has been confirmed ineffective.

Technology in use
The variable moderating technology in use consists of five categories: computer, mobile application, online media, social media, and other media/technology.The results of the moderating variable analysis as show in Table 4, that the effect size means a score of the five categories under the technology in use moderating variable has been significantly different from one to another (Qb = 30.197;p < 0.010).The statement indicates that the kind of technology in use influences the effectiveness of the use of technology in language learning compared to traditional learning.From the five categories under the variable moderating technology in use, the category mobile application has been the most effective compared to the other media/technology (g = 0.724; p < 0.010), followed by online media (g = 0.523; p < 0.010), computer (g = 0.496; p < 0.010), and other media/technology (g = 0.261; p < 0.050).On the contrary, the use of social media in language learning has been confirmed ineffective compared to traditional learning.

Evaluation of publication bias
The funnel plot from the 43 studies shows that all plots of effect size have inclined to shape symmetrical patterns as shown in Figure 2. The statement indicates that there is not any issue of publication bias in the data that have been used for the study.Then, the results of Egger's test are z = 1.515 and p = 0.130, which confirm that the funnel plot that has been shaped is symmetrical.According to Rothstein et al. [41], when the fail-safe N value is higher than 5 K + 10 (K = a number of individual studies), there is no publication bias within the meta-analysis.In the current study, K = 43 and therefore 5(43) + 10 = 225.Afterwards, the failsafe N value that has been earned in the study is 3464 with a target of significance 0.050 and p < 0.001.These results also confirm that there has been no publication bias issue in the meta-analysis study.Hence, the publication bias in the meta-analysis study is not found.

Discussions
Technology has developed rapidly [42], [43].Consequently, technology influences all aspects, including education [44]- [47].Hence, the use of computers, smartphones, and laptops in the learning process has been a familiar scene in education [48].In other words, technology has made something challenging to be done quickly.At the same time, technology can solve the issue of representation and the issue of time and space [49], [50].The statement was deemed more prevalent when the world was hit by the COVID-19 pandemic [51], [52].This situation has urged all countries to integrate technology into their education [53].As a result, the conventional practice has been replaced by the virtual practice [54], [55], the offline classroom has been turned into the online classroom, the face-to-face meeting has been turned into the screen-to-screen meeting, and the blackboard has been replaced by the monitor [49], [50].
Many studies show that technology has delivered positive results to the learning process output [56], including one in language learning [15], [19], [57].However, the implementation of technology in the learning process entails numerous requirements, such as facilities, teacher competencies, and student literacy [58].Thereby, it can be immediately concluded that technology has delivered positive results to the learning process [15], [18], [28].Hence, the better the preparedness of all supporting requirements is, the better the impact will J Edu & Learn ISSN: 2089-9823  Technology's impact on language learning: Meta-analysis on variables and … (Syarief Fajaruddin) 519 be [51], [53].On the contrary, if the requirements are not fulfilled, then there is a possibility that the technology will not deliver a significant impact on the learning process output quality [59].Thus, something logical will be found in the various impact of technology on language learning quality.It makes sense that the findings from each country's data show different impacts since each country has specific educational criteria [60].However, in general, it is found that technology is able to support language learning output quality.
Behind the general description of the role of technology in the learning process, several pieces of information have been more detailed.These facts show that the technology can be implemented in all educational degrees and display positive results [14], [15], [24], [26].The nature of technology learning that can pack the learning into more variative activities has been alleged as the cause behind these findings [53].For example, native speaker-based and culture-based learning [61], [62] have made learning more attractive [63].Such implementation will be successful if the substance and the content pay attention to the profile of the students [48].Implementing technology-based learning designed based on the characteristics of the students will improve the learning process output quality in each educational degree.
The number of students in a classroom has also been discovered to have an impact.The study's sample size supports the assertion.The data indicate that implementing technology in both small and large classrooms has a significant positive effect.The optimal number of students in a classroom is a topic of ongoing debate [64], and there have yet to be any clear conclusions about which size is better [65].Another crucial aspect of the learning process has an influence over the language learning process [66].The classroom size in language lessons will still be maximized if the teachers can appropriately set the learning process [67].Thereby, implementing technology-based language learning can be recommended for small and big classrooms.
Optimizing technology for language learning is another equally intriguing topic of discussion.There is evidence that certain types of language proficiency have benefited from using technology, but the technology has had no discernible effect on the other skills.During the learning process, every competency exhibits particular characteristics [68], [69].Reading is an example of a competency whose characteristics make students uneasy when the technological intervention is implemented [70], [71].Reading on the device is less effective [71] as the students feel more comfortable reading through papers [72]- [74].If students view instructional materials on a screen for an extended time, they may develop eyesight problems [75], [76].Implementing technology in language learning can be accomplished by focusing on the competencies to be enhanced.The analysis results indicate that technology contributes significantly to enhancing language proficiency, writing skills, vocabulary mastery, and sign language comprehension.

CONCLUSION
In conclusion, the effectiveness of technology in language learning has been demonstrated by the authors, with technology surpassing traditional language learning methods.The impact of technology on language learning has been categorized as "Moderate" and "Significant," highlighting its substantial contribution to the learning process.Several factors that influence the effectiveness of technology in language learning have been identified, including the geographical location of the studies (countries), the specific language proficiency being measured, and the type of technology employed.Importantly, no significant impact has been found on the effectiveness of technology in language learning due to educational degree and sample size.To advance our understanding further, future research should delve into the underlying factors contributing to the varying rates of technology's effectiveness in language learning across different countries.This investigation can shed light on the nuanced aspects of technology integration in diverse educational contexts.Practitioners and educators in the field of language learning are encouraged to carefully select the appropriate technology for enhancing specific language proficiency skills.Tailoring technology solutions to match the desired language competency can optimize the learning experience and outcomes.As technology continues to evolve, its role in language learning is expected to expand and diversify.By continually exploring the dynamic relationship between technology and language acquisition, the full potential of technology in language learning can be harnessed.

Figure 1 .
Figure 1.Forest plot for the 43 independent samples

Figure 2 .
Figure 2. Funnel plot of the 43 independent studies

Table 1 .
Inclusion/exclusion criteria for studies in the meta-analysis

Table 2 .
Database of the main study

Table 6 .
The impact of the use of technology in language learning: overall results and moderating variable analysis ISSN: 2089-9823  Technology's impact on language learning: Meta-analysis on variables and … (Syarief Fajaruddin) 517 **p < 0.050; k = the number of studies; CI = confidence Interval; Qw = Q within; Qb = Q between

Table 5 .
Descriptive statistics of the included studies