Optimizing Student Academic Performance Prediction Using Heterogeneous Ensemble Learning

This study presents a heterogeneous ensemble learning approach to improve the prediction of student academic performance through educational data mining techniques. The proposed model integrates three diverse classifiers—Random Forest, K-Nearest Neighbor (KNN), and Averaged One-Dependence Estimator (A1DE), integrated through Majority Voting. Data from 300 students enrolled in a postgraduate computer science program has been used for model training and testing. Comprehensive evaluation has been performed using 10-fold cross-validation and metrics such as accuracy, precision, recall, F-measure, and ROC. The ensemble model achieved a prediction accuracy of 96.88%, significantly outperforming individual models. The results highlight the potential of ensemble learning in educational contexts, particularly in accurately identifying at-risk students and informing timely interventions.

Introduction

Student academic performance is a critical indicator of educational effectiveness, reflecting the success of teaching methodologies and learning outcomes [1]. It holds substantial importance for stakeholders such as educators, administrators, students, and policymakers, offering valuable insights that support timely interventions, personalized instruction, and informed institutional planning [2].

Educational Data Mining (EDM) is an interdisciplinary field that leverages Machine Learning (ML), Data Mining (DM) techniques, and statistical analysis to uncover meaningful patterns in educational data. Recent advancements in EDM have encouraged the development of predictive models that assist in identifying at-risk students and optimizing academic support strategies [3]–[5].

Numerous studies have applied classification algorithms to predict student performance with varying degrees of success [1]. However, existing literature consistently shows that no single classification algorithm performs optimally across all datasets or learning contexts [6]. To address this limitation, ensemble learning has gained prominence by combining multiple classifiers. All these classifiers have unique strengths and inductive biases due to which ensemble methods can enhance generalization and predictive accuracy [7], [8]. These are typically categorized into two types: homogeneous ensembles, which use multiple instances of the same algorithm (e.g., Bagging), and heterogeneous ensembles, which combine different types of classifiers. The latter can be particularly effective, as it integrates diverse learning perspectives to yield more robust results. Such approaches have been successfully applied in domains including bioinformatics, text classification, medical diagnosis, manufacturing, and educational analytics [9]–[11].

This study proposes a heterogeneous ensemble framework for predicting student academic performance, aiming to develop a scalable, generalizable, and accurate predictive model. The proposed approach integrates three distinct classifiers—Random Forest, K-Nearest Neighbor (KNN), and Averaged One-Dependence Estimators (A1DE)—to capitalize on their complementary strengths. This ensemble-based method demonstrates improved classification performance compared to individual models, addressing the limitations of single-algorithm approaches.

The remainder of this paper is structured as follows: Section 2 presents the literature review, Section 3 describes the methodology, Section 4 discusses the experimental results and findings, and Section 5 concludes the study, presents limitations, and outlines future research directions.

Related Work

Educational Data Mining and Performance Prediction

EDM is an interdisciplinary field that focuses on developing and applying ML, DM, and statistical techniques to analyze educational datasets and uncover meaningful patterns [12]. Romero and Ventura [4] conducted a foundational review of EDM techniques from 1995 to 2005, highlighting classification, clustering, association rule mining, text mining, and visualization as key methods. Building on this, Baker and Yacef [13] reviewed 64 papers from EDM conferences and expanded the thematic scope to include modeling of student behavior, performance prediction, and course adaptation systems. These studies collectively establish EDM as a vital research domain for enhancing educational outcomes. Romero and Ventura [5] categorized EDM research published up to 2009 into ten thematic areas: data analysis and visualization, instructor support through feedback, student recommendation systems, performance prediction, student modeling, detection of undesirable behaviors, grouping, social network analysis, concept map development, and courseware planning and scheduling.

Machine Learning Approaches in Academic Prediction

Various ML algorithms have been used to predict student performance, including decision trees, support vector machines, Bayesian classifiers, and instance-based learners such as K-Nearest Neighbor (KNN) [1], [6]. Each algorithm offers unique strengths; however, their effectiveness often varies based on data characteristics, leading to inconsistent results across studies. This variability highlights a key limitation in relying solely on individual classifiers for academic prediction.

Ensemble Methods for Enhanced Predictive Accuracy

To overcome the limitations of single models, ensemble learning has emerged as a robust alternative. Ensemble methods improve predictive performance by integrating multiple base classifiers, thus reducing bias, variance, or both [14], [15]. These methods can be broadly categorized into homogeneous ensembles (e.g., bagging with decision trees) and heterogeneous ensembles (e.g., combining decision trees with probabilistic or instance-based learners). Techniques like boosting, stacking, and voting have shown significant improvements in diverse domains, including educational analytics [16].

Alwarthan et al. [17] demonstrated that ensemble classifiers consistently outperformed individual models in predicting student success metrics. Similarly, Luong et al. [18] introduced a framework incorporating both homogeneous and heterogeneous modules, achieving superior results on benchmark datasets. Ostvar and Moghadam [14] developed a Heterogeneous Dynamic Ensemble Classifier (HDEC), which integrated multiple algorithms and outperformed traditional ensemble approaches across UCI datasets.

A significant challenge in ensemble learning, however, is determining the optimal combination of base classifiers and ensuring sufficient diversity among them [19]. Researchers have proposed several evaluation metrics to guide the selection of base classifiers, including accuracy, training time, classification latency, and memory efficiency [20].

Identified Gaps and Motivation for the Proposed Study

While ensemble methods have shown promise, much of the existing work emphasizes homogeneous ensembles or limited classifier diversity. Few studies have explored the potential of combining fundamentally different classifiers—such as decision-tree based learners (Random Forest), instance-based learners (KNN), and probabilistic models (A1DE)—in the educational domain. Furthermore, there remains a need for scalable and generalizable frameworks validated on real-world academic datasets. This study addresses these gaps by proposing a heterogeneous ensemble model that integrates diverse classifiers using a majority voting mechanism to enhance prediction accuracy and model robustness.

Research Methodology

The dataset employed in this study consists of academic records of 300 students enrolled in the Master of Computer Applications (MCA) program at a university in India. It encompasses a range of demographic and academic features pertinent to academic performance prediction. The primary objective of this research study is to predict students’ end-semester examinations outcomes by leveraging various supervised ML algorithms. The proposed methodological framework is organized into three sequential stages or phases: data preprocessing, model construction, and model validation. Visual representation of framework is presented in Fig. 1.

Fig. 1. Research methodology.

Stage 1: Data Preprocessing

The first stage of the methodology focuses on preparing the raw datasets. This involves a series of preprocessing steps beginning with data cleaning to handle missing or inconsistent values, removal of outliers etc., following this dimensionality reduction has been employed to eliminate redundant or irrelevant features. This preprocessing step ensures that the dataset is both reliable and optimized for training. The training dataset consists of 200 student records from the 2018–2019 academic session, and the test dataset includes 100 records from the 2019–2020 session. Students’ academic performance is graded on a six-point scale (A to F). Each student record in the dataset includes demographic and academic features.

Stage 2: Model Construction

After preprocessing and storing the data in the required structured format, predictive models are built in two phases:

• Phase 1: Individual classification models are developed using various supervised ML algorithms from different algorithm families. These families are based on the theoretical process being used to build the models [21]. The models built can be deployed to predict the class/label of unseen or new data instances.

In the current study a set of learning algorithms are identified and selected from various families to build single classifier-based models [22], [23]. The classifiers families include Decision trees comprising of well-known methods such as ID3, J48, LMT, M5P, Random Tree, and Random Forest. Bayesian- these are statistical classifiers which include Naive Bayes and Bayesian Networks. Lazy learners, the approach underlies a range of instance-based learning techniques, including methods such as K-NN, IBk, KStar, and LWL. Functions include techniques like SMO, Logistics, and MLP etc., that belong to this family [24].

• Phase 2: A heterogeneous ensemble model is constructed by selecting and integrating the best performing classifiers from Phase 1 known as base-classifiers and combining their outputs through majority voting as presented in Fig. 2. The complementary strengths of individual classifiers selected enhance the overall predictive performance.

Fig. 2. Heterogeneous ensemble model creation using majority voting.

Heterogeneous Ensemble Model Using Majority Voting

In majority voting-based ensemble methods, multiple classifiers independently generate predictions for each instance in the test dataset. The final predicted class is determined by selecting the label that receives the highest number of votes, typically requiring a simple majority (i.e., more than half of the classifiers agree) [25], [26]. In the proposed work, majority voting serves as a core mechanism to integrate diverse classifier outputs, thereby enhancing overall performance and mitigating the weaknesses of any single classifier [27]. The pseudocode used to build heterogeneous ensemble model using three base classifiers is presented in Fig. 3.

Fig. 3. Pseudocode for majority-vote based heterogeneous ensemble model.

In this experiment, a heterogeneous ensemble is constructed using three base classifiers namely, Random Forest (RF), KNN, and A1DE which are integrated using Majority voting.

Random Forest

It is applicable to both classification and regression [28]. It constructs multiple decision trees; each trained on a bootstrap sample of the training dataset. This approach enhances model robustness, reduces variance, and generally improves predictive accuracy compared to single decision trees. Random Forests are effective in handling high-dimensional datasets and are resilient to overfitting in many scenarios, although they may still be overfit on small or noisy datasets. Their ability to aggregate the outputs of diverse base learners makes them particularly suitable for complex prediction tasks [29].

K-Nearest Neighbor

The K-Nearest Neighbor (K-NN) algorithm is a foundational instance-based learning method [30]. It operates by storing the training data and, upon receiving a test instance, identifies the k closest training tuples—its “nearest neighbors” within an n-dimensional attribute space, using distance metrics such as Euclidean distance. K-NN’s simplicity and accuracy have sustained its relevance in pattern recognition and classification tasks [6].

A1DE

The Averaged One-Dependence Estimator (A1DE), introduced by Webb et al. in 2005 [31], enhances the traditional Naïve Bayes classifier by relaxing its strict attribute independence assumption [32]. This method retains computational efficiency while offering improved accuracy through limited dependence modeling.

Stage 3: Model Evaluation

To evaluate the heterogeneous model for future instances, the 10-fold cross-validation technique has been applied. In 10-fold CV datasets are randomly divided into 10 equal parts having the same number of instances. In each iteration, one part is used as a validation set, while the rest of the 9 parts are combined as training set used for building the classifier. The validation set is utilized for evaluating the classifier model [33], [34]. Different evaluation metrics are used to evaluate the performance of heterogeneous classifier model. During the evaluation process, a set of evaluation measures have been used to assess heterogeneous models, which include accuracy, precision, recall F-measure, and ROC metrics [6], [35], [36]. These metrics provide a comprehensive view of the model’s effectiveness in classifying academic outcomes.

Experimental Results and Discussion

This section presents the experimental evaluation of both individual classifiers and the proposed heterogeneous ensemble model. The performance was measured using student academic data and assessed with multiple evaluation metrics.

Initially, individual classification algorithms including Random Forest, NB, A1DE, KNN, MLP, SMO, and KSTAR were implemented to build predictive models individually. These algorithms were selected based on previous studies performed by researchers in the field of EDM. The individual models were trained and tested using student academic datasets. Among these models, A1DE displays higher accuracy when compared to other individual models. Consequently, models with good or high accuracy rates as well as complementing algorithms such as A1DE, Random Forest, and KNN have been used as base learners for constructing the ensemble model in Phase 2. Further, A1DE has been selected for its probabilistic modeling that outperformed traditional Naïve Bayes by relaxing independence assumptions. Random Forest has been chosen due to its robustness, ability to handle high-dimensional data, and it mitigates overfitting and boosts generalization. KNN added instance-based reasoning and capturing fine-grained patterns that other models may overlook.

The ensemble model enhances the predictive accuracy by combining the outputs of individual classifiers using majority voting. This approach selects the prediction of receiving the maximum vote as the final decision for each instance.

Comparison of Performance Accuracy of the Single Classifiers and Heterogeneous Ensemble Model

Table I presents the results of 10-fold cross-validation for individual classifiers and the ensemble model. The models were implemented in the Weka machine learning platform using default parameters. Cross-validation helps evaluate model stability by dividing data into training and validation subsets [36].

Table I. Performance Comparison of Ensemble and Individual Models

Evaluation metric	Heterogeneous ensemble	Random forest	A1DE	Naïve bayes	KNN	SMO	MLP	KSTAR
Correctly classified instances	96.88%	75.19%	93.99%	84.21%	69.17%	72.93%	68.42%	81.88%
Incorrectly classified instances	3.13%	24.81%	6.02%	15.79%	30.83%	27.07%	31.58%	18.13%
Kappa statistic	0.959	0.641	0.9116	0.7692	0.5325	0.5985	0.5242	0.762
MAE	0.0104	0.0857	0.0217	0.0523	0.1539	0.2297	0.1105	0.0733
RMSE	0.1021	0.2865	0.1328	0.2145	0.2716	0.3221	0.2962	0.2479
RAE	4.08%	37.14%	9.42%	22.65%	66.68%	99.53%	47.87%	28.66%
TP rate	0.969	0.752	0.94	0.842	0.692	0.729	0.684	0.819
FP rate	0.006	0.090	0.012	0.061	0.16	0.117	0.158	0.068
Precision	0.97	0.764	0.94	0.71	0.77	0.74	0.57	0.818
Recall	0.969	0.752	0.94	0.842	0.692	0.729	0.684	0.819
F-measure	0.968	0.756	0.938	0.7204	0.5923	0.734	0.5778	0.815

The results demonstrate that the heterogeneous ensemble model consistently outperforms individual classifiers across all metrics. It achieves the highest accuracy of 96.88% and the lowest error rates, including a mean absolute error of 0.0104 and a root mean squared error of 0.1021. The Kappa statistics of 0.959 indicate a very strong agreement between predicted and actual classifications.

Models like KNN, SMO, and MLP showed relatively lower performance, with accuracy values below 70% and higher relative errors. Probabilistic models such as A1DE and Naïve Bayes performed moderately well but still fell short of the ensemble model in terms of precision, recall, and F-measure. These findings underscore the advantage of ensemble methods for achieving robust and reliable classification outcomes in educational data mining.

Fig. 4 illustrates the performance of the ensemble model and its base classifiers on an independent test dataset consisting of 100 academic records. Preprocessing steps applied to the training set were also used on this dataset. The results affirm that the ensemble model maintains superior accuracy on unseen data, confirming its generalizability and real-world applicability.

Fig 4. Comparison of Percentage accuracy on test data.

Conclusion and Future Work

This study demonstrated that a heterogeneous ensemble model—combining Random Forest, K-Nearest Neighbors (KNN), and Averaged One-Dependence Estimator (A1DE)—effectively predicts student academic performance. Achieving a 96.88% accuracy through 10-fold cross-validation, the model consistently outperformed individual classifiers across various evaluation metrics, indicating its robustness and reliability. Despite its high performance, the model heavily depends on the quality and representativeness of the training dataset. Any class imbalance or feature bias can disproportionately influence outcomes, especially in the KNN and A1DE components, which are sensitive to data distribution.

The results affirm the potential of ensemble methods in identifying at-risk students and supporting academic planning. For future work, incorporating behavioral and psychometric data, validating results with a larger dataset to enhance generalizability, exploring advanced ensemble techniques like boosting, and testing deep learning models could further enhance performance. Real-time deployment and longitudinal analysis will also be key to assessing the model’s practical impact and adaptability across academic terms. The model will also be trained and tested on non-academic datasets to check its generalizability.