Machine learning trims AI proficiency survey without independent validation

A support vector classifier could reproduce proficiency categories derived from a 30-item artificial intelligence questionnaire using information from only 17 of those items, according to a study of 600 undergraduates at the University of Casablanca. The model classified students with 96.7% accuracy on a held-out test set and achieved an area under the receiver operating characteristic curve, or AUROC, of 0.9666. Those results show that a reduced set of responses can closely recover a proficiency label built from the full questionnaire, but they do not demonstrate that the model predicts an independent measure of student ability. The researchers explicitly describe the task as construct-level classification rather than external prediction, because the selected predictors and the target were derived from the same instrument.

The study combined a descriptive survey analysis with a supervised machine-learning workflow. Participants were first- through fourth-year students recruited by convenience sampling through Google Classroom at the University of Casablanca. The sample included 260 male and 340 female students, with 330 studying scientific disciplines and 270 studying humanities. Respondents completed a validated 30-item questionnaire covering three dimensions of artificial intelligence-related technological proficiency: the use of AI applications and systems, the skills needed for AI, and strategies for strengthening AI proficiency. Each item used a five-point Likert scale, with negative statements reverse-scored so that higher totals consistently represented greater proficiency. Missing item responses were replaced with each feature’s mode, producing 600 complete records for analysis.

Overall proficiency was described as moderate, with the questionnaire total reaching 66% and the sample mean exceeding the scale’s hypothetical midpoint. A reported t-test gave a value of 7.65 against a critical value of 1.96 at the 0.05 significance level. The binary classification used a threshold of 90 points on the questionnaire’s 30-to-150 scale, corresponding to an average response of 3.0, the neutral midpoint of the Likert scale. Scores of 90 or above were labeled proficient, while lower scores were labeled non-proficient. This produced 380 proficient students, or 63.3% of the sample, and 220 non-proficient students, or 36.7%. The threshold was theoretically motivated but was not validated against a behavioral or clinical criterion.

The investigators reassessed the instrument’s validity and reliability in the Moroccan sample. Exploratory factor analysis using principal-axis factoring with varimax rotation supported the expected three-factor structure. The Kaiser–Meyer–Olkin measure was 0.89, Bartlett’s test of sphericity was significant at p < 0.001, and factor loadings exceeded 0.50 for most items. Cronbach’s alpha was 0.88 for AI applications, 0.91 for AI skills, 0.87 for enhancement strategies, and 0.93 for the complete scale. The authors noted that oblique rotation could be examined in future work because the three theoretical dimensions may be correlated, even though the varimax solution supported the proposed structure.

A central aim was to reduce the risk of target leakage, a methodological problem that arises when machine-learning predictors and their target come from the same test. The target category was first calculated from the construct-level total across all 30 questionnaire items, before any predictors were selected. The dataset was then divided into a training set of 75% and a test set of 25%. Correlation-based feature selection and all preprocessing were confined to the training partition, and the resulting model, scaling parameters, and transformations were then applied to the held-out data. This ordering prevented test-set information from entering feature selection or model construction, unlike a pipeline in which information can flow from the full dataset into training decisions.

Seventeen of the 30 questionnaire items were retained as predictors, representing all three theoretical dimensions. Their correlations with the aggregate target ranged from 0.65 to 0.75, while correlations among retained items were generally moderate rather than indicative of near duplication. The selected responses were standardized using means and standard deviations estimated from the training set. Two classifiers were then trained: a support vector classifier with a radial basis function kernel and Gaussian Naïve Bayes. Grid search on the training set selected a regularization parameter of C = 10 and a kernel parameter of γ = 0.1 for the support vector classifier. Class weights were balanced because proficient students formed the larger group. The implementation used Python 3.10 and scikit-learn 1.3, with a fixed random seed of 42 where applicable.

On the held-out test data, the support vector classifier achieved 96.7% accuracy and an AUROC of 0.9666, while Gaussian Naïve Bayes reached 94.7% accuracy and an AUROC of 0.9560. The support vector classifier’s precision was 0.9500 for the non-proficient class and 0.9778 for the proficient class, with corresponding recall rates of 0.9661 and 0.9670. By comparison, Gaussian Naïve Bayes had perfect recall for the non-proficient class and perfect precision for the proficient class but lower performance on the complementary measures. Simpler baselines produced accuracy of 85% for logistic regression and 88% for a decision tree. The AUROC difference between the two principal classifiers was not formally tested on the held-out set, although a paired test of their cross-validated accuracies gave p < 0.01.

Repeated stratified tenfold cross-validation produced mean accuracy of 96.6% and a standard deviation of 0.9% for the support vector classifier; Gaussian Naïve Bayes averaged 94.6% with a standard deviation of 1.1%. Re-running the pipeline with threshold scores of 85 and 95 left support vector classifier accuracy between 95.89% and 96.67% and AUROC between 0.9598 and 0.9666. These results suggest some robustness to small changes around the midpoint, but the feature subset was held fixed during the threshold sensitivity analysis. The cross-validation should also be interpreted cautiously: because feature selection and hyperparameter tuning were performed once on the original training partition rather than repeated within every outer fold, those estimates may be optimistically biased and are not strong evidence of generalization to independent data.

Subgroup analyses found associations between proficiency classification and both gender and academic specialization. Female students in scientific programs had the highest observed proportion classified as proficient, at 150 of 200 students, or 75.0%, while male students in humanities programs had the lowest, at 61 of 130, or 46.9%. Chi-square tests reported significant associations for gender and specialization and for their combined categories, but the gender-by-specialization interaction was not significant. When gender and the interaction were included in a logistic regression, the specialization effect was attenuated and was no longer statistically significant, suggesting that the unadjusted pattern may partly reflect an unequal gender distribution across disciplines. The study did not measure instructional exposure, infrastructure, prior AI training, or frequency of tool use, so the observed differences are associations rather than causal effects and remain hypotheses for investigation.

Important limitations constrain how the findings can be applied. The target and predictors still came from the same self-report questionnaire, leaving part–whole inflation and construct circularity possible despite the leakage-aware sequence. A convenience sample recruited from one public university may not represent Moroccan students generally, particularly if digitally engaged students were more willing to participate. Self-reported answers may also be affected by social desirability, and the instrument was adapted from a tool originally developed among Saudi Arabian students without formal testing of cross-cultural measurement invariance. The authors argue that future work should validate selected items against behavioral performance criteria, use fully nested cross-validation, test whether feature selection changes under alternative thresholds, and examine the three questionnaire dimensions as distinct views in a multi-view model. The central result is therefore a promising demonstration of measurement efficiency: 17 responses recovered labels based on 30, but the classifier remains an approximation of its own underlying questionnaire rather than a validated assessor of real-world AI skill.

Subject of Research: Machine-learning classification of AI-related technological proficiency among Moroccan university students using questionnaire data

Article Title: A leakage aware machine learning approach to classify the technological proficiency of Moroccan university students in artificial intelligence

Article References: Chahid, A., El Marzak, Y., Aouane, O., & Mansouri, K. (2026). A leakage aware machine learning approach to classify the technological proficiency of Moroccan university students in artificial intelligence. Discover Artificial Intelligence, 6(1), Article 1179. https://doi.org/10.1007/s44163-026-02001-9

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02001-9

Keywords: artificial intelligence, technological proficiency, machine learning, support vector classifier, target leakage, psychometrics, university students, Morocco, questionnaire validation, leakage, aware, machine