AI Learns to Summarize Indonesian Friday Sermons Without Human Labels

Friday sermon transcripts in Indonesia pile up faster than anyone can read them, with typical sermons running twenty to forty-five minutes and transcripts stretching well past a thousand words. A new study published in Discover Informatics has built the first automatic summarization system designed specifically for this material, fine-tuning the Indonesian language model IndoBERT to pick out the sentences that matter most from Khutbah Jumat transcripts. Because no human-written reference summaries exist for this domain, the researchers trained their model on pseudo-labels, noisy but informative labels generated entirely by algorithms, and the approach paid off: their fine-tuned classifier achieved the highest ROUGE scores of all the extractive methods they evaluated, offering a practical template for summarization in languages and domains where annotated data simply does not exist.

The corpus behind the study is itself a first. The researchers collected 210 Friday sermon transcripts from YouTube using the YouTube Data API and a transcript library, keeping only complete sermons delivered in Indonesian with at least 200 words after cleaning. After stripping timestamps, bracketed non-verbal markers such as music and applause cues, and normalizing whitespace, the corpus totaled roughly 293,000 words, with a median sermon length of about 1,424 words and durations ranging from 2.5 to 54.5 minutes. The transcripts span five themes common in Islamic discourse: creed (aqidah), jurisprudence (fiqh), ethics (akhlak), spiritual purification (tazkiyah), and social affairs (muamalah). Viewer counts ranged from community-level sermons to videos with more than a million views, giving the dataset a wide spread of preaching styles and audiences.

The central technical challenge was supervision. Prior Indonesian summarization work has focused almost exclusively on news articles, using large datasets such as Liputan6 and IndoSUM, where human-written summaries already exist and salient information clusters near the top of the article in a lead-heavy structure. Sermon transcripts are different on every axis: they are spoken rather than written, they code-switch between Indonesian and transliterated Arabic religious terminology, and they follow a rhetorical arc of opening praise, thematic exposition, and closing supplication that bears little resemblance to news prose. Sentences carrying essential Islamic terminology could easily be under-weighted by generic relevance signals, so the team designed a pseudo-labeling pipeline that combines three complementary scores rather than relying on heuristics built for journalism.

Each sentence in the corpus received a composite score blending three signals. The first is TF-IDF relevance, a classical measure of how distinctive a sentence’s vocabulary is across the corpus, normalized for sentence length. The second is a position bias: sentences falling in the first 15 percent or the last 10 percent of a transcript receive a bonus, reflecting the rhetorical structure of sermons, where the introduction and closing supplication tend to frame the message. The third is Islamic keyword density, computed against a curated lexicon of 236 domain-specific terms across the five thematic categories, filtered down from an initial list of 277 candidates by retaining only words that actually appear in the collected transcripts. The three normalized scores were combined with weights of 0.5, 0.3, and 0.2 respectively, a configuration chosen on a held-out validation subset where performance proved insensitive to small weight changes. Sentences scoring above the 70th percentile within each document were labeled as summary-worthy, yielding 38,247 labeled sentence instances across the 210 documents.

With pseudo-labels in hand, the researchers fine-tuned IndoBERT-base-p1, a 110-million-parameter BERT model pre-trained on a 220-million-word Indonesian corpus, formulating summarization as binary sentence classification. Each sentence was truncated to 128 tokens, passed through the model’s classification head, and trained end-to-end with the AdamW optimizer at a learning rate of 2e-5 over ten epochs. At inference time, sentences were ranked by predicted probability and the top 30 percent extracted as the summary. The training dynamics revealed a familiar tension: training accuracy climbed to nearly 99 percent, but validation loss bottomed out at epoch 2 and validation accuracy plateaued around 78 percent, clear signs that the model began overfitting the noisy pseudo-labels after the early epochs. Because no early stopping was applied, the final-epoch checkpoint was the one evaluated, and the authors flag early stopping and label smoothing as natural refinements for future runs.

On a held-out test set of 21 documents, the fine-tuned IndoBERT posted a ROUGE-1 of 0.5181 (95 percent bootstrap confidence interval 0.460 to 0.575), ROUGE-2 of 0.4273, and ROUGE-L of 0.4400, the highest of all evaluated methods on those three metrics. Paired bootstrap comparisons showed the gains were statistically consistent over TextRank and a generic multilingual-BERT BERTSum-style baseline, but not over the simple TF-IDF baseline, whose confidence intervals overlapped. On BERTScore F1, a semantic similarity measure, IndoBERT scored 0.7310, actually below the untuned BERTSum baseline’s 0.7407. That divergence matters: it suggests the fine-tuned model more closely reproduces the lexical choices favored by the pseudo-labeling heuristics without necessarily producing summaries that are semantically closer to the reference in embedding space.

The evaluation comes with an honest and unusual caveat. Because no gold-standard human summaries exist for Indonesian sermons, both the training labels and the evaluation references derive from unsupervised heuristics, making the assessment partly circular: high scores indicate agreement with heuristic sentence selection, not validated summary quality as judged by human readers. LexRank was excluded from the headline comparison for exactly this reason, since scoring it against its own output would trivially yield perfect scores. A complementary expert validation, in which three Indonesian-language specialists rated 20 system summaries on a five-point scale, produced a mean of 4.90 to 4.97 with 93 percent of ratings at the maximum, but the authors caution that a strong ceiling effect, linguistically rather than Islamically trained validators, and light readability edits applied before rating all limit what those numbers can show.

Finer-grained analyses added nuance rather than drama. Sentence-level classification on the test set showed recall of 0.734 exceeding precision of 0.537, meaning the model favors capturing important sentences at the cost of including some irrelevant ones, a trade-off the authors note is acceptable for search indexing and preview generation but could be tuned toward precision for very short outputs. An exploratory topic-wise analysis found no statistically significant difference in ROUGE-1 across the five sermon categories (Kruskal-Wallis p = 0.11, with only 3 to 7 documents per topic), though ethics sermons scored highest at 0.662 and spiritual purification lowest at 0.403. An ablation study delivered a humbler result than the team anticipated: removing TF-IDF produced the only decline, a mere 0.008 ROUGE-1, while removing position bias or keyword density slightly improved validation scores, suggesting the multi-signal design’s benefit over TF-IDF alone is smaller than hypothesized and warrants re-examination on larger evaluation sets.

The practical implications reach beyond the laboratory. Automatic summaries could power indexing and search over large sermon archives on da’wah platforms and mosque-management systems, letting users locate sermons on specific topics without listening to full recordings. Short previews could help time-constrained listeners decide which sermon to watch, and text summaries improve accessibility for hearing-impaired users or those in low-bandwidth settings where streaming video is impractical. The authors are careful to note that human review remains essential, particularly for religious content where dropping a conditional clause attached to a jurisprudential ruling could change its meaning, something neither lexical overlap nor embedding similarity can guarantee.

The study’s limitations map directly onto its future work. The test set of 21 documents yields wide confidence intervals; YouTube auto-captions contain speech recognition errors that can propagate into summaries, as one extracted passage of garbled transliterated Arabic illustrated; the train-test split was not controlled for speaker independence; and doctrinal correctness was never assessed. The team’s stated priorities are constructing a human-annotated gold-standard reference set for the existing test documents, manually auditing pseudo-label quality, extending the framework to abstractive and multi-document summarization with models such as IndoBARTv2 and multilingual T5, and eventually incorporating review by Islamic studies scholars. For now, the work stands as evidence that pseudo-labeling offers a viable low-resource strategy for extractive summarization, one that substantially reduces, though does not eliminate, the need for human annotation in domains where the annotators have not yet arrived.

Subject of Research: Extractive summarization of Indonesian Friday sermon transcripts using pseudo-labeled data and a fine-tuned IndoBERT model

Article Title: Extractive summarization of Indonesian friday sermons using fine-tuned IndoBERT and pseudo-labeled data

Article References: Rahmiasari, S. A., Supriyono, & Aziz, O. Q. (2026). Extractive summarization of Indonesian friday sermons using fine-tuned IndoBERT and pseudo-labeled data. Discover Informatics, 1(1), Article 31. https://doi.org/10.1007/s44564-026-00032-z

Image Credits: AI Generated

DOI: 10.1007/s44564-026-00032-z

Keywords: IndoBERT, extractive summarization, pseudo-labeling, Indonesian NLP, Friday sermons, Khutbah Jumat, ROUGE, BERTScore, low-resource NLP, natural language processing, religious discourse, transformer models