Query Selection Using Latent Similarity Improves Reinforcement Learning Performance by 15.92%

A novel two-stage query selection strategy that leverages latent-representation similarity has demonstrated a 15.92 percent improvement in policy performance alongside higher query efficiency over baseline methods in preference-based reinforcement learning, according to empirical tests on standard benchmarks.

Preference-based reinforcement learning operates by distilling an underlying reward function from binary human preference feedback evaluated on pairs of query trajectories. Because collecting human feedback is often associated with substantial costs, the overall efficiency and effectiveness of the learning framework depend heavily on the quality of the trajectory pairs submitted for human evaluation.

Existing query sampling techniques have not adequately accounted for the influence of feature similarity between trajectory pairs. Researchers observed that both excessively high and excessively low similarity across trajectory pairs can undermine the ultimate performance of preference-based reinforcement learning systems.

To overcome this performance bottleneck, the proposed method exploits structural differences within the latent representations of query trajectories across two distinct stages. In the initial stage, a reward-guided adjacent cluster sampling module selects candidate query samples designed to ensure fundamental feature differences and maintain high entropy between trajectory pairs.

In the subsequent stage, an entropy-based similarity filtering mechanism evaluates the candidate query samples to eliminate trajectory pairs that exhibit excessive similarity. This filtering step aims to maximize the information value of the remaining samples, establishing the final query set used for preference feedback.

The researchers implemented the query selection strategy within the QPA algorithm framework and evaluated it across DMControl and Metaworld benchmarks. In addition to the reported 15.92 percent performance gain, the authors made their implementation code publicly accessible in an online repository.

Subject of Research: Query selection in preference-based reinforcement learning

Article Title: Query Selection with Proper Latent-Representation Similarity in Preference-Based Reinforcement Learning

Article References: Dong, C., Wang, A., Wang, Y., Hua, Q., & Zhang, F. (2026). Query Selection with Proper Latent-Representation Similarity in Preference-Based Reinforcement Learning. Machine Learning, 115(11), Article 256. https://doi.org/10.1007/s10994-026-07197-6

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07197-6

Keywords: preference-based reinforcement learning, query selection, latent representations, trajectory pair similarity, reward function distillation, binary human preference feedback, adjacent cluster sampling, entropy-based similarity filtering, QPA algorithm, DMControl benchmark, Metaworld benchmark, policy performance, query efficiency, candidate query samples, reward-guided sampling, feature similarity, reinforcement learning