Publications
WORKSHOP (DOMESTIC) Top-k方策分解に基づくランキング方策の効率的なオフ方策学習
岸本 廉 (東京科学大学), 田中 滉一 (慶應義塾大学), 清原 明加 (コーネル大学), 成田 悠輔 (イエール大学), 清水 伸幸, 山本 康生, 齋藤 優太 (コーネル大学)
2026年度 人工知能学会全国大会（第40回） (JSAI2026)
June 12, 2026
We study Off-Policy Learning (OPL) for ranking policies, which learns new ranking policies solely from historical logged data. OPL is particularly challenging in ranking settings because the action space is extremely large. Existing methods, which primarily adopt either policy-based or regression-based approaches suffer from high variance and high bias, respectively. To overcome these issues, we propose Ranking Policy Optimization via Top-k Policy Decomposition (R-POD). R-POD decomposes a ranking policy into two stages: a first-stage policy that selects the top-k actions and a second-stage policy that determines the remaining actions. The first-stage policy is trained via a policy-based approach, while the second-stage policy is trained via a regression-based approach. We introduce a novel policy gradient estimator that applies importance weighting only to the top-k actions, substantially reducing variance. Comprehensive experiments demonstrate that R-POD significantly improves OPL performance in large ranking action spaces.
Paper :
Top-k方策分解に基づくランキング方策の効率的なオフ方策学習
(external link)