Publications
カンファレンス (国際) Efficient Offline Learning of Ranking Policies via Top-k Policy Decomposition
Ren Kishimoto (Institute of Science Tokyo), Koichi Tanaka (Keio University), Haruka Kiyohara (Cornell University), Yusuke Narita (Yale University), Yasuo Yamamoto, Nobuyuki Shimizu, Yuta Saito (Hanjuku-kaso, Co., Ltd.)
The 35th ACM International Conference on Information and Knowledge Management (CIKM 2026)
2026.11.8
Many recommender systems such as for e-commerce and news platforms aim to provide users with rankings they are likely to interact with. The particular goal in these interactive systems is to replace a deployed policy with a more effective one to increase user experience, sales, retention, etc. Off-Policy Learning (OPL) of ranking policies is attracting a lot of attention because it enables us to learn new ranking policies using only historical logged data. However, ranking settings make OPL remarkably challenging be- cause their action spaces consist of permutations of unique items, being extremely large. Existing methods primarily use either policy- or regression-based approaches. The policy-based approach, which typically uses importance-weighted policy gradients, can suffer from high variance due to large action spaces. The regression- based approach, on the other hand, estimates the expected reward using conventional machine learning methods, avoiding variance issues but potentially suffering from severe bias. To circumvent these issues of existing methods, we propose a new OPL method for ranking, named Ranking Policy Optimization via Top-𝑘 Policy Decomposition (R-POD), which combines the policy- and regression-based approaches in an effective fashion. Specifically, R-POD decomposes a ranking policy into a first-stage policy for selecting top-𝑘 actions and a second-stage policy for choosing the bottom actions given the top-𝑘actions. It then learns the first-stage policy via the policy-based approach and the second-stage pol- icy via the regression-based approach. In particular, we propose a new policy gradient estimator to learn the first-stage policy via the policy-based approach. This method can substantially reduce variance, since it applies importance weighting only to the top-𝑘.
Paper :
Efficient Offline Learning of Ranking Policies via Top-k Policy Decomposition
（外部サイト）