We present a music recommendation ranking system that uses Transformer models to better understand the sequential nature of user actions based on the current user context.