5-HT 2A and 5-HT 2C receptor antagonism differentially modulates reinforcement learning and cognitive flexibility: behavioral and computational evidence

Abstract:

ABSTRACT Cognitive flexibility, the ability to adapt behavior in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive–compulsive disorder (OCD) and major depressive disorder (MDD). Evidence suggests that flexibility, which can be operationalized using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behavior and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown. We investigated the effects of 5-HT 2A receptor (5-HT 2A R) and 5-HT 2C receptor (5-HT 2C R) antagonism on cognitive flexibility and underlying RL mechanisms. Thirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT 2A R and 5-HT 2C R antagonists M100907 and SB-242084, respectively, on reversal learning performance and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task. 5-HT 2A R antagonism impaired reversal learning during certain phases. 5-HT 2C R antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT 2A R antagonism decreased punishment learning rate at high and low doses. The low dose also increased exploration (beta) and increased stimulus and side stickiness (kappa). 5-HT 2C R antagonism also increased beta, but reduced side stickiness. These data indicate that 5-HT 2A and 5-HT 2C Rs both modulate different aspects of flexibility, with 5-HT 2A Rs modulating learning from negative feedback and 5-HT 2c Rs for learning from positive feedback.