Human feedback is commonly utilized to finetune AI assistants. But human feedback may
also encourage model responses that match user beliefs over truthful ones, a behaviour
known as sycophancy. We investigate the prevalence of sycophancy in models whose
finetuning procedure made use of human feedback, and the potential role of human
preference judgments in such behavior. We first demonstrate that five state-of-the-art AI
assistants consistently exhibit sycophancy across four varied free-form text-generation tasks …