He do remember the people he got along with or those who paused a major threat or hurt his friends, he probably tends to forget minor vilains and background characters.