When you get attacked by a Dementor and it is stealing your soul, listen closely. You will hear a male voice yell 'Lily! Its him run!' or 'Lily! Take Harry outta here!', which we can assume its his dad telling his mom to get Harry to safety the night Voldemort came and murdered them.