Is the book timing for the Potters' death wrong?

So I've had this question for a while now. I was reading Philosopher's stone a while back, and I realised: on the night of 31st October, 1981, Lily and James died, and Harry Potter survived, leaving Voldemort disembodied. Then for 1 whole day, 1st November 1981, the Wizarding World was celebrating and the night of 1st November, 1981, Harry Potter arrived at the Dursleys' doorstep. This is the time-line in the book if we go logically. But didn't Dumbledore mention that he had just visited the Potters' cottage. So why, according to the books did Voldemort die a day earlier? The only thing I could think of was that the travelling time was 1 day, but then Harry must've gotten hungry so how would Hagrid have fed him? This is just a possible loophole I found and I can't wrap my head around it...