Why didn't Remus visit Harry?

It's been ages since I read the books and I don't remember if there was any explanation as to why Remus never visited Harry all those years he was with his aunt and uncle. So why didn't he? If Petunia did forbid him, he could've visited Harry in his school maybe!