After Saruman is released from Isengard why doesn't he just go to Mordor where can gather allies and be around Saurons forces?

Saruman in the books went to Bree and The Shire right after leaving Isengard, why didn't he go to the East and enter Mordor using his voice to gain entrance and attempt to manipulate Sauron since they were allied and he could have protection there?