Sag Harbor or Shelter Island?

For an anniversary weekend? Never been to either. Hoping for some feedback. TIA