Collar fit

Hi guys! I'm a new knitter working on the Badger and Bloom sweater pattern by Anne Ventzel, which starts with an Italian cast-on and twisted rib collar. The collar is loose, and maybe because of my LB Fishermen's Wool and the twisted rib, it's very stiff and sticks straight up instead of laying flat. It just doesn't look right on my scrawny neck lol.

I'm considering placing a lifeline and undoing the cast-on and collar to either redo it on smaller needles (I used 4.5mm for ribbing, 5mm for body) or switching to something like a taller, folded collar. This is my first sweater, so I'm not sure if either of these options are viable.

If you have any suggestions, I'd love to hear them. Or if I shouldn't touch anything until after washing and blocking, that would be helpful too. Thank you!