When to get a new cast?

I got a cast for my broken wrist a couple days ago, the doctor said it should be fine because the swelling would last a while, I told him my body heals abnormally fast, so the swelling would go down faster than 4 weeks, and he shrugged his shoulders. As I expected, my swelling went down pretty much the next day and now my cast almost completely slips off my wrist, when should I ask for a new cast?