Have either a fixed width side panel, or one that follows the contours of the legs.

NOTE

Even a fixed width side panel will taper towards the ankle