Area around button hole stretched out

So I am quite happy with how my buttonholes are coming along. However one thing I realized is happening, is the the fabric above the hole is getting stretched out. This probably happens when I open the buttonhole to position the needle on the backside, particularly for the last dozen or so stitches. In the picture here I already tried to manipulate some of the stretch back into place as can be made out by the fold above the stitches; beforehand it can be seen as an outward curve at the top of the waistband, some of which can still be seen. Any advice other than be gentler?