Don’t we all!
When you release the arrow, the tension in the bow vanishes and the bow aligns itself in whatever direction the position of your hand suggests. If these two directions aren’t exactly the same, the bow will rotate slightly and give the arrow a kick to the side.
The bow hand shouldn’t ‘grip’ the bow but hold it in place. If you hold the bow handle too tightly then when you are at full draw, the tension in the string will force the bow to align itself in a certain direction. You can tell when you’re doing this, because the arrow will wiggle from side to side (“fishtail”) while it’s in flight, and your horizontal grouping will be poor. You will also notice that when you draw, the bow twists in your hand from its original position.
Apart from not gripping the bow a good solution is to invest in a Finger Sling:
The Finger sling wraps around the bow handle and fits on the thumb and 1st or 2nd finger of your bow hand.
With this in place you don’t have to grip the bow at all and the finger sling stops the bow jumping out of your hand when you release.