I am a pom pom person and think it looks adorable with. What I sometimes do with pom poms when I am not sure if a person will want it is leave long ends to the yarn that ties them, pull them through, and tie them with a bow so the pom pom can be removed if the recipient doesn't want it, or can knot the ends to make the pom pom permanent. That may help solve your dilemma.
I do this, also. It works well if the hat is washable, you can remove the pom pom because they do not fair well in the laundry process sometimes. MN