The sweater is really lovely. I see what you mean about the spit splice, and I think that would bother me too. The trick would be to find a solution that will look better, and I'm not sure about that. You could put a lifeline in the row above and the row below, cut the middle stitch, remove those stitches you don't like, and graft (Kitchener stitch) them back together with a fresh piece of yarn. If you do an outstanding job, you may like the look better. If not, it still might bother you. Be careful not to go so far as to involve the raglan stitches themselves.
All things considered, Only you can decide if it will be a worthwhile effort. Before deciding, take a tapestry needle or a thin knitting needle and carefully and gently see if you can adjust / tease some of the looseness of those stitches away from where it is and spread the looseness across that row. That might just be enough to even everything out.
This video, although it won't help you with this fix, may help with future spit splices. It's very short and by Suzanne Bryan. She has managed to simplify it, and when I spit splice, it's the way I do it. Good luck, and let us know how it goes.
I felt really good when I saw that video because I had already figured it out. How many of us can claim thinning out some plies for spit splicing our unvention? I was quite sure I wasn't the first to do it.