My hubby's Grandma made many, MIL made some and that is what she used to teach crocheting, I have made many all with acrylic yarns and have never had a problem. I prefer them to put things on rather than take out of oven (I usually use mitts I have bought) but do use them for both at MIL's house. I will probably make some mitts now that I have seen some patterns for that. (Why buy a quilted one when I can knit one instead?) I think I will not use acrylic when I do.
I guess it is all a matter of personal preference, for years I have been making, using, selling and giving as gifts Hot pads and pot holders using acrylic yarn and have never had a complaint or a problem with them. I gave red and green ones for Christmas presents 15 years ago and the last time my Step-daughter visited she told me that she was still using hers, said they got kind of dingy and she had soaked them overnight in Basic-H and they came out looking almost new.