Can I legally tell the father of my 6 mth old son he cant attend a doctors appointment?

The father of my son wants to attend one of my sons doctors appointments. My son is 6 months old and up until this point his dad has not been too involved. We have joint custody but I am his primary guardian. His dad gets him for a total of about 12 hours a week. I pay for his medical insurance and have released that information to my sons father along with anything else he asks for, but do not want him attending this appointment with me do to the fact that we cannot be civil with one another. The appointment is on my time with my son. I told him that he is more then welcome to call his doctor and ask any questions he may have or to set up his own appointment. Legally can I keep him from attending?