You should have a parenting plan which defines your time and his time. He cannot block you from having your time. If he does then you need to go to court and tell the judge. There is no "primary custody." The child is shared and you are entitled to your time per the court ordered parenting plan.