As Ms. Whitbeck suggests, he is entitled to custody of some type, and joint physical custody is possible. But if you can prove he has only seen the child a few times since birth, it's less likely he will end up with joint. You will certainly want to have an experienced custody attorney working with you.
Ultimately he can get joint custody. You need to hire an attorney and respond to his paperwork with an Answer and Counterclaim. You need to gather your evidence and prepare for court. The judge has to decide what custody/visitation situation is in the best interests of the child, utilizing 13 separate factors. If you cannot afford an attorney, you need to at least consult with one to better understand both your rights and your responsibilities.