You need to hire an attorney to discuss all these facts, particularly the reason why the judge said initially that he would not be permitted to visit his daughter. but remember that the presumption that he will be part of this child's life was made when you and he had sexual intercourse, and the rights that he obtained at that time are hard to take away.
