You are right, the estate and her heirs have a claim against your brother for breach of his fiduciary duties to your Mom. In fact, it may be a crime as well, if you can prove the facts which show he stole her money. There are both civil and criminal penalties which may apply to this case. APS does not prosecute anyone, they may investigate and then turn over their findings to the county attorney or a local police agency. Only the county attorney or grand jury can bring criminal charges against a person.
You can open a probate case for your Mom's estate and seek appointment as personal representative of the estate. Then, you would have the legal authority to file an action against your brother to force him to provide an accounting of your mother's money and what he did with it, and then the court could take action against him.
You should retain legal counsel and go after him.