It sounds like your employer is dumping more than the responsibility for his kids on you, and you are right to be concerned. I suggest you speak with them about your concerns and let them know that you feel this falls outside the purview of your job responsibilities, and they they could be exposing themselves to a lot of liability by letting kids come over unsupervised. Maybe they will take it to heart.

1. You should not let other children in. You should mention this to your employer. Pools, and particularly, the one you described are accidents waiting to happpen.
2. If you do not follow rule "1" you should make other children come with a parent.
3. Most townships have a rule that there must be a 6' foot fence surrounding the pool. A 3' fence is an added danger and the best thing you can do is enforce rule "1"

To improve pool and spa safety, the U.S. Consumer Product Safety Commission (CPSC) launched Pool Safely: Simple Steps to Save Lives, a national public education campaign to reduce childhood drownings, submersion injuries and entrapments.

The CPSC “estimates each year nearly 300 children younger than five drown in swimming pools and spas and more than 3,000 that age go to hospital emergency rooms due to non-fatal submersion injuries. An unknown number of these hospitalizations result in permanent disability, including brain damage. These deaths and injuries are preventable.”

Below you will find a link to the Pool Safely website and important information that you can review, download and share with others. For example,

Simple Steps Save Lives:

Learn how simple safety steps save lives in and around pools and spas.
Parents and families can build on their current safety systems at pools and spas by adopting additional water safety steps. Adding as many proven water safety steps as possible is the best way to assure a safe and fun experience, because you can never know which one might save a child’s life—until it does.

Never leave a child unattended in a pool or spa and always watch your child when he or she is in or near water

Teach children basic water safety tips

Keep children away from pool drains, pipes and other openings to avoid entrapments

Have a telephone close by when you or your family is using a pool or spa

If a child is missing, look for him or her in the pool or spa first

Share safety instructions with family, friends and neighbors

Learn how to swim and teach your child how to swim

Learn to perform CPR on children and adults, and update those skills regularly

Understand the basics of life-saving so that you can assist in a pool emergency

Install a four-foot or taller fence around the pool and spa and use self-closing and self-latching gates; ask your neighbors to do the same at their pools.

Install and use a lockable safety cover on your spa.

If your house serves as a fourth side of a fence around a pool, install door alarms and always use them. For additional protection, install window guards on windows facing pools or spas.

Install pool and gate alarms to alert you when children go near the water

Ensure any pool and spa you use has compliant drain covers, and ask your pool service provider if you do not know

Maintain pool and spa covers in good working order

Consider using a surface wave or underwater alarm

Your Employer can ask you to watch other kids as part of your job. You also can negotiate not to to it. Ultimately, what it comes down to is that you and your employer need to hash this out. The problem here is figuring out who is in the better position to get what they want. If you think they can't do without you, maybe you have the edge. But unless you have a contract pushing to hard on a point like this case get you fired.

The homeowner should put up a fence and bump up the homeowner's insurance coverage. A release would not work and likely be void for public policy reasons.

That raises a lot of questions and you are right to be concerned. Do the other kids parents know you are the only adult there? Are kids telling their parents that you are supervising? I can see many ways this could go bad. The homeowners coverage probably excludes employees such as you.
You need to ask owners to not allow kids to be there w/o an adult other than you present.

As a former pool owner I share your concern. You would only be responsible for an injury if you were found to be negligent in supervising children using the pool. The best rule is generally no children in the pool area without parents present. While a Release could be used, it could always be challenged in court.

The Homeowner should also have a very significant Umbrella Insurance Policy to cover the possibility of such an occurrence. The Home Owner should also speak to their agent to be certain that there coverage would also apply to you under the circumstances of a pool injury.

As another suggestion, whenever there is a special swimming event such as a birthday party, the Homeowner should hire a professional Lifeguard to watch the pool.

Require that parent be present IN WRITING to supervise

Have you approached your employer about this?
A pool or lifeguard may have better thoughts on rules. The type of pool involved, and other factors will be of concern.
You are not going to be eliminate the risk of liability. However, good rules and understanding by guests can go a lont way. Having parents accompany minors may be a good thought.
If I were to pick the #1 rule, it would be no diving. However I can not give you a complete list. I hope this is some help.

Yes. That's a good idea.