First of all, if you don't want the responsibility of these kids in your pool, I would NOT have them there. They can end up inviting friends and next thing you know, when you and your family go to use YOUR pool, you can't because there will be a MOB of kids there! Friends of friends, and tons of kids, all who will see YOU told them they could.
Next make sure for insurance and safety reasons you have a fence with a lock around the pool so no one including pets can get into this pool and drown accidentally. Including younger kids who may just want to see what it's like. Your house insurance should have it mandatory an so should your town that you have a fenced in locked area for the saftey of the area and for your own peace of mind.
I would think long and hard before agreeing to let these kids in your pool any time they want. Do you want them in there, whenever they please? When you're not home? When you and your wife and kids want some quiet family time and can't have it because YOU told them in an e-mail that they could come whenever they wanted?
They have NO right showing up at your door that way. You have a right to do whatever you want in your yard and get whatever you want for you and your family without having to have a ton of kids there to baby-sit them! You're also going to have to feed these kids, give them drinks when they're thirsty, and then IF and ONLY IF they tell you when they have to use the bathroom they will be in and out of your house or peeing in the pool itself.
I know you're close knit, but that's more of a reason why the parents should have more respect for your space and call you and talk to you about it before sending their kids to your door, that's just wrong! They should have talked to you first and not let their kids do that and I'm sure they knew their kids did it, and it's wrong of them. They need to respect that you want to spend time with your family in the pool and may have plans and could've had functions going on when the kids showed up and it was wrong. If you're all close you should be able to talk to them about this.
This is a decision that you should sit down with your wife and talk long and hard about. Do you want these kids coming and going as they please? When you have company and dinner parties? When you want some quiet time? Do you want to buy extra food and drinks for them? Do you just want the responsibiliy if someone should get hurt, I'm sure if the parents send them over they are going to expect that YOU are watching them. Do you want these going in and out of your yard and pool as they please? You're sitting down for dinner and there out your window are kids coming and going through your yard.
I would say that you are willing to have some pool parties over the summer and that there are some days when you and your wife are home that you will welcome the kids over with some RULES, but you will let them know in advance when those days are. You're sorry it can't be everyday but you will let them know when they can come over and what are expected of when they do come over.
For us, our pool rules were always:
NO pushing each other into the water
Keep hands to yourself!
NO rough housing in the water or around the pool!
No Jumping in the shallow ends
No peeing in the pool, let someone know when you have to go!
No eating around the pool
NO drinks in the pool NO food in the pool
Life jackets must be worn (depending on age, if they can swim, etc)
NO running around the pool area (someone can fall and get hurt)
NO hanging on the ladder
Use the ladder ONLY to get in and out of the pool
NO jumping on and off the pool toys
NO pushing anyone under water!
Always let someone know when you are here!
Always let someone know if you're not feeling well!
NO one enters the pool area without an adult present!
It's sad you have to be this way but you don't want to be taken advantage of either and you don't want anyone getting hurt. If this was your child.
Good luck. It's hard because these are your neighbors and if you're friends with them it can be hard, but at the same time if they're your real friends, they WILL understand! You bought this pool for you and YOUR family, NOT for the community! You have a right to enjoy it with your family and don't owe anyone in the neighborhood anything. You can just narrow down the pool use to pool parties with the neighbors or certain days when you know you're not doing anything and feel like having the kids there. You know you're going to be the ones babysitting, so you have to make sure you're up for that and if you're not, you're not. There's going to be times when you're going to want to be in the pool with you and your wife and you can't be made to feel guilty about that, and if they don't understand they're not real friends.
Good luck and I hope all goes well for you!