What I think is that you're very lucky. The chlorine will actually reduce infection as it's job in a pool is to kill bacteria however it will increase the time it takes for a wound to heal. Things like clay all harbour bacteria and also you could get bits of it in the wound which could cause further problems and could turn out really nasty. By no means are you immune from infection at all and really especially for activities such as using clay you should have the wound dressed (and with swimming it should be really for a public health point of view; if you look in a pool somewhere there is likely to be a sign, if only little, saying not to swim with open cuts).
You could argue that severity of wounds affects infection and yep, deeper wounds especially if gaping and ones with other factors are more likely to get infected however a wound of any type or any size can do; I've seen huge problems where the only thing we could put it down to was a tiny little scratch you'd think nothing of.
As said above, make sure you AT LEAST keep the areas clean and I'd also strongly suggest that you cover them up when dealing with clay/ play sports where you could get things in them. Your immune system can be ace against small scale infections but you don't want to keep chancing it as much as you are.
Wound Care Guide