The book "Bounce" by Matthew Syed argues the case of 10,000 hours to be an expert at anything.
For example, Kenya has some greater runners, not because of genetics but because they lived in a part of Kenya where they had to run to school and back on a regular basis.
Like you say, plenty of black people can swim. It is because they have had the chance to practice.