Nah, you'll be fine. There's absolutely no way you were trying to offend anyone, and it doesn't even particularly read as offensive. It's not really racist, just a little patronising... At the very worst, you'll be asked to think before you speak, I would've thought.
It's not like you've had a really bad colleague on the line and then said to the customer "Ah well, at least we won't have to swim home!"
He didn't have a funky name, did he? I was speaking to a fella from Mumbai last week called Caesar.