Apparently Tommy Simpson, one of the first Brits to cycle as a pro on the Tour circuit, and who's death on Mont Ventoux during the '67 Tour de France was due, at least in part, to his taking "stimulants", had previously said that almost every cyclist in the peloton was "on some substance".
In fact, it seems that anyone who wasn't indulging, was viewed with suspicion and, in some cases, ostracized by the other riders.