We are making a mistake.
These are discourse errors; the facts are important. We need to make a distinction between legal and illegal drugs and legal and illegal drug use, because this is where society makes the distinction and often puts legitimate drug users into the latter category. When we say drugs we have to make the distinction.
So to say that prescription Ritalin (methylphenidate, not an amphetamine) or Adderal (synthesized amphetamine) is "drugs" like uppers/speed is an error. Uppers and speed refer to illegally used or illegal drugs. If you get these on the street you could be buying any of a number of drugs including amphetamines, methylphenidate or ephedrine.
The relatively new idea that drugs taken when natural remedies would suffice has negative connotations, also ascribed to people who use medication at the suggestion of their doctors, most of whom have been trained to alleviate suffering but not in natural alternatives to drugs. This too is an error. In Florida I had the benefit of having a doctor trained in both (I searched until I found one), and so for example was able to avoid prescription medications for a bad sinus infection.
An interesting note: MRSA is one of the results of abuse of antibiotics. No stigma there though, I think because antibiotics don't result in a "high."
Caffeine is legal, and so is alcohol. But abuse of these two substances outstrips the abuse of most other single substances in the U.S.A. But because they are legal, there is less stigma. Herbal supplements are legal, and until people start becoming ill or dying from them, no one tracks the incidence of abuse.
Some of the substances on this hit list:
Tryptophan - eosinophilia-myalgia syndrome (EMS), which can cause death (there is ongoing debate about this one)
St. John's Wort and Gingko - both blood thinners
Bitter Orange (used in place of ephedrine) - cardiac arrythmia
And my all time favorite: Grapefruit juice: it can either make meds too strong or not work at all. Go figure. Pomegranate juice has the same effect but it's not as strong as grapefruit juice, which I drank in prodigious amounts before I found out. A sadness of my life - pomegranates were another favorite.
The truth is that nearly anything can be abused. Most medications are synthesized versions of things people have been using for centuries. When I lived in Northampton I took a natural remedy called White Willow Bark. Almost no one takes it because the synthesized version - aspirin - was cheaper to make.
Amphetamines and methamphetamines come from natural medicines. Amphetamines were used for people with asthma at first, but people began taking the inhalers apart and getting high. There was military use of both in one of the world wars.
I used guarana thinking that it was a caffeine enhancer but actually it is an amphetamine. There are a couple of herbal remedies that also are amphetamines, but I don't think any are as commonly used (outside of athletics) as guarana. Carnitine is one of them. Cathione (known as Khat) is banned in the U.S., but widely used in other countries. Methamphetamine comes from ma huang, which you can still get as an herbal extract. Ephedrine, the main derivative, can't really be found these days, and pseudoephedrine is nearly a prescription medicine. I used to take ephedrine (Bronkaid, formerly OTC) for asthma.
Interesting note - the further south you go, the stricter they are about dispensing pseudoephedrine or other stimulants. Florida was a nightmare, even to get my methylphenidate prescription filled. I was treated with suspicion at every pharmacy I tried. In Massachusetts no questions were asked. In MA, I found ephedrine at 7-11. In Florida no ephedrine anywhere. Luckily I still have a stock of Bronkaid. In Canada they don't have the long acting version of pseudoephedrine, but the short acting is sold on the shelves with no ID required. I haven't looked for Bronkaid equivalents yet.