I was pretty thin as a boy and I'm just average in height. I just had to hit as hard as I could.
This guy was much larger than I was and he had been picking on me and my friends for three years, almost non-stop. I was scared of him. The only way I had ever responded to him was to cower and apologise. One day, I was just simply done with that and flew into him, swinging like one of those machines that plucks the feathers off of poultry.
I got into several more fights in the next few years, but I was never touched (well, except for that one time I mentioned in my "ask" thread, but no reason to go into that here, since it was personal and they had a grudge against me, they touched me hard, but I was the one still able to crawl away when it was over). I am five eleven and at fifteen, I was wiry, well muscled, kind of big-boned and about one hundred seventy five pounds. I'm not big, but I'm fast (used to be, that is).
In my senior year, I was on the boxing team, but I was unable to keep my weight from entering the "heavy weight" class (one hundred eighty nine and one half pounds was the line - I was often over one ninety), so I always had a rough time at the matches as the smallest heavy weight. It was damn good for me, though.
I took some martial arts beginning in my twenties, it helped a lot and I think everyone should try it. You gain a good deal of experience knowing JUST how much force to use in hostile situations, without hurting people, and it is very true that you can also gain enough personal confidence that you often diffuse scary situations with no violence at all. The exercise is enough reason to give it a try, alone, but the other benefits make the arts almost essential and it affects all aspects of your life.
Obviously, I have slowed down quite a lot since I was in my thirties and forties, lost some of my strength, hardly ever work out anymore, but I am still the wrong old man to fuck with and I never have those "same old fears" anymore.