Your opponents are not afraid of you, because they are ready for your every move. What they are afraid of is your constant self-improvement.
In other words, they are not afraid of you now, they are afraid of what you will become. This fear will eventually make them your allies.
Don’t apologize to others when saying no to them is saying yes to yourself.