In my own experience it is better to focus your energy on the activities you love (don't fear to do that, is the best you can do) and you will meet people in the way. They will go if you stop hanging with them.
Maybe it is your ego telling you that if you continue hanging with them you will waste time and energy.
If you follow your path they will have an example to do so themselves