We all are in movement physically and more importantly, socially. Our actions (and whether they are purposeful or not) all influence those around us. And just as a normal item in motion pushes and degrades other items around it passively, so do our actions do to those around us.
The key note is that both ambient and passive actions push people away. So even as you act passively towards those in your life, you push them away. The only way to combat this is with purposeful actions that draw those you want with you closer. Inviting people to events, calling them, engaging their values.
You push away those you do not actively pull in simply due to the nature of influential social motion. Remember this and keep who you want, and realize those whom you assume will always be there due to sheer historic momentum, will slip away.