I'm 52 and I love horses, and have all my life. I have "horse posters" all over my walls, if you can call framed lithographs by Richard Stone Reeves "horse posters." I also have a saddle stand in my living room with my Aussie saddle on it, which I am currently not using for riding; it makes a nice point of interest.
As for what my friends think, most of my best friends are also into horses (some of them make their living working with horses, one way or another); the ones who aren't into horses share one of my other passions, such as flying or literature, and we are happy in each other's company.
If you love horses, why on earth would you want to have someone for a friend who objects to your passion for them? What kind of friend asks a friend to give up something they care about? If you've got "friends" in your life who will think less of you because you love horses, you've got the wrong kind of people in your life.
Learn this and your life will be easier: good people, people who care about you, don't try to make you over or make you change. They especially don't try to make you give up things you love, because those things are central to your being. Giving up those things is like giving up a part of yourself. People who truly care about you would no more ask you to do that than to cut off your hand.
In your life you will encounter people who will attempt to belittle you for various reasons, scorn you, put you down, patronize you or make fun of you. These people are not friends, and are not people you should care about. Good people do not behave that way; they are respectful of other people's interests, even if they don't share them, and are accepting of people the way they are.
If you let fear of what other people might think drive you away from horses, you will have lost out on something that can enrich your life in ways you can't even imagine, not the least of which is finding people who share your love of horses.
So get out the posters, or lithographs; be a "barn rat" and hang out at a stable; visit equine-oriented websites and learn about things that are obscure and archaic, and enjoy what you find. And turn your back on the people who aren't supportive of you in this, because they are people you are better off without.
http://www.thoroughbredtimes.com/
http://www.bloodhorse.com/
http://www.harnessracing.com/
http://www.ustrotting.com/
http://www.marylandsteeplechasing.com/
http://www.tbheritage.com/index.html
http://www.arabianhorses.org/default2.asp
http://www.pyramidarabians.com/
https://www.drafthorsejournal.com/
http://ww2.keeneland.com/default.aspx
And those are just a start...