As an ex-school horse, he is probably totally shut down emotionally around people. For all those years, he was expected to be a machine - the student pushes this button, the machine trots. The student pulls this lever, the machine turns on the forehand.
If you can help him find a connection with you beyond pushing buttons and pulling levers (beyond just giving cues and expecting mechanical responses), he won't feel like he's alone when he's with you. He will learn to be happy to go out for an adventure with his friend.
If it was my lease horse or new purchase, I would stop riding him and start to build a connection and communication with ground work. I don't mean chasing him around a round pen or longe line! That's just more 'pushing buttons' for him to do what he's told to do. I would spend lots of time grooming him, and if he indicates an itchy spot I'd give him extra-good skritchies there. Take him for walks, on the trails where you will want to ride. Think about what's going to make it interesting for him. Stop at a patch of green so he can get a nice fresh snack. Step over and around logs, rocks, trees. You say he's pretty bombproof, but you can find some things that will get his attention. Ask a friend to ride their bike up to you while you're out walking. Meet someone with a dog on a leash, or off a leash if the dog and the horse are OK with it. Open an umbrella, carry a backpack (maybe with a nibble of feed in it.) Just do anything that will engage the horse's mind and emotions, and make it an enjoyable experience for him.
He's surely had enough of 'being made to do things'! You (or a more experienced rider) could surely *maike him* go out alone, but neither of you will enjoy it much. If you can reawaken the life in him while he's under saddle, he'll be a happier horse in general and you'll enjoy his company a lot more too.