Eating like some hinterlands peasant is better but who is going to do that?
Anyway, MM, you say you have a physical job, so if that is not keeping your weight down it probably is sleep and nutrition to a large degree.
I like exercise though. If the labor part of your job is low level but consistent I would go for a few intense workouts of some kind per week, definitely stay away from the adult softball league, basket ball yes, if you want to be social. Go for things known to change body composition , hill running , sprinting( actual running or substitute a bike or eliptical), jumprope, weights. You don’t want to do long slow things that just add fatigue to your life.
If you actually do heavy work then it’s probably your diet.
Problem with exercise is that it has a tendency for us to remove activity from the rest of our day. Work out hard then be fatigued and sit around the computer and in proximity of the fridge instead of going out and washing the car, for instance. Stealthy, we can remove a lot of calorie burning activities over the course of time because of exercising. I personally have never had a good exercise routine and gained weight though, only when I got real inconsistent or cut back overall activity a lot.
I don’t have a perfect diet, i binge on average a couple of times a month, sweets. I eat a lot of good stuff though, I think I have my macros figured our more or less for me. I don’t eat much of the worst stuff other than your regular desert item here and there, or drink non-nutritive beverages much, sodas, or alcohol.