I say set up a sting operation with cameras! If the landlord is coming over and walking in without notifying you, and then sneaking off, there’s a good chance are that he’s up to no good.
I saw something last week where this lady was suspicious about lights in her place being on when she got home, so she set up a clock radio camera and caught this guy coming in and putting on her underwear and lingerie.
You could set up a small camera to a laptop and hide it as well. These devices are super cheap, and you can easily set this up and remotely monitor it.
If you don’t want to go the sting route and instead notify your landlord, I would change the locks. If the landlord has a problem with this, then I would move.