I always had the same problem with my wallet and keys and other stuff. Now I do pretty much the same thing. they always go back into my backpack as soon as I’m done with them, or upon returning home (if I went out without the backpack). It works well.