A piece of bread here and there is not a bad thing. I think diets work best if you have a “cheat day” where you can eat what you want. That keeps cravings from getting out of hand.
Even with one cheat day a week, I’ve hit my weight loss goal and actually am struggling to keep weight on.