For the simple reasons that population is growing and the productivity is also growing. Do the math, if GDP does not grow, the hours worked per person must drop. Now if everyone values leisure hours just as much reduced work hours does not have to translate into increasing involutanry unemployment. The reality in the US points the other way though.
By the way, the FRB’s charter is to maintain price stability AND full employment. Whether they can be achieved at the same time is another question.