3″ duct’s area is about 2 1/2 times smaller than a 5″ duct.

But you have what you have, and you can run it longer to compensate for less airflow if you don’t want to replace the duct.

Don’t replace the fan yet. See what kind of motor is in there. If it’s noisy, the bearings are likely shot. You can buy replacement motors for about $10.