I bit the bullet a few weeks ago and downgraded to 3.1.3. Phone works great again–it’s like having a new phone.
They are probably gonna have to provide official support for downgrading. However, itunes is gonna cause them some grief. Apparently itunes is quick to remove old upgrade files and backups. I had to keep copying the 3.1.3 image into the itunes directory because everytime I’d restart itunes it would delete it. It also kept no copies of my old 3.x backups…only the last one it did after I upgraded to 4.0..which meant that I lost all the preferences for all my apps.