How much do you trust him and how well do you know him? Ultimately, because our courts are such a disaster, all small business dealings are a matter of trust. (Individuals suing each other is almost always a loss for both sides)
If you have extensive experience working with him in the past and found him to be trustworthy you could consider investing a modest sum (travel expenses would probably fit that) without going the attorney route, but still I would try to get a term sheet or something in writing.
If however you haven’t worked with him extensively, if this is your first business dealing with him, or you don’t know how trustworthy he is, then you should calmly but very firmly restate your position that you need to have a written agreement before you are willing to invest any money.
Also, the more resistant he is to a written agreement the more I would become concerned. If he is just trying to avoid the trouble and cost of an attorney, I understand. (They can be expensive and enforcing a contract can be even more expensive) But if he is reluctant to come to well defined terms of what your relationship is going to be, and to put that in writing, then that’s a huge red flag.