Stephanie,
You need to confront your husband. Communication is key here. Don’t hold anything back because if you do it could very well mean trouble in the future. My mother held everything in and never confronted my father about things that bothered her. Now she has left my father after 30yrs of marriage and she justifies it because of all this “ammo” she has against my father. Your husband needs to wake up! There is nothing wrong with helping others but when it’s affecting your marriage that’s when it becomes a problem. How would your husband like it if you were helping and talking to men all the time? Would he accept it if you gave him the same reasons he’s given you? And if your husband is not a pastor then the women he talks to should know better as well. They should seek guidance from a pastor or counselor, not another married man because that is just asking for trouble! I will keep you in my prayers Stephanie, try to remain positive. I know it’s easier said than done but trust in God and pray to Him. He will hear you