Both of the above posts make good points. Sex is a very powerful drug. (Not just for SA survivors) Lloydy's first three sentences are the plain honest truth. If your BF is seriously committed to you and loves you, he will probably be totally faithful. There is a big difference between fantasizing and cheating. My hubby and I talked about this. (Communication is SO important) We decided that fantasizing is okay, looking at others is fine, touching or anything further is NOT okay.
You mentioned you had a history of intimacy and sex rushing yourself. Do you think that might bother him a little too, but he just keeps it to himself? TALK about it together. If you two are planning on this relationship lasting and possibly having children, then sit down together to talk and maybe even make a plan for times when sex and intimacy might not be frequent or even possible. (like when you're pregnant.) There are many options during these times. (private message me for specific suggestions, I don't wish to be that blunt here.)
Communicate, learn, and try to understand what causes the desire for him (and you) to be sexually acting out. You've got to know what you're fighting before you can beat it. Best of luck to you both. Let us know if we can help.