Should I leave him alone and get on with my life?

I had been with my b/f for almost a year and 4 mnths ago he moved in, I am 34 and he is 39 and we both have 2 children. We started bickering nothing serious and one day out of anger I told him to leave, I said afterwards I am sorry. We got on so well and he said that he wanted to spend the rest of his life with me etc and had even thought about proposing. He then lost his job and we had a row one night while we were out and when I came home he had packed his stuff and gone to his parents which is 1 1/2 hours drive away. When I spoke with him the next day he said that he was not coming back but he wants a future with me but where he is. Since then he has visited me at the weekend but made comments about we would of had a nicer day had we been where he lives, which hurt my feelings. I was prepared to up and leave and move to where he is but he wasn't really giving me any urance and had been quite cold towards me. I told him my feelings and he ended it. I love him so much.