Over Sensitive?

Last night I was talking to my husband on the phone (since he is always out of town) and I was giving myself a pat on the back from losing weight (because I can't count on him to do it...) and I said:

Me: I am only 20lbs away from my goal weight. I am excited because for a while I didn't think I would make it by the end of the year.

Him: Only 20lbs? That is good. Maybe you'll keep losing after that.

I am probably being over-sensitive, but that comment kinda stung. My mind translated this to: you look like you could stand to lose more than 20lsb. (And for the record, I do plan on losing weight after 145 but I am not setting any goals or timeline on it.)

I am pretty sure I am never going to be good enough for him at this point and there is always going to be some way for him to make me feel like I haven't done enough. It bugs me. He has never really struggled with weight or food issues, what gives him the right to judge me? My mind could go on and on with this if I start to over-analyze the statement.  I am not going to bring it up with him because talking about my weight is not a comfortable topic with him, possibly because it becomes clear that he wishes I was a size 4 kinda girl (which isn't really fair since I was a size 24 when we got married - I wasn't hiding the obese girl in the closet or anything). So help me out - how would interrupt this comment? Would you be offended? Brush it off?