I have assumed the penalty only applies in the all-city "provinces". The other provinces, after all, are marked as either mountain, forest or plain. But I've wondered about it too.
Update: It looks like it is only the all-city provinces which count as cities. I am currently looking at a lone level 1 medium tank unit in a city in a plains province, it has lost half its morale and hit points and is still listing with a strength of 4.6 which is 0.6 higher than its normal value. If it was halved for being in a city it should have a strength factor of less than 2.