There is a chance that they convert to your side. However, there is also a chance those units might revolt to a popular AI nation, losing you both the city and wasting your time. And if those units are strong enough, they will probably just suppress the revolt (if high garrison strength), although maybe the revolt will lower their HP.
In other words, I wouldn't bother. Let the artillery rain on them, and just take the city. Why leave it up to RNG?