Intended mechanic, so that allied players are able to protect eachother's provinces.
That is because Country A is allied with Country B, while Country B is allied with Country C. Country B's troops protect Country C's province from being conquered by Country A.
3 possible solutions:
- Country B and Country C stop being allied (downgrade relations to right of way is enough)
- Country A declares war on Country B
- Country B is asked to kindly move their troops out of the province