You spent gold to get the units. There was no timeline when you had to upgrade the building.