Here is a clip of the President’s speech in which he made the unforced error of saying ‘You didn’t build that,’ and grammarically the sentence reads to say you did not build your own business. However I accept that if you look at the speech before and after then sentence it seems the president had a brain cloud moment using the antecedent ‘that’ when he meant the antecedent ‘those.’
The text before the sentence in question is about infrastructure and support systems, the conclusion statement that follow immediately are about infrastucture and support, so I think it is perfectly reasonable and not spin to accept that he was referring to the roads and bridges.