I am going down town, i.e. to the business part of the town.
Down the country
properly means down the slope of the land, or as the rivers run. We
say “I am going up to town” when we mean out of the country into the
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894