To go by beggar's bush, or Go home by beggar's bush — i.e. to go to ruin. Beggar's bush is the name of a tree which once stood on the left hand of the London road from Huntingdon to Caxton; so called because it was a noted rendezvous for beggars. These punning phrases and proverbs are very common.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894