I am not going to bandy words with you—i.e. to dispute about words. The reference is to a game called Bandy. The players have each a stick with a crook at the end to strike a wooden or other hard ball. The ball is bandied from side to side, each party trying to beat it home to the opposite goal. (Anglo-Saxon, bendan, to bend.)
“The bat was called a bandy from its being bent.” —Brand: Popular Antiquities (article “Golf,” p. 538).
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Bandy from Fact Monster: