Money given to free-booters by way of exempting property from depredation. (Anglo-Saxon, mal, “rent-tax;” French, maille, an old coin worth .083 farthing). Grass mail was rent paid for pasturage. Mails and duties (Scotch) are rents of an estate in money or otherwise. “Black” in this phrase does not mean wicked or wrongful, but is the Gaelic, to cherish or protect. Black mail was a rent paid to Free Companies for protecting the property paid for, from the depredations of freebooters, etc.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894