Down on Him
(To be). I was down on him in a minute. I pounced on him directly; I detected his trick immediately. Also to treat harshly. The allusion is to birds of prey.
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Down on Him from Fact Monster: