(To). What are you driving at? What do you want to prove? What do you want me to infer? We say the “wind drove against the sails,” i.e. rushed or moved violently against them. Falstaff tells us of “four rogues in buckram [who] let drive at him,” where at means against or towards. “What are you driving at?” is, against or towards what object are you driving or moving?
Source: Dictionary of Phrase and Fable, E. Cobham Brewer, 1894
More on Drive at from Fact Monster: