For much of our history, America was a predominantly agrarian society. Law makers therefore took into account that November was perhaps the most convenient month for farmers and rural workers to be able to travel to the polls.
The fall harvest was over, (remember that spring was planting time and summer was taken up with working the fields and tending the crops) but in the majority of the nation the weather was still mild enough to permit travel over unimproved roads.
Since most residents of rural America had to travel a significant distance to the county seat in order to vote, Monday was not considered reasonable since many people would need to begin travel on Sunday. This would, of course, have conflicted with Church services and Sunday worship.
Why the first Tuesday after the first Monday? Lawmakers wanted to prevent election day from falling on the first of November for two reasons. First, November 1st is All Saints Day, a Holy Day of Obligation for Roman Catholics. Second, most merchants were in the habit of doing their books from the preceding month on the 1st. Apparently, Congress was worried that the economic success or failure of the previous month might prove an undue influence on the vote!