Often times, N64 controllers will drift in one direction due to the stick being worn out. You can compensate for this in three ways.
Ensure the stick is perfectly straight when starting the console. The console uses the starting position as the "zero" position for the game session, so if it is in the incorrect.
Hold the stick in a position that the drift doesn't occur, unplug the controller, and plug it back in. This will reset the "zero" position.
Hold the stick in a position that the drift doesn't occur, then press L+R+Start. This will reset the "zero" position.