1864, as a mining term; 1883, "boat fishing with drift-nets;" agent noun from drift (v.). Meaning "vagrant" is from 1908.
A derelict; BUM (1908+) drift off track
verb phraseTo deviate from proper conduct: I sometimes drift off track a bit, but I really try to do what is right (1970s+)