1796, slang, said to be an old term of reproach to a woman signifying that she was a witch; extended 1823 to "anyone who departs hastily from a recent activity," especially while owing money. The two senses involve the two verbs fly.
Shady or untrustworthy: “Before buying stock in a newly formed company, the prudent investor will check its owners' credentials to make sure it's not a fly-by-night operation.”
Undependable and dishonest: fly-bynight correspondence school