Mid 19th century; earliest use found in Sylvester Judd (1813–1853), novelist. From riff-raff + -y.