Well, you're clearly doing it wrong!
Check out the video below for proof:
Having trouble spreading and tamping the ballast on your model railroad? Tiny rocks getting everywhere, except where you want them? Lots of itsy bits left on top of the ties? Try using an orbital sander to vibrate those little grains and nestle them snugly in-between the ties, where they belong!
And, just for the record, no - I would not recommend that you apply the sander directly to the rails. . .