If you hold down the Ctrl key, you can selectively pick all the files you want with the mouse as you do now (or alternatively hit Ctrl-A, then, whilst holding the Ctrl key, unselect those you do not want.). However, if you keep the Ctrl key pressed whilst dragging the files, then this is instructing Windows to COPY the files (as opposed to moving them). So, if you want to move them, simply release the Ctrl key before dragging, or when you have selected all the necessary files, simply right-click on any of them and select Cut, then paste them into the desired folder.
Also, be aware that if you have several drives (or partitions), when you drag files across drives, even without holding the Ctrl key down, they are still copied and the original remains, this is a 'feature' of Windows :)
Paul