Thanks for the answer. I want to have three-finger swipe for mission control and that stuff so my only way to drag a file without clicking is to enable Dragging from: Accessibility/Mouse & Trackpad/Trackpad Options/Enable dragging.
But when I enable that, and you can try it by yourself the tap-to-click response is slower.