Not sure if you are 'scrolling' the map by moving the cursor to the edges of the screen, or if you are using WASD to move the map around. WASD is by far the better option.
Also, you can speed up the scrolling in the Options menu.
Finally, probably the best thing is to click in the direction of the crystal to get your DG moving that way, then scroll over to the crystal and click on it.