That's a very valid concern. Several things can make a big difference;
One thing to watch, you should avoid spraying during a temperature inversion; this refers to when the air at the ground level becomes cooler than higher air, unlike warm air that rises, cool air is dense and remains at the surface. Sprays applied in these conditions can become trapped in this cool air layer and can only move horizontally. Some of the seemingly best weather conditions for spraying are often the worst. For example, around sunrise on a clear, nearly calm morning, air temperature measured 5 feet above the surface may be 4 to 10 F greater than air temperature measured near the surface. Dew should serve as a warning that a temperature inversion may exist. Spraying when the sun shines creates less drift than when it's cloudy, so, sunshine several hours after daylight before it gets windy is a good time to spray.
Another thing to keep in mind is; what spray tips are on my spraying rig? There are tips designed to produce more of a fog, these are bad for spray drift. I am currently using Teejet AIXR11004 nozzles which have air induction technology for better drift management. These are best for systemic sprays like roundup and raptor, which are well suited to coarse droplets, but not quite as good for contact herbicides like Gramoxone.
Wind speeds below 10 mph, and knowing wind direction is a must, but wind is finicky and hard to predict. I declare every time I start spraying the wind kicks up. Keep a buffer zone from the downwind edge of the field if there is a breeze. Driving straight into the wind makes a lot of spray drift. Spray with the wind from the side if you can.
Keeping the sprayer booms lower helps a lot, just don't go so low that there's gaps between the individual spray nozzles. Driving slower leaves less small drift particles hanging in the air behind the spraying rig.