It IS actually quite scary when you look at the information that NASA HAVE made available on NEO's.
I was looking into it before and the sheer number of large rocks flying around the inner solar system is a real eye opener.
http://neo.jpl.nasa.gov/orbits/
When you look at the orbital data given for each of these asteroids and then look into the uncertainty involved with the orbit of the
Apophis NEO, it becomes apparent that the given data in each case is little more than a guesstimate based on a primitive set of dynamics.
We simply don't know enough about the composition, rotation, tilt etc. of these rocks to give predictions with the level of accuracy that NASA is claiming. Add to that the prospect of these objects colliding with one another or their orbital paths being altered by 'near misses' with planets other than our own and the uncertainty grows. I'm not saying that it's a completely futile exercise, but even with a substantial increase in spending, there's just too many of them (each with their own set of variables) to really be able to do anything about it.
If the human race survives long enough, then no doubt at some point in the future we'll be able to defend ourselves against NEO hazards, but it certainly won't happen in our lifetimes.