I think of it this way when I choose which to destroy:
With Barraks= advanced infantry and techies to repair building faster
No barraks= no advanced infantry, no pestry little people who can kill your tank easily
With Factory= vehicles "better than infantry" most ALOT EASIER TO SEE AND SHOOT AT!!!!!
No Factory= you get overwhelmed with little infantry that can dish out some good damage