Other animals see your cat as prey - dogs, raccoons, coyote, foxes, etc.
There are plenty of highly communicable and sometimes fatal diseases such as feline infectious peritonitis, feline
immunodeficiency, and rabies.
There is a risk of easily spread fungal diseases like ringworm (which you and your family can catch from
the cat).
Your cat may return with intestinal parasites, respiratory infection, flea and ear mite infestation, and tapeworms.
Your cat may be inadvertently poisoned from toxic materials and plants.
Further, how would you feel if your next door neighbor let his cat use your garden as a litterbox? So why do you think it's okay for you to do it? Your wandering cat may also cause territorial battles in your area, meaning your cat gets injured or worse in fighting, and other cats are drawn to the area and begin marking behaviors - like spraying all around your house. Plus, if you haven't spayed or neutered your outdoor pet, you are contributing to the problem of overpopulation.
So, is there any upside to letting your cat outdoors? The answer is NO. Unequivocally. And if you won't listen to me, watch Emergency Vets on Animal Planet. You will see that cats are generally brought in suffering from one of two terminal conditions - renal failure from extreme old age (after two decades in a loving home) or fatal injury to outdoor cats caused by cars, dogs, or poisoning, including most commonly antifreeze poisoning. When your one year old cat is dying because it ingested antifreeze, it is not "his time." You have a responsibility to your animal just as you do your children. You would not allow them to wander in the cold, exposed to dangerous conditions. You should do the same for your cats (and dogs!!).
My feeling is the same as the opinion officially expressed by all cat registry organizations, as well as competent veterinarians. The bottom line is, if you let your cat outside unsupervised, you are risking his life - every single time.