You are walking along and you have 'found' a baby bird.  BEFORE assuming it is orphaned, let's check out a few things.
Is it naked, or downy?  Cup it in your hand to warm it, and try to find the nest.  It will be well hidden, perhaps on the ground in a grass tussock, in a bush, or in a tree.  It may be in a cavity (within a tree trunk, inside a building soffit/vent), and not visible. If it is daytime, move twenty feet away from the location the baby was found, sit down (still cupping the baby in your hands keeping it warm) and be quiet.  Observe the birds around for at least 30 minutes.  You will likely have better luck watching for and following the parents then just searching for the nest.  Replace the baby within the nest as soon as possible.  BIRDS HAVE NO SENSE OF SMELL, they will not reject a healthy baby.  If they toss it out again, there is something wrong with the baby - it's only chance is a vet or wildlife rehabilitator that specializes in these very delicate and labor intensive babies.
Does it have feathers?  Can it hop?  Does it have short, stumpy tail feathers?  DO NOT RESCUE, this is what we call a fledgling or brancher.  These are 'teenagers' not quite ready to 'drive' or fly yet, but very close.  Depending on the size of the bird it may take from two days to two weeks before it is ready to fly.  In this time it will strengthen it's muscles, and it's feathers will continue to grow.  If they waited in the nest for their feathers to mature the materials the nest is made from would damage the new feathers.
In the same way you cannot keep a toddler in a crib, a baby bird cannot remain in the nest forever.  It's parents will still continue to care for it on the ground, until it is old enough to care for itself.
But what about my/the neighbor's cat?  Unfortunately, cats are responsible (according to some reports) for the deaths of over 50 million songbirds annually.  But no more then we can take a human child from a parent 'because it might get hurt' we cannot take a baby bird from it's parents.  The negligent party here is those that allow their cats the dangerous privilege of roaming free.
***NOTE: The average life span of outdoor cats is four years.  This is due to disease, predation, intentional injury/poisoning, and while hunting they are not paying attention to things like roads and cars.  The average outdoor cat has double the annual vet bills due to these hazards.
The average lifespan of indoor cats is 17 years, most require only annual vaccination, and are quite content and happy indoors.  Many folks are now creating outdoor rooms for their cats - fully wired/meshed in, accessible via a kitty door, so that their indoor cat Has a safe outdoor play area, and is not engaging in hunting for 'sport'. 
The nest fell out of the tree, can we save the eggs/babies?  If it is dusk, or dark, wait until morning, keep the babies warm (100 degrees farenheight, NO more and NO less), and DO NOT, please, do not feed.  If at all possible, replace the nest in the exact spot it fell from.  If the nest is too damaged, find a open weave berry basket, or a plastic container (poked full of holes to prevent collection of rainwater) to place the damaged nest, or as best you can create a nest in and wire it back in place.   Wait several DAYLIGHT hours, observing from a distance (via video camera or binoculars) to ensure the parents have found, and reclaimed their babies.  If not, a wildlife rehabilitator will be needed and should be contacted straight away for delivery instructions. 
Why can't we keep it?  The government realizes that very few people possess the skills and abilities to meet the needs of these creatures, and has decided that they must be placed in the care of those trained in meeting their unique demands.  This is why it is actually illegal in most places to "keep" or "raise" orphaned wildlife. 
A baby bird must be kept at a specific temperature, with specific humidity, and fed a specific diet - the problem is, each baby bird must be identified FIRST so that each ones unique requirements can be met.  Diet alone is a huge challenge as most songbirds feed their babies on a diet of bugs (NOT worms, milk, bread or seeds!).  But each bird uses different bugs in different quantities.  For an ordinary person to have access to the proper diet, let alone the ability to meet the needs of a hatchling (fed every 15 minutes) or nestling (fed every 30 minutes) from dawn until dusk is unrealistic.  They need to be taught to how to self feed - improper diet, caging, and/or handling (creating birds that are 'friendly' to humans or pets) leads to birds that die upon release.