The Lowdown:
The reason for "Canon" is that Trek as a franchise has spawned so many episodes, films, novels, CD-Roms, and whatnot that there has been a lot of contradiction and discontinuity; this is also largely because the production of the novels (handled by Pocket Books) is run separately from the production of the TV show and the films. So there has to be a decision as to which of two contradictory sources (Is it William Thelonious Riker--from the novel "Imzadi"--or William Thomas Riker--first established in the TNG ep "Second Chances"?-- to cite one teensy-weensy example) is the "real thing". This is not as much for the benefit of readers and viewers (who are free to decide what's part of the Trek universe in their own minds) as it is for the writers of future Trek novels and scripts who may need to supply accurate background information.
Take the ST novel "Federation," by Michael Jan Friedman, for example, published around the time "Generations" hit the theaters. It's a completely different (and superior, IMHO) retelling of the invention of warp drive by Zefram Cochrane than was told two years later in "First Contact." If I'm writing the next episode of Voyager, or the next TNG novel, and I want to have them meet up with Zefram Cochrane, which Cochrane would I need to base it on--the "Federation" Cochrane, or the "First Contact" Cochrane? Because there is such a thing as Canon, the decision is made more obvious for me (I'll have to use the FC Cochrane; if there are elements of the Federation Cochrane I want to use, I'll have to reconcile them somehow with the FC Cochrane), and I won't get laughed out of Paramount Studios or Pocket Books' offices for offering a story that many fans would reject because it diverges too much from what most people accept as the Star Trek universe.
A General Definition of "Canonicity":
CANON:
1) The episodes of TOS, TNG, DS9, VOY, and ENT
2) The ten motion pictures
3) Elements of the Animated Series
episode "Yesteryear," by D.C. Fontana, as "canonized" by some episodes
of TNG.
NON-CANON:
1) All novels
2) The Animated Series, except for
"Yesteryear"
3) CD-ROM's, games, and other spin-off
products.
4) Fan-created works
Obviously, the list above is not a precise definition of canon, since there's plenty of contradiction *between* canonical episodes as well (matters of O'Brien's rank, Spot's sex, etc... many of the things we discuss in these pages) in which case, newer episodes would be "more canonical" than older ones. One wouldn't write a script now which explicitly gives O'Brien's rank as "lieutenant," when the current canon has it as Chief Petty Officer, regardless of early TNG episodes that say otherwise. Gene Roddenberry also considered ST 5 and parts of ST 6 non-canonical, but this is a case where the creation seems to have a bit more standing than its creator, and although it's unlikely (and it would probably be discouraged) to write an episode that relies on ST 5 as its foundation for its plot, it would probably stand up against GR's wishes. Reference material (the Encyclopedia, the Tech Manual, the Chronology), though based on canon, is not canon itself (part of the original edition of the Chronology was disregarded in favor of "Star Trek: First Contact", for example). So it's not a black-and-white issue; "canon" has many gray areas.
Who decides what's canon? Well, given the definition that canon is "stuff that can be used as the basis of future canonical Star Trek material" (yes, it's a circular definition but you know what I mean) that responsibility falls to one group of people alone...the CURRENT producers of the Star Trek series and films: at this point, it's Berman, Braga, et al. The reason why "Mosaic" and "Pathways" were once canon isn't that Voyager producer Jeri Taylor wrote them, it's because Taylor, as a producer, decided they were canon (elements of "Mosaic" were further canonized by their mention in the Voyager series); the fact that she wrote them, though a major factor, was not the deciding factor. However, now that she's no longer the final word on Star Trek, the novels have effectively been "decanonized" to some degree--I'm sure future novelists will feel a little more justified to use a "Mosaic" reference in their work than one from another Voyager novel, but future films and episodes are not constrained from contradicting "Mosaic" either. Truth is, Berman could wake up tomorrow and decide to make "The Return" canon because he thinks it has some bearing on the next Star Trek movie's storyline. Not likely to happen, though.
So, the purpose of canon isn't to limit the possibilities for fans. Like I mentioned before, each fan decides for himself and herself what's canon for themselves, though they should realize that others may have a different opinion (in *my* idea of a Trek universe, for example, certain novels like "Vendetta," "Sarek," and the Shatner Trek novels are "canon"). The "Production Canon" is an attempt to define a coherent Star Trek continuity for the benefit of those who would add to the continuity. I think that if Paramount had kept a tighter creative rein on its novels and spinoff products and made an effort to preserve continuity throughout (like Lucas did, with somewhat more success, with the SW franchise), this wouldn't have been a problem, but because of the sheer size of the ST franchise, rules for canon had to be made.