Kafei is to marry Anju, the sweet, red-headed innkeeper, but he made quite a hefty engagement promise: Anju has the Moon's Mask, and he vowed to bring the Sun's Mask to the wedding. When put together, the two create the Couples' Mask (also known as "The Mask of the Wedded Couple").
The next part of his tale is a disputed case. Kafei tells Link that he was turned into a child by an "imp with a smiling face", or something of that sort. The first theory is that Kafei stole the mask from the Happy Mask Man (who is pretty much always smiling) and was caught, but this really doesn't fit with Kafei's character. The more plausable idea is that Skull Kid cast the spell. Not only have several people referred to him as an "imp", but Majora's Mask has a face drawn onto it. Skull Kid has also been causing trouble all over town.
The townsfolk must have seen Kafei in his boyhood state, because he tells Link and his fairy, Chat, that he was "targeted because of what he'd become" before the wedding. Hence, he donned his Keaton mask (a gift from his old friend at the Curiosity Shop) and hid out at the laundry pool. Now, only the Postman knows where he is, and he's obviously been sworn to secrecy. Of course, Link can ring the bell by the stream and make Kafei appear for a few seconds, but the game isn't going to make revealing his location that easy.