He played with a special educational operating system called Minix that was a small version of Unix. But it, too, had limitations, including a restrictive license that he felt stifled improvement. After a while, he decided to write a new operating system that would take better advantage of his hardware. His stated goal at the time was to "write a better Minix than Minix".
For more information on Linus Torvalds, look here. For some more details on the early development of Linux, see this article.