The art therapist in me understands that most people think this way because watercolors is on the very loose end of the art medium continuum. Pencil is at the other end because it is a medium that can be controlled. The majority of people like to be in control and are uncomfortable if not. Thus, watercolors place them in a challenging situation where they are "out of control" and have to do something about it. Most people do not welcome this feeling or the challenge.
Joe Garcia, a famous watercolor artist, does not believe that watercolors is the most difficult medium to use. In his opinion, "Anyone can learn to paint with watercolors; however, a person must be willing to start with the basics. That starting point is the wash. It is the foundation of watercolor painting. Learning to control the wash will open the door to the versatility and creativity of watercolors. This knowledge will allow you to tightly render or loosely interpret your subject."