FIFA Laws of the game An indirect free kick is awarded to the opposing team if a goalkeeper, inside his own penalty area, commits any of the following four offences:
* controls the ball with his hands for more than six seconds before releasing it from his possession
* touches the ball again with his hands after he has released it from his possession and before it has touched another player
* touches the ball with his hands after it has been deliberately played to him by a team-mate
* touches the ball with his hands after he has received it directly from a throw in taken by a team-mate.
...
The indirect free kick is awarded from where the offence occurred (see Law 13 - Position of Free Kick).
The suggestion that you only get penalised for picking the ball up or holding the ball is comical. I never realised such a simple rule was so difficult for people to comprehend!