“Because it is disrespectful”
Prolonged eye contact has enduring uncomfortable effects on those who try to get out of their shells. Truth is that whoever told that you are staring too much is probably uncomfortable with eye contact. His or her reaction is different from that of a person who just shies away but it’s out of that same discomfort nonetheless.
In some cultures eye contact with older people is discouraged. This is due to the fact that older folks don’t want to find themselves in psychological battles for dominance with younger folks, particularly those who seem to have high spirits. While a man or woman in one culture may say ‘look at me when I’m talking to you’, other cultures say ‘look down when I’m talking to you!’
How you interpret it is a matter of culture.
Apply this to the idea of “perspective”.