How those who can't speak English would think of those who can speak English

I'm not talking about Americans or any English speaking people. I'm talking about Japanese people.

There are some Japanese people who speak English. And there are many people who don't speak English. It's not strange, is it? Recently, I've recognized that some of those who belong to the latter group are miunderstanding the former group.

For example, even Japanese who speak English with native-like fluency cannot do translation or simultaneous interpreting without any training. Few Japanese who got a TOEIC score of over 950 can write perfect English. To me, these things are quite natural. But, Japanese who are not good at English often say to me, "Why not? You speak English, right?"