When you think you are making progress in a language it can be a little depressing when you come across some that you don't understand and feel that you should have. Actually I have observed and worked out a number of valid reasons not to feel so bad. So next time don't beat yourself up.
You can't hear it properly
This one is two fold: firstly when eavesdropping on passersbys it is very difficult to catch what they are saying, even in your mother tongue (try it), particularly if the ambient noise level is high. The second reason can be when you are watching TV or a film that is intended for English speakers. The director may have included some Chinese talking but expects you to get the meaning from context or convenient subs. therefore the sound man doesn't actually go to the trouble of making sure the voices are clear. You may feel disappointed you missed some basic stuff but replay it and you discover the sound quality of the conversation is so poor you didn't have much of a chance anyway.
You are hearing something out of context
Even if you are good, you are going to struggle with out of context snippets, as above this can even cause confusion in your mother tongue.
They are speaking another dialect
Perhaps even another language. Particularly with Chinese, remember that what you may be hearing is a dialect they are more at home speaking in. Dialects close to Mandarin can be a particular problem, they sound familiar somehow but you don't quite get it. You would need some heavy exposure to get used to it. Having said that I think is is good to listen to target language enough to at least determine whether what you are listening to is in the right ballpark.
They don't want you to understand
Yup it happens don't take it personally, for whatever reason (maybe testing you, maybe suspicion etc.), they are actively trying to mess you up with funky language and or speaking very fast.
They don't speak very well themselves
This happens too, they told you they spoke Mandarin (perhaps didn't expect you to know much) and actually their Mandarin isn't so good, making it harder for you to understand
You weren't actually ready for it
Your are a learner, your window of opportunity suddenly appeared and you missed it, sometimes you may catch it, there are times you will be tired, focused on something else and your second language skills will be comparitively poor.
The language is highly specialised
This will happen a lot, you just haven't any experiance of the vocabulary and context being used, or the film is a specialised genre etc. etc. Think of your own language, which is easier for a foreigner do you think, a standard chat-show or a "gangsta" movie.
Of course you may just need to practice more ;) but even you are feeling bad about an experiance then don't, there is always another opportunity. Don't get me wrong this is not setting myself up for failure just providing reasons to remain confident. I am sure there are many more, do you have some you can share?