I often experience the same behaviour: custom portraits either change around (custom passengers exchange pictures among them) or cannot be loaded at all (default picture replaces your custom one).
Here is a related error message (from output_log.txt):
[USER DATA - PORTRAITS] Decode failed, user portrait doesn't exist: *|0.0000|0.0000|0.9584|1.0000
Hope that some day something will be done about this persistent issue.