Can you try this experiment on a random file on an XFS partition outside of swift? This sounds like a bug in the ls command or possibly in XFS.
What are the options you specify for? More specifically, why the double 'l' (read: ell) in the command ls -llAh?