Fixes#2229.
For dynamic matrices with fixed-sized storage, only copy/swap
elements that have been set. Otherwise, this leads to inefficient
copying, and potential UB for non-initialized elements.
(cherry picked from commit d213a0bcea2344aa3f6c9856da9f5b2a26ccec25)
