2x16GB now, the other 32GB later?

I would like to start with 32GB RAM, with possibility to later upgrade to 64GB.
If I fill only 2 slots with 2x16GB, and leave the other two slots empty, would that affect the compatibility/performance?

My apologies if this is a silly question, but I could not find the answer so far...
Cheers!