Can I replace only the physical card?

No, you cannot replace the physical card alone. To replace the physical card, you must first replace the digital card, and then you can issue a new physical card.