I don't understand why anyone would leave the keycard behind—even in a perfect scenario, you can be stranded by something as simple as your phone battery dying, or dropping your phone and breaking it.
I know at one point early on I had to completely remove all Rivian Bluetooth pairings and reset up the app once. I'm not sure what steps you tried, but in this case, I'd probably remove the phone from within the truck (I think you can), remove the app completely, and delete all Rivian bluetooth pairings. Maybe even reboot the phone once. Then re-install the app and set it up from scratch.
I know that when my PaaK gets flaky (like the truck won't unlock), I've learned the best fix is to reboot the phone, then start the Rivian app. This seems to improve responsiveness for awhile.