Log a readable message for issues that we know can happen, and exit cleanly.

This avoids unnecessary crash reports for issues we know we can run
into.