(#12572) Revert fix for #7106 and implement a more minimal fix
The original fix for #7106 (last_run_summary.yaml only readable by root) used
a catalog to write out the last run summary. This created extra output if
show_diff is turned on. This reverts that change, and adds a chmod call to set
the mode based on settings.