(maint) Set required conf parameters to make benchmarks do something

The backport did not consider that environments needed to be configured
differently than for the new env support in 3.5.