[backport] Serialize the filehash DoS tests - #2465

Cannot build an overlay database because build-mode is set to "undefined" instead of "none". Falling back to creating a normal full database instead.