[AyanSinhaMahapatra]

This PR adds a new compact JSON output to minimize scan results filesize by ensuring this option skips duplicate and non-essential attributes. This implements the following for making the scan results compact:

1. remove all duplicated/inline data present in multiple levels (example: removes license detection matches in resource-levels and only keeps top-level unique license detections which makes sure data for a detection is only present once)
2. skip all duplicated codebase/resource attributes (example: keep only one license expression, instead of both scancode/spdx expressions)
3. skip all large/non-essential codebase/resource attributes (example: remove description, license text, diagnostics and other fields, which can be fetched from the original file too if required)
4. remove all empty attributes recursively (keep only the key/value pairs which are non-empty)

Fixes #4555

Tasks

• Reviewed contribution guidelines
• PR is descriptively titled 📑 and links the original issue above 🔗
• Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
  Run tests locally to check for errors.
• Commits are in uniquely-named feature branch and has no merge conflicts 📁
• Updated documentation pages (if applicable)
• Updated CHANGELOG.rst (if applicable)

[JonoYang]

You have some test failures, maybe some expected results need to be updated:
https://dev.azure.com/nexB/scancode-toolkit/_build/results?buildId=19559&view=logs&j=ab555dbb-56bb-5376-00ab-57e873ae1b15&t=de085b8d-0c41-5334-b374-922272c4f743&l=15038
https://dev.azure.com/nexB/scancode-toolkit/_build/results?buildId=19559&view=logs&j=98269aaf-545e-5e24-84bd-2aa29b2a994f&t=ab3eb585-8ac9-5c58-be9d-ada6c8dcd8ce&l=486

[AyanSinhaMahapatra]

@JonoYang this was quite a headscratcher of a failure, I thought first this was due to old entrypoints being present but actually what was happening is an old version of scancode-toolkit getting installed from pip here and this having different entrypoints, could not load the plugins properly anymore.

This should be fixed now.