pre-commit#

pre-commit is a tool to automatically run software tools, referred to as hooks, when committing code to a versioning control repository like git. See the pre-commit website.

To enable this:

  • Install pre-commit, for instance conda install -y pre-commit.

The following pre-commit hooks are used:

black

An opinionated autoformatter.

check-xml

Checks xml files for proper format.

check-yaml

Checks yaml files for proper format.

clang-format

Format files with ClangFormat (optional).

flake8

A style checker with many different plugins to enforce different rules.

format-xmllint

Feed all XML files through xmllint.

isort

An opinionated import sorter.

mypy

Performs type checking on the code (optional).

ruff

An extremely fast Python linter, written in Rust.

ts_pre_commit_conf#

ts_pre_commit_conf is a tool to generate configuration files for the pre-commit hooks for the Telescope and Site SoftWare (TSSW) team at the Vera C. Rubin Observatory.

These configuration files facilitate pre-commit to maintain black formatting, flake8 compliance and isort imports sorting. If mypy is used, a configuration file can be generated for that as well. The configuration files are generated by using the generate_pre_commit_conf command. Since TSSW uses conda for their software installations, a conda recipe named ts-pre-commit-config was created for this project and the conda package was made available in the lsstts conda channel. To install it, run conda install -y -c lsstts ts-pre-commit-config.

In order for the generate_pre_commit_conf command to know which pre-commit hooks need to be applied, a .ts-pre-commit-config.yaml configuration file needs to be created. This .ts-pre-commit-config.yaml file contains information that indicates for every pre-commit whether it should be applied or not. The .ts-pre-commit-config.yaml configuration file can be generated using the generate_pre_commit_conf --create command, or manually. For more information on the command line options, use generate_pre_commit_conf --help. An example where all pre-commit hooks are enabled is:

black: true
check-xml: true
check-yaml: true
clang-format: true
flake8: true
format-xmllint: true
isort: true
mypy: true
ruff: true

The generate_pre_commit_conf command fails with a comprehensive error message if a mandatory or optional pre-commit hook is missing. The black, check-xml, check-yaml, flake8 and isort hooks are mandatory for TSSW projects, so those hooks need to be set to true. Setting one or more of the mandatory hooks to false will make the generate_pre_commit_conf command fail with a comprehensive error message. The clang-format, format-xmllint, ruff and towncrier hooks are optional and do not get included by default. They can be included by using the corresponding --with-XXX command line option. The mypy hook is optional and gets included by default. It can be excluded by using the corresponding --no-mypy command line option. Setting one or more of the optional hooks to false or omitting them from the .ts-pre-commit-config.yaml file will make the generate_pre_commit_conf command skip those hooks.

It is possible to add custom pre-commit hook configuration files. This may be necessary in case the TSSW configuration doesn’t match what is required for the project. In such a case, the configuration file needs to be committed to git. By default, the generate_pre_commit_conf command skips overwriting existing hook configuration files. If the hook configuration files need to be overwritten, for instance when the TSSW configuration rules change, then the --overwrite command line option can be used.

The configuration files will be updated whenever the pre-commit hooks get updated. Apart from that, all generated configuration file names get added to .gitignore as well. The TSSW Jenkins CI jobs update these configuration files and execute pre-commit every time the jobs run, so you know that you need to update them locally and fix your code if a job fails. You can update the config file locally by running the generate_pre_commit_conf command again.

Projects with a .ts_pre_commit_config.yaml file#

There are several situations in which a project already may have a .ts_pre_commit_config.yaml file. One is when another developer has configured the project and you have cloned the repo. Another is when a project was configured by you and one or more pre-commit hooks were updated. A third is when the project previously didn’t use clang-format or mypy and now it does.

In all cases the config files for pre-commit and the hooks can easily be created or updated. All that needs to be done is to run the generate_pre_commit_conf command without any arguments. The generate_pre_commit_conf command will read the existing .ts_pre_commit_config.yaml file and use that to do its work. Note that this will overwrite the existing .gitignore and .pre-commit-config.yaml files as well as any existing config file for the pre-commit hooks.

Projects without a .ts_pre_commit_config.yaml file#

For a project that already has a .pre-commit-config.yaml configuration file, execute these steps:

  • Inspect the .pre-commit-config.yaml file to see if the project uses clang-format and/or mypy or not.

  • Remove the .pre-commit-config.yaml file with git rm --cached .pre-commit-config.yaml.

  • Remove setup.cfg if it exists with git rm setup.cfg (note: this file has been replaced by .flake8).

  • Update conda/meta.yaml to remove setup.cfg, pytest-flake8, and pytest-asyncio, if present. See Conda for information on what conda/meta.yaml should contain.

  • Update pyproject.toml to remove all linter configuration. This includes all flake8 options and the addopts line in the tool.pytest.ini_options section. See Python for information on what pyproject.toml should contain.

Then, in all cases, execute these steps:

  • Install ts_pre_commit_conf if not already done, as per the installation instructions above.

  • Execute generate_pre_commit_conf --create if clang-format and mypy are used. Use the --no-clang-format option to exclude clang-format and the --no-mypy option to exclude mypy.

  • Add the newly created .ts_pre_commit_config.yaml to git with git add .ts_pre_commit_config.yaml.

  • Run the pre-commit hooks on all of your code, using pre-commit run --all-files. If this changes anything, fix as needed:

    • Fix mypy errors.

    • If isort changes any __init__.py files, run unit tests and fix any breakage. Other isort changes should be innocuous, but it never hurts to run unit tests.

    • Changes made by black should never break anything.

  • Once this is all done, create a git commit to reflect the change with git commit -a -m "Use ts_pre_commit_conf.".

Adding a new hook#

In order to add a new hook, do the following:

  • Create a new ticket branch in the ts_pre_commit_conf project following the development-workflow.

  • Edit the lsst/ts/pre_commit_conf/pre_commit_hooks.py file.

  • Add a new entry to the registry dict providing the following information:

    • pre_commit_config: the config for the .pre-commit-config.yaml file. Provide this as a triple quoted string without leading or trailing whitespace apart from a newline character at the end. See the other hooks in the registry for examples.

    • config_file_name: the name of the config file of the hook, or None if the hook doesn’t require a config file.

    • config: the config file contents as a string. Provide this as a triple quoted string without leading or trailing whitespace apart from a newline character at the end. See the other hooks in the registry for examples. Note that this needs to be set to None if config_file_name is set to None.

    • rule_type: this can be one of

      • MANDATORY: The hook is mandatory for all TSSW projects. Adding such a type of hook needs to be discussed at TSSW standup first.

      • OPT_IN: The hook is optional and does not get included by default. The majority of the hooks will be of the OPT_IN rule type.

      • OPT_OUT: The hook is optional and gets included by default. Adding such a type of hook needs to be discussed at TSSW standup first.

Note that config_file_name and config may be omitted when they are None and optional and excludable when they are False since None is the default value.