Development workflow¶

Note

The instructions here are for developing python-flint e.g. if you want to contribute to python-flint such as by making changes to its code.

For most users it is recommended to install prebuilt binaries from PyPI or conda-forge as explained in Install with pip or conda. If you are just trying to build and install python-flint from source, see Simple build instructions.

Outline of building `python-flint` for development¶

The python-flint project is a Python wrapper around the FLINT library. The FLINT library is a C library that depends on GMP and MPFR. The python-flint codebase is almost entirely written in Cython which is a superset of Python that allows writing C extensions for Python. The Cython code in python-flint is used to be able to call the C functions in the FLINT library from Python.

To develop python-flint we need to be able to build it which means that we need to have the FLINT library installed, and to have a C compiler and also the Python build dependencies such as Cython installed. Installing the FLINT library is covered in Installing the dependencies although you may want to use a local build of FLINT for development as is explained in Building and installing dependencies locally below.

Once the dependencies are installed, python-flint itself needs to be built which happens in four steps:

The Cython code in python-flint is compiled to C code using cython.
The C code generated by cython is compiled to Python extension modules (shared libraries) using a C compiler and these are linked against the FLINT library and other dependencies.
The extension modules and Python files are all assembled into a Python package directory.
The Python package directory is bundled into a wheel (a zip file).

When building to install the next step would be:

The wheel is installed using a Python package installer such as pip.

When building for distribution on PyPI the next steps would instead be:

All of the dependencies such as the FLINT library are bundled into the wheel.
The wheel is uploaded to PyPI.
Users install the wheel from PyPI (pip install python-flint).

The python-flint project uses meson, meson-python and spin to manage steps 1-4 and other development tasks like running tests. For local development, only steps 1-3 are needed and then the package directory is used directly to run tests or other commands. Using meson and spin for developing python-flint is explained below.

Building and installing dependencies locally¶

It can be useful to build and install the dependencies of python-flint locally rather than system-wide. This is useful for development because it allows you to have precise control over the versions of the dependencies used for development without conflicting with system-wide installations.

In the python-flint git repo there is a script bin/build_dependencies_unix.sh which will download and build GMP, MPFR and FLINT and install them under the current directory in a subdirectory called .local. The versions used and the installation directory can be changed by editing the bin/build_variables.sh script. This script is useful for building python-flint on systems where the system-wide FLINT is too old or if precise control over the versions of GMP, MPFR and FLINT is needed (e.g. when developing python-flint and testing different versions). This script is used for building the binaries for PyPI and also takes care of ensuring that GMP and FLINT are built as redistributable shared libraries (this is not the default behaviour of the configure scripts for these libraries and disables some optimisation features of FLINT on some x86_64 micro-architectures).

Since this local installation is not system-wide, see Using FLINT from a non-standard location and the instructions below for how to configure meson to use the locally installed dependencies.

Meson and spin¶

The build system for python-flint uses meson and is configured using meson.build and meson.options files. When installing python-flint with e.g. pip install . the meson-python PEP 517 build backend will instruct meson to build a python-flint wheel that pip will then install.

For development it is preferred not to install python-flint into the active environment but to use meson commands directly to create a local build and then run tests or other commands using the local build. The spin tool is then used as a frontend to meson to simplify common development tasks. We will first explain how to use meson directly and then show how to use spin to simplify the process. This section gives an overview of how meson and spin are used and in the next section we will see how to get started with using these for python-flint development.

To build and install a typical (non-Python) project that uses meson you would run:

meson setup build
meson compile -C build
meson install -C build

These three commands create a build directory, compile the project, and install it and are analogous to the autotools commands:

./configure
make
make install

What each of these commands does is:

meson setup build: Create a build directory called build. Options can be passed to the meson setup command to configure how the project will be built.
meson compile -C build: Build the project and place the built files in the build directory. After the initial build, if some code is changed then meson compile performs an incremental build which is faster then rebuilding from scratch.
meson install -C build: Transfer the built files to the install directory (e.g. /usr/local or somewhere).

In a Python project that uses meson, the meson install command is not usually used like this because the meson build system is typically used to build e.g. a wheel that is then installed using a Python package installer such as pip.

In the spin/meson workflow for Python projects, we would instead “install” the project into a local directory with a command like:

meson install --only-changed -C build --destdir ../build-install

This command installs the project into the local build-install directory which is a subdirectory of the project root directory. For common development tasks like running the tests we need to make it so that Python can import this local build of python-flint which can be done by either setting PYTHONPATH or by changing directory to where the local build is installed:

cd build-install/usr/lib/python3.12/site-packages
python -m flint.test

This will run the tests for python-flint using the local build of python-flint. The spin tool simplifies this process by providing a frontend to meson that can be used to run common development tasks like running the tests. To run the tests using spin you can run:

spin test

This will rebuild the project if necessary and then run the tests using the local build of python-flint. The spin tool will show what commands it is effectively running so you can see what is happening if you want to run the commands directly. In this case spin test is roughly equivalent to:

meson compile -C build  # rebuild
meson install --only-changed -C build --destdir ../build-install
export PYTHONPATH="/path/to/python-flint/build-install/usr/lib/python3.12/site-packages"
python -m pytest --pyargs flint

After any change to the code, common development tasks such as running the tests require the project to be built and installed first. With spin and meson we emulate this without needing to perform a full rebuild and without actually installing the project into any Python environment or system-wide location.

Setting up the development environment¶

First create a fork of the python-flint repository on GitHub. Clone your fork to your local machine using git and then change directory into the cloned repository:

git clone git@github.com:your-username/python-flint.git
cd python-flint

Now add the upstream repository as a remote so that you can pull in changes in future:

git remote add upstream git@github.com:flintlib/python-flint.git

Note

The git URLs with git@ are for SSH access to the repository. If you do not use SSH keys with GitHub then use the HTTPS URLs instead.

It is worth reading the Simple build instructions instructions first because they cover the basic dependencies needed to build python-flint from source and Installing the dependencies. For local development, you may want to install non-Python dependencies such as FLINT locally rather than system-wide in which case the instructions in Building and installing dependencies locally and Using FLINT from a non-standard location are also useful.

It is also useful to use a virtual environment to manage the Python-level dependencies for python-flint so that it is kept separate from other Python environments on your system. You can create and activate a virtual environment using e.g.:

python3 -m venv venv
source venv/bin/activate

Now all commands such as pip and python will use this activated virtual environment. You can install the Python development dependencies using:

pip install -r requirements-dev.txt

This will install the dependencies such as cython, meson, etc that are needed for building and developing python-flint into the activated virtual environment.

The first step in developing python-flint is to build it and the first step in building it is to configure the build using meson setup (or spin build). If you have already installed FLINT system-wide then you can run:

meson setup build

This will check the system for the dependencies needed to build python-flint such as FLINT and GMP and MPFR. It will also check for C compilers and for Cython. If setup was successful then you can now build python-flint with:

meson compile -C build

By default, python-flint’s build configuration will reject newer versions of FLINT or Cython than the ones that are known to work. If you want to override this behaviour (e.g. because you have FLINT or Cython from a newer version or latest git) then you can pass the -Dflint_version_check=false option:

meson setup build -Dflint_version_check=false

If you have installed the dependencies in a non-standard location then you need to tell meson where to find them when running meson setup. For example, if you have installed FLINT in a directory called /some/dir/lib then you can run:

meson setup build \
    --pkg-config-path=/some/dir/lib/pkgconfig \
    -Dadd_flint_rpath=true

This tells meson to look for the pkg-config files such as flint.pc in the /some/dir/lib/pkgconfig directory and to add the /some/dir/lib directory to the runtime library search path in the python-flint extension modules. The add_flint_rpath option may not be needed depending on your OS.

Usually it is not necessary to use meson directly as shown above becuase the spin tool provides a frontend to meson that combines common steps. The spin build command can be used to setup and build the project in one step:

spin build -- -Dflint_version_check=false

# Equivalent to:

meson setup build -Dflint_version_check=false
meson compile -C build
meson install --only-changed -C build --destdir ../build-install

Most spin commands are primarily a wrapper for some other command (not necessarily a meson command) and will pass any additional arguments through. In this case the -Dflint_version_check=false option is passed to the meson setup command.

The spin build command is the one case where it is recommended to use meson directly instead of using spin. For some reason spin build does not always configure the project correctly and so the recommended way is:

meson setup build -Dflint_version_check=false
spin build

After an initial call to meson setup all subsequent tasks can use spin which will automatically rebuild the project when needed. For example, to run the tests you can run:

spin run python -m flint.test

This will build or rebuild python-flint if necessary and then run the tests. This is equivalent to installing python-flint and then running the same command e.g.:

pip install .
python -m flint.test

More generally the spin run command can be used to run any command in the local build environment as if python-flint was installed.

Common development tasks¶

The most common development task is to rebuild the project and run the tests and there are a few ways to do this. The most straight-forward way is

spin test

The spin test command will rebuild the project if necessary and then run pytest. Additional arguments can be passed to pytest by using the -- separator e.g.:

spin test -- -k test_name  # run only tests that match 'test_name'
spin test -- --pdb         # drop into the debugger on test failure

Note though that there are two kinds of tests in python-flint:

The general tests in the flint/test directory.
The doctests in the docstrings throughout the codebase (and also in the docs).

The spin test command only runs the general tests but not the doctests. To run both you can use python -m flint.test when python-flint is installed but in the development environment you can use:

spin run python -m flint.test  # run all tests and doctests

The two most useful spin commands are:

meson setup build: Configure the project.
spin run python -m flint.test: Run all tests and doctests.

Other useful spin commands are:

spin build: Build the project.
spin test: Run the general tests.
spin run: Run a command in the local build environment.
spin python: Start a Python shell in the local build environment.
spin ipython: Start an IPython shell in the local build environment.
spin shell: Start a system shell in the local build environment.
spin docs: Build the documentation.

One other command is provided but not recommended for general development:

spin install: Install the project editably in the active Python environment.

Sometimes it is useful to install the project editably but it can conflict with other spin commands. The editable install uses the same build directory as the spin install and so the normal spin way of doing things is not compatible with the editable install. You can uninstall the editable install using pip uninstall python-flint and then wipe the build directory:

rm -r build

In future perhaps other spin commands could be added to python-flint’s spin configuration.

Measuring code coverage¶

To measure code coverage it is first necessary to build the Cython code with coverage enabled. This can be done by passing the -Dcoverage=true option to meson setup or spin build. Measuring coverage of Cython code does not currently work with spin (issue). However python-flint has a local coverage plugin that can be used to measure coverage of the Cython code in python-flint. There is a script bin/coverage.sh that can be used for this. Its contents are:

spin build -Dcoverage=true
spin run -- coverage run -m flint.test
coverage report -m --sort=cover
coverage html

Note that the setting -Dcoverage=true enables tracing in the Cython code. This considerably slows down the build as well as making python-flint a lot slower to run. The setting is persistent and so needs to be explicitly disabled when no longer needed:

meson setup build -Dcoverage=false

(Note that this is an example where spin build is not used because it does not trigger a rebuild correctly for some reason unlike meson setup.)

Building in release mode¶

Another setting that is worth configuring is the build type. By default meson setup configures a debug mode build which means that the C code is not fully optimised by the compiler. If you want to measure the performance of python-flint then you should build in release mode:

meson setup build -Dbuildtype=release

This will build the C code with full optimisations enabled. Note that building in release mode takes longer than building in debug mode and so it is not always convenient for development. As for the coverage setting, the build type is persistent and so needs to be disabled explicitly when no longer needed:

meson setup build -Dbuildtype=debug

Note that the build type setting here only applies when compiling the C code that is generated from the Cython code. This has no effect on the optimisation level that is used for FLINT or GMP or MPFR. Setting the build type to release only reduces the overhead of the python-flint wrapper code (which may or may not be significant depending on what is being timed).

Differences between meson and autotools¶

Some differences between meson and autotools are worth noting for the benefit of those who are familiar with autotools but not meson. Firstly, the way that meson is intended to be used is that many different build directories can be created like:

meson setup build-debug -Dbuildtype=debug
meson setup build-release -Dbuildtype=release

This allows different configurations and builds to be kept simultaneously. What this means though is that all subsequent commands must be told which build directory to use e.g. meson compile -C build-debug.

The meson configure command can be used to view or change the configuration of a build directory:

meson configure build-debug  # view the configuration
meson configure build-debug -Dsome_option=true  # change the configuration

It is expected that meson setup would only be called once per build directory and that meson configure would be used to change the configuration of an existing build directory:

meson setup build
meson configure build -Dsome_option=true -Dsome_other_option=false

It is still possible to run meson setup multiple times (and does work) but meson complains (needlessly) that the directory is already configured:

$ meson setup build --pkg-config-path=.local/lib/pkgconfig -Dadd_flint_rpath=true -Dbuildtype=debug
Directory already configured.

Just run your build command (e.g. ninja) and Meson will regenerate as necessary.
Run "meson setup --reconfigure to force Meson to regenerate.

If build failures persist, run "meson setup --wipe" to rebuild from scratch
using the same options as passed when configuring the build.

Unlike an autotools ./configure script the configuration options passed to meson setup are persistent and are combined in repeated calls:

meson setup build -Dfirst_option=true
meson setup build -Dsecond_option=false  # first_option is still true

With meson all generated files are placed in the build directory and the source directory is kept clean. This means that rather than running e.g. make clean you can just delete the build directory (rm -r build). Note that the meson setup command has a --wipe option that will delete all of the built files while keeping the configuration options:

meson setup build -Doption=true
...
meson setup build --wipe  # deletes all built files, option is still true