.. _development_workflow:
Development workflow
====================
.. note::
The instructions here are for developing ``python-flint`` e.g. if you want
to contribute to ``python-flint`` such as by making changes to its code.
For most users it is recommended to install prebuilt binaries from ``PyPI``
or ``conda-forge`` as explained in :ref:`install_pip_conda`. If you are just
trying to build and install ``python-flint`` from source, see
:ref:`simple_build_instructions`.
Outline of building ``python-flint`` for development
----------------------------------------------------
The ``python-flint`` project is a Python wrapper around the `FLINT
`_ library. The FLINT library is a C library that
depends on `GMP `_ and `MPFR `_.
The ``python-flint`` codebase is almost entirely written in `Cython
`_ which is a superset of Python that allows writing C
extensions for Python. The Cython code in ``python-flint`` is used to be able
to call the C functions in the FLINT library from Python.
To develop ``python-flint`` we need to be able to build it which means that we
need to have the FLINT library installed, and to have a C compiler and also the
Python build dependencies such as Cython installed. Installing the FLINT
library is covered in :ref:`installing_the_dependencies` although you may want
to use a local build of FLINT for development as is explained in
:ref:`local_dependency_install` below.
Once the dependencies are installed, ``python-flint`` itself needs to be built
which happens in four steps:
1. The Cython code in ``python-flint`` is compiled to C code using ``cython``.
2. The C code generated by ``cython`` is compiled to Python extension modules
(shared libraries) using a C compiler and these are linked against the FLINT
library and other dependencies.
3. The extension modules and Python files are all assembled into a Python
package directory.
4. The Python package directory is bundled into a wheel (a zip file).
When building to install the next step would be:
5. The wheel is installed using a Python package installer such as ``pip``.
When building for distribution on PyPI the next steps would instead be:
5. All of the dependencies such as the FLINT library are bundled into the
wheel.
6. The wheel is uploaded to ``PyPI``.
7. Users install the wheel from PyPI (``pip install python-flint``).
The ``python-flint`` project uses ``meson``, ``meson-python`` and ``spin`` to
manage steps 1-4 and other development tasks like running tests. For local
development, only steps 1-3 are needed and then the package directory is used
directly to run tests or other commands. Using ``meson`` and ``spin`` for
developing ``python-flint`` is explained below.
.. _local_dependency_install:
Building and installing dependencies locally
--------------------------------------------
It can be useful to build and install the dependencies of ``python-flint``
locally rather than system-wide. This is useful for development because it
allows you to have precise control over the versions of the dependencies used
for development without conflicting with system-wide installations.
In the ``python-flint`` git repo there is a script
`bin/build_dependencies_unix.sh
`_
which will download and build GMP, MPFR and FLINT and install them under the
current directory in a subdirectory called ``.local``. The versions used and
the installation directory can be changed by editing the
`bin/build_variables.sh
`_
script. This script is useful for building ``python-flint`` on systems where
the system-wide ``FLINT`` is too old or if precise control over the versions of
GMP, MPFR and FLINT is needed (e.g. when developing ``python-flint`` and
testing different versions). This script is used for building the binaries for
PyPI and also takes care of ensuring that ``GMP`` and ``FLINT`` are built as
redistributable shared libraries (this is not the default behaviour of the
``configure`` scripts for these libraries and disables some optimisation
features of ``FLINT`` on some ``x86_64`` micro-architectures).
Since this local installation is not system-wide, see
:ref:`non_standard_location` and the instructions below for how to configure
``meson`` to use the locally installed dependencies.
Meson and spin
--------------
The build system for ``python-flint`` uses `meson `_
and is configured using ``meson.build`` and ``meson.options`` files. When
installing ``python-flint`` with e.g. ``pip install .`` the `meson-python
`_ PEP 517 `build backend
`_ will instruct ``meson`` to build a
``python-flint`` wheel that ``pip`` will then install.
For development it is preferred not to install ``python-flint`` into the
active environment but to use ``meson`` commands directly to create a local
build and then run tests or other commands using the local build. The `spin
`_ tool is then used as a frontend
to ``meson`` to simplify common development tasks. We will first explain how to
use ``meson`` directly and then show how to use ``spin`` to simplify the
process. This section gives an overview of how ``meson`` and ``spin`` are used
and in the next section we will see how to get started with using these for
``python-flint`` development.
To build and install a typical (non-Python) project that uses ``meson`` you
would run::
meson setup build
meson compile -C build
meson install -C build
These three commands create a build directory, compile the project, and install
it and are analogous to the autotools commands::
./configure
make
make install
What each of these commands does is:
- ``meson setup build``: Create a build directory called ``build``. Options can
be passed to the ``meson setup`` command to configure how the project will be
built.
- ``meson compile -C build``: Build the project and place the built files in
the ``build`` directory. After the initial build, if some code is changed
then ``meson compile`` performs an incremental build which is faster then
rebuilding from scratch.
- ``meson install -C build``: Transfer the built files to the install directory
(e.g. ``/usr/local`` or somewhere).
In a Python project that uses ``meson``, the ``meson install`` command is not
usually used like this because the ``meson`` build system is typically used to
build e.g. a wheel that is then installed using a Python package installer such
as ``pip``.
In the ``spin/meson`` workflow for Python projects, we would instead "install"
the project into a local directory with a command like::
meson install --only-changed -C build --destdir ../build-install
This command installs the project into the local ``build-install`` directory
which is a subdirectory of the project root directory. For common development
tasks like running the tests we need to make it so that Python can import this
local build of ``python-flint`` which can be done by either setting
``PYTHONPATH`` or by changing directory to where the local build is installed::
cd build-install/usr/lib/python3.12/site-packages
python -m flint.test
This will run the tests for ``python-flint`` using the local build of
``python-flint``. The ``spin`` tool simplifies this process by providing a
frontend to ``meson`` that can be used to run common development tasks like
running the tests. To run the tests using ``spin`` you can run::
spin test
This will rebuild the project if necessary and then run the tests using the
local build of ``python-flint``. The ``spin`` tool will show what commands it
is effectively running so you can see what is happening if you want to run the
commands directly. In this case ``spin test`` is roughly equivalent to::
meson compile -C build # rebuild
meson install --only-changed -C build --destdir ../build-install
export PYTHONPATH="/path/to/python-flint/build-install/usr/lib/python3.12/site-packages"
python -m pytest --pyargs flint
After any change to the code, common development tasks such as running the
tests require the project to be built and installed first. With ``spin`` and
``meson`` we emulate this without needing to perform a full rebuild and without
actually installing the project into any Python environment or system-wide
location.
Setting up the development environment
--------------------------------------
First create a fork of the `python-flint repository on GitHub
`_. Clone your fork to your local
machine using ``git`` and then change directory into the cloned repository:
.. code-block:: bash
git clone git@github.com:your-username/python-flint.git
cd python-flint
Now add the upstream repository as a remote so that you can pull in changes in
future:
.. code-block:: bash
git remote add upstream git@github.com:flintlib/python-flint.git
.. note::
The git URLs with ``git@`` are for SSH access to the repository. If you do
not use SSH keys with GitHub then use the HTTPS URLs instead.
It is worth reading the :ref:`simple_build_instructions` instructions first
because they cover the basic dependencies needed to build ``python-flint`` from
source and :ref:`installing_the_dependencies`. For local development, you may
want to install non-Python dependencies such as ``FLINT`` locally rather than
system-wide in which case the instructions in :ref:`local_dependency_install`
and :ref:`non_standard_location` are also useful.
It is also useful to use a `virtual environment
`_ to manage the Python-level
dependencies for ``python-flint`` so that it is kept separate from other Python
environments on your system. You can create and activate a virtual environment
using e.g.:
.. code-block:: bash
python3 -m venv venv
source venv/bin/activate
Now all commands such as ``pip`` and ``python`` will use this activated virtual
environment. You can install the Python development dependencies using:
.. code-block:: bash
pip install -r requirements-dev.txt
This will install the dependencies such as ``cython``, ``meson``, etc that are
needed for building and developing ``python-flint`` into the activated virtual
environment.
The first step in developing ``python-flint`` is to build it and the first step
in building it is to configure the build using ``meson setup`` (or ``spin
build``). If you have already installed ``FLINT`` system-wide then you can run:
.. code-block:: bash
meson setup build
This will check the system for the dependencies needed to build
``python-flint`` such as ``FLINT`` and ``GMP`` and ``MPFR``. It will also check
for C compilers and for ``Cython``. If setup was successful then you can now
build ``python-flint`` with:
.. code-block:: bash
meson compile -C build
By default, ``python-flint``'s build configuration will reject newer versions
of ``FLINT`` or ``Cython`` than the ones that are known to work. If you want to
override this behaviour (e.g. because you have ``FLINT`` or ``Cython`` from a
newer version or latest git) then you can pass the
``-Dflint_version_check=false`` option:
.. code-block:: bash
meson setup build -Dflint_version_check=false
If you have installed the dependencies in a non-standard location then you
need to tell ``meson`` where to find them when running ``meson setup``. For
example, if you have installed ``FLINT`` in a directory called
``/some/dir/lib`` then you can run:
.. code-block:: bash
meson setup build \
--pkg-config-path=/some/dir/lib/pkgconfig \
-Dadd_flint_rpath=true
This tells ``meson`` to look for the ``pkg-config`` files such as ``flint.pc``
in the ``/some/dir/lib/pkgconfig`` directory and to add the ``/some/dir/lib``
directory to the runtime library search path in the ``python-flint`` extension
modules. The ``add_flint_rpath`` option may not be needed depending on your OS.
Usually it is not necessary to use ``meson`` directly as shown above becuase
the ``spin`` tool provides a frontend to ``meson`` that combines common steps.
The ``spin build`` command can be used to setup and build the project in one
step:
.. code-block:: bash
spin build -- -Dflint_version_check=false
# Equivalent to:
meson setup build -Dflint_version_check=false
meson compile -C build
meson install --only-changed -C build --destdir ../build-install
Most ``spin`` commands are primarily a wrapper for some other command (not
necessarily a ``meson`` command) and will pass any additional arguments
through. In this case the ``-Dflint_version_check=false`` option is passed to
the ``meson setup`` command.
The ``spin build`` command is the one case where it is recommended to use
``meson`` directly instead of using ``spin``. For some reason ``spin build``
does not always configure the project correctly and so the recommended way is:
.. code-block:: bash
meson setup build -Dflint_version_check=false
spin build
After an initial call to ``meson setup`` all subsequent tasks can use ``spin``
which will automatically rebuild the project when needed. For example, to run
the tests you can run:
.. code-block:: bash
spin run python -m flint.test
This will build or rebuild ``python-flint`` if necessary and then run the
tests. This is equivalent to installing ``python-flint`` and then running the
same command e.g.:
.. code-block:: bash
pip install .
python -m flint.test
More generally the ``spin run`` command can be used to run any command in the
local build environment as if ``python-flint`` was installed.
Common development tasks
------------------------
The most common development task is to rebuild the project and run the tests
and there are a few ways to do this. The most straight-forward way is
.. code-block:: bash
spin test
The ``spin test`` command will rebuild the project if necessary and then run
``pytest``. Additional arguments can be passed to ``pytest`` by using the
``--`` separator e.g.:
.. code-block:: bash
spin test -- -k test_name # run only tests that match 'test_name'
spin test -- --pdb # drop into the debugger on test failure
Note though that there are two kinds of tests in ``python-flint``:
1. The general tests in the ``flint/test`` directory.
2. The doctests in the docstrings throughout the codebase (and also in the
docs).
The ``spin test`` command only runs the general tests but not the doctests. To
run both you can use ``python -m flint.test`` when ``python-flint`` is
installed but in the development environment you can use:
.. code-block:: bash
spin run python -m flint.test # run all tests and doctests
The two most useful ``spin`` commands are:
- ``meson setup build``: Configure the project.
- ``spin run python -m flint.test``: Run all tests and doctests.
Other useful ``spin`` commands are:
- ``spin build``: Build the project.
- ``spin test``: Run the general tests.
- ``spin run``: Run a command in the local build environment.
- ``spin python``: Start a Python shell in the local build environment.
- ``spin ipython``: Start an IPython shell in the local build environment.
- ``spin shell``: Start a system shell in the local build environment.
- ``spin docs``: Build the documentation.
One other command is provided but not recommended for general development:
- ``spin install``: Install the project editably in the active Python
environment.
Sometimes it is useful to install the project editably but it can conflict with
other ``spin`` commands. The editable install uses the same ``build`` directory
as the ``spin`` install and so the normal ``spin`` way of doing things is not
compatible with the editable install. You can uninstall the editable install
using ``pip uninstall python-flint`` and then wipe the build directory:
.. code-block:: bash
rm -r build
In future perhaps other ``spin`` commands could be added to ``python-flint``'s
``spin`` configuration.
Measuring code coverage
-----------------------
To measure code coverage it is first necessary to build the Cython code with
coverage enabled. This can be done by passing the ``-Dcoverage=true`` option to
``meson setup`` or ``spin build``. Measuring coverage of Cython code does not
currently work with ``spin`` (`issue
`_). However ``python-flint``
has a local coverage plugin that can be used to measure coverage of the Cython
code in ``python-flint``. There is a script ``bin/coverage.sh`` that can be
used for this. Its contents are:
.. code-block:: bash
spin build -Dcoverage=true
spin run -- coverage run -m flint.test
coverage report -m --sort=cover
coverage html
Note that the setting ``-Dcoverage=true`` enables tracing in the Cython code.
This considerably slows down the build as well as making ``python-flint`` a
lot slower to run. The setting is persistent and so needs to be explicitly
disabled when no longer needed:
.. code-block:: bash
meson setup build -Dcoverage=false
(Note that this is an example where ``spin build`` is not used because it does
not trigger a rebuild correctly for some reason unlike ``meson setup``.)
Building in release mode
------------------------
Another setting that is worth configuring is the build type. By default ``meson
setup`` configures a debug mode build which means that the C code is not fully
optimised by the compiler. If you want to measure the performance of
``python-flint`` then you should build in release mode:
.. code-block:: bash
meson setup build -Dbuildtype=release
This will build the C code with full optimisations enabled. Note that building
in release mode takes longer than building in debug mode and so it is not
always convenient for development. As for the coverage setting, the build type
is persistent and so needs to be disabled explicitly when no longer needed:
.. code-block:: bash
meson setup build -Dbuildtype=debug
Note that the build type setting here only applies when compiling the C code
that is generated from the Cython code. This has no effect on the optimisation
level that is used for FLINT or GMP or MPFR. Setting the build type to release
only reduces the overhead of the ``python-flint`` wrapper code (which may or
may not be significant depending on what is being timed).
Differences between meson and autotools
---------------------------------------
Some differences between ``meson`` and autotools are worth noting for the
benefit of those who are familiar with autotools but not ``meson``. Firstly,
the way that ``meson`` is intended to be used is that many different build
directories can be created like:
.. code-block:: bash
meson setup build-debug -Dbuildtype=debug
meson setup build-release -Dbuildtype=release
This allows different configurations and builds to be kept simultaneously. What
this means though is that all subsequent commands must be told which build
directory to use e.g. ``meson compile -C build-debug``.
The ``meson configure`` command can be used to view or change the configuration
of a build directory:
.. code-block:: bash
meson configure build-debug # view the configuration
meson configure build-debug -Dsome_option=true # change the configuration
It is expected that ``meson setup`` would only be called once per build
directory and that ``meson configure`` would be used to change the
configuration of an existing build directory:
.. code-block:: bash
meson setup build
meson configure build -Dsome_option=true -Dsome_other_option=false
It is still possible to run ``meson setup`` multiple times (and does work) but
``meson`` complains (needlessly) that the directory is already configured:
.. code-block:: console
$ meson setup build --pkg-config-path=.local/lib/pkgconfig -Dadd_flint_rpath=true -Dbuildtype=debug
Directory already configured.
Just run your build command (e.g. ninja) and Meson will regenerate as necessary.
Run "meson setup --reconfigure to force Meson to regenerate.
If build failures persist, run "meson setup --wipe" to rebuild from scratch
using the same options as passed when configuring the build.
Unlike an autotools ``./configure`` script the configuration options passed to
``meson setup`` are persistent and are combined in repeated calls:
.. code-block:: bash
meson setup build -Dfirst_option=true
meson setup build -Dsecond_option=false # first_option is still true
With ``meson`` all generated files are placed in the build directory and the
source directory is kept clean. This means that rather than running e.g. ``make
clean`` you can just delete the build directory (``rm -r build``). Note that
the ``meson setup`` command has a ``--wipe`` option that will delete all of the
built files while keeping the configuration options:
.. code-block:: bash
meson setup build -Doption=true
...
meson setup build --wipe # deletes all built files, option is still true