Developer’s guide#

Working with the source code repository#

Dependencies#

non-Python#

  • A C++ compiler that supports C++14 or later. On Linux, gcc is preferred, although clang is very useful to compare against. On macOS, use clang.

  • cmake

  • The GNU Scientific Library, version 2.3 or later.

  • The development files (headers) required for your Python installation. For example, on Ubuntu-based Linux distributions, you need the libpython3-dev package.

  • The rust programming language. If you are working in a conda environment, install rust through conda. If not, see here, or use the appropriate package for your operating system. For example, on Fedora Linux, sudo dnf install rust will get you started.

  • Your rust environment needs cbingden:

cargo install cbindgen

The C++ test suite requires:

  • boost. Specifically, the boost test libraries are required. There are many ways to install boost, but conda or your OS packages will be the most reliable.

Warning

conda users must install all of these dependencies using that system, including the compilers! See here for info about conda compilers for Linux and for macOS.

The requirements/ folder of the source code repository contains a list of minimal conda dependencies that we use for CI.

Python#

To reproduce the environments that we use for CI and deployment:

python3 -m pip install -r requirements/development.txt

Note

It may be useful to add --upgrade --no-cache-dir to the above command.

Building the code for development#

To generate an “editable” (in-place) installation for development work:

git submodule init
git submodule update
python -m pip install -e .

The above command is equivalent to the deprecated python setup.py build_ext -i.

Installing from a clone of the source repository#

Install the build module if you have not:

python -m pip install build

Then:

# from the repository root
python -m build .

The install the wheel found in dist/.

Installing from the source distribution at PyPi#

With all of the dependencies in place, tell pip to ignore the binary wheels:

# from the repository root
pip install fwdpy11 --no-binary

Running the Python test suite#

After completing an editable build (see above):

python3 -p pytest tests -n 6

Replace 6 with a number of threads appropriate for your machine.

Advanced build methods using cmake#

Building for development#

We need to compile in release mode and enable building Python modules required by the test suite:

cmake -Bbuild -S. -DCMAKE_BUILD_TYPE=Release -DBUILD_PYTHON_UNIT_TESTS=ON
cmake --build build -j 6

This method can be preferable to the Python commands shown above because you get full control over parallelism.

Building the core library on its own.#

Doing this is useful for C++-level testing:

cmake -S. -Bbuild -DCMAKE_BUILD_TYPE=Release
cmake --build build -t fwdpy11core -j 6

Building and running the C++ test suite.#

Warning

The commands in this section build both the Python module and all related libraries in debug mode. While this is very useful for running the C++ tests, there are side effects! For example, the Python tests will become unbearably slow. (The build will place the debug-mode Python module in place so that it will be used for the Python tests.) Further, the docs will not build because the examples will time out due to the unoptimized build.

cmake -Bbuild_cpp_tests -S. -DBUILD_CPP_UNIT_TESTS=ON
cmake --build build_cpp_tests -j 6
cmake --build build_cpp_tests -t test
rm -rf build_cpp_tests

It may be useful to force the C++ code into debug mode, which will enable more assert statements from fwdpp and exceptions from the fwdpy11 code:

cmake -Bbuild_cpp_tests -S. -DBUILD_CPP_UNIT_TESTS=ON -DCMAKE_CXX_FLAGS="-UNDEBUG -O -g"

Enabling compiler command support for developing the C++ code#

Many language servers support processing a local file called compile_commands.json. See here for background.

To generate this file, execute the following steps from the root of the source code repository:

cmake -Bccommands . -DCMAKE_EXPORT_COMPILE_COMMANDS=1
mv ccommands/compile_commands.json ..
rm -rf ccommands

Now, language servers supporting clangd will have nice error checking and code completion for the C++ code!

C++ code standards#

This is a large topic. Briefly,

  • Functions should not be concerned with ownership! If an input argument cannot cannot be nullptr, then pass it by reference. (Or by value if it will be consumed.) Likewise, smart pointers are not function arguments unless they will be consumed. If the calling environment manages an object with a smart pointer, and nullptr is not a valid value to pass to a given function, then the function should be written to take a reference.

  • If a pointer type is an argument, nullptr must be checked for and handled.

  • Always use smart pointers instead of raw allocations.

  • Classes should respect the “rule of 5”. For abstract base classes, this usually means that the copy constructors and assignment operators must be marked delete to avoid slicing. (In practice, we don’t have any code that could lead to slicing, but we design the classes with this in mind.)

  • Virtual functions in derived classes are marked override. Where appropriate, they are also marked final.

Enabling code profiling#

By default, fwdpy11 is compiled with aggressive optimizations to help reduce the library size. One side effect is that it becomes impossible to accurately profile the code. To override these defaults:

cmake -DENABLE_PROFILING=On -Bbuild -S.
cmake --build build

Note

The package should not be installed with profiling enabled. This method of building is for developers who need to accurately profile the C++ back-end. Also note that only the main package is affected. Building the unit test modules is not affected.

Writing documentation#

We use jupyter book to write the manual. This tool uses a flavor of markdown called MYsT. At the moment, there aren’t many tools available to do things like auto-format code, etc..

The dependencies for the documentation are described above.

Building the documentation#

cd doc
make

Then, point your browser to _build/html/index.html.

To remove the manual:

make clean

Remember to cd .. to get out of the doc/ folder and get back to the repository root when you’re done.

Docstrings in C++ code#

We are moving away from docstrings in C++ code. Where possible, we prefer to either have a Python class inherit from a pybind11 class. Alternately, encapsulation of the pybind11 class in a Python class is a possibility. Then, the Python classes contain the docstrings.

For functions, we apply the same idea, defining a Python function that calls the low-level C++ function.

Although this is a bit more work, it provides better support for tools like mypy, jedi, etc..

Deprecating Python features#

When deprecating something, we want to:

  1. Stop testing it

  2. Stop documenting it

  3. Remove all examples of its use

In theory, doing this is simple. We can just run python with -We and the deprecation warnings will become errors. However, this does not work well in the manual.

To address this problem, we take a two part approach when deprecating. Consider the following class:

@dataclass
class Foo:
    x: int

To deprecate it:

from deprecated import deprecated

# This is our DeprecationWarning.
# The Python test suite will now fail.
@deprecated(reason="a good one")
@dataclass
class Foo:
    x: int

    def __post_init__(self):
        # For cases like our manual,
        # we simply terminate execution
        # whenever this type is instantiated
        assert False

Our approach involves:

  • A warning and a hard error at the point of deprecation

  • Update all tests, docs, etc., to stop using this type.

  • When the previous step passes, remove the assert False