Meson as a backend build system for F2PY Link to heading

Last week we discussed F2PY’s frontend and how it is undergoing re-implementation using the argparse python library. This week we will look at the backend build system of F2PY, and how meson will be used to build the extension module.

What is the use of a build system? Link to heading

Simply explained, build system exist to automate compilation and linking of big programs. A program’s source code might consist of several hundred C++ files, some C and a few Fortran files. It might also contain library dependencies which will be provided by some object files. A build system takes a simple configuration file, where user can specify the source files, libraries, and other dependencies. The build system will then compile and link the program, and provide the executable.

But can’t we do that using a simple bash script? Link to heading

Aha, this is a good question. Compiling and building programs from scratch take a lot of time. Imagine you are a software developer in 1997 working in a small division in IBM. Your project contains ten thousand source C files. Building the entire program from scractch takes 2 hours. You saw a bug in one of the source files, you changed some code and started building it using the shell script to test it. Even though you made changes in only a single file, the shell script will delete the previous build (or move it to archives) and create the new build from scratch, taking the full stretch of 2 hours. It can’t identify the changes in source code and only rebuild the changed files.

Here the build systems come into play, they can create a dependency tree of your source code, and only rebuild the changed files. If your manager allows, you can create a CMake configuration file, include all the source files and dependencies, and code down how your want to build your project. For the first time, CMake will obviously take a lot of time to build the project (around the same 2 hours). But the next time you make changes, and rebuild using CMake, it will smartly detect the changes and only recompile and link what is necessary.

Build systems make life vastly easier for developers. A lot of them provide cross-platform building capabilities, will detect external libraries for you, install the built program for you, run tests, generate documentation, and much more. Thats why every major developing software uses a populare build system for distribution.

Why does F2PY need a build system Link to heading

Most of the libraries written for Python are actually written in C and wrapped by Python C API (for ex - NumPy, Pandas, TensorFlow). This RealPython article on how to build a Python C Extension Module is a good place to start if you want to build your own Python library in C. But if your want to build a Python library using Fortran, F2PY is a go-to tool.

Python does not have any official supported API for Fortran. But then how does F2PY create Python extension modules from Fortran sources? The way it pull this off is by intelligently understanding the Fortran source code and generates a C wrapper for it. This C wrapper is in turn wrapped by Python C API and thus it can be understood and used by the Python interpreter.

The C wrapper and fortran source files need to be compiled and linked by a compiler family to produce a shared module (.so file). Here is where the build systems come in. They compile and link the source code and F2PY generated C wrapper. We can use various build systems to do this job. F2PY’s documentation explains how to use F2PY with some famous build systems.

What would we need from a build system to support F2PY? Link to heading

F2PY provides a great deal of freedom to the user of how they was to produce their build. If you go through its module building manual, you can find a ton of flags and variables you can set in the command line to create a build perfect for your needs. F2PY currently uses NumPy’s distutils build system, which is highly flexible system for build Python libraries. F2PY leverages unmatched flexibility provided by numpy.distutils to provide user such choices.

Unfortunately, distutils will be deprecated which means we have to find a new build system. CMake and Meson are great contenders, and we choose Meson because it is simple, has great documentation, and provides us enough features to maintain F2PY’s all core building functionalities. This github issue by Ralf Gommers is an excellent source to get insights of why Meson is the better alternative for many Python libraries.

Meson as F2PY’s backend build system Link to heading

For the next section, I recommend you go through this small article on how to use Meson with F2PY. Like most other build systems, meson uses a config file: meson.build to understand how the user wants the project to be built. You can go through Building a module doc and see all the functionalities we need to cover with a build system. The table below explains what the NumPy’s developer team as agreed upon now and how Meson will implement F2PY’s building functionalities.

Suppose you enter the following command to produce an extension module from your fortran sources:

f2py source1.f source2.f object.o -c --fcompiler=gfortran --compiler=gcc \
	--inclue-paths /usr/local/ --f77flags --blah=0 --link-atlas --debug \
	--noarch -L/some/path/ -lsomelibrary -Ddebug -UFINT=12

The equivalent meson.build file will be:

project('f2py_examples', 'c',
  version : '0.1',
  default_options : ['warning_level=2'])

add_languages('fortran')

py_mod = import('python')
py3 = py_mod.find_installation('python3')
py3_dep = py3.dependency()
message(py3.path())
message(py3.get_install_dir())

incdir_numpy = run_command(py3,
  ['-c', 'import os; os.chdir(".."); import numpy; print(numpy.get_include())'],
  check : true
).stdout().strip()

incdir_f2py = run_command(py3,
    ['-c', 'import os; os.chdir(".."); import numpy.f2py; print(numpy.f2py.get_include())'],
    check : true
).stdout().strip()

add_global_arguments('--blah=0', language : 'fortran')

fibby_source = custom_target('fibbymodule.c',
                            input : ['source.f'], # .f so no F90 wrappers
                            output : ['fibbymodule.c', 'fibby-f2pywrappers.f'],
                            command : [ py3, '-m', 'numpy.f2py', '@INPUT@',
                            '-m', 'fibby', '--lower'])

inc_np = include_directories(incdir_numpy, incdir_f2py)
inc_dir1 = include_directories('usr/local/')

dep1 = dependecy('atlas')

py3.extension_module('fibby',
           'source.f',
           fibby_source,
           incdir_f2py+'/fortranobject.c',
		   'object.o'
           include_directories: [inc_np, inc_dir1],
           dependencies : py3_dep,
           install : true,
		   link_args : ['-L/some/path/', '-lsomelibrary'],
		   c_args : ['-Ddebug' '-UFINT=12'])

This will be run as follows:

FC=gfortran CC=gcc meson setup builddir -Ddebug=true -Doptimization=2

Support for --help-fcompiler will be discussed later seperately. Its Meson implementation is rather difficult and requires more discussions. This covers our core operations provided by the F2PY CLI, and soon we will implement a program for generating meson file as described above.