Working with packages using setuptools_dso¶

Packages using setuptools_dso can for the most part be treated like any other python package. Distribution through pypi.org as source and/or binary (wheels) is possible. Like other packages containing compiled code, use of egg binaries is not supported.

Use of PIP is encouraged. The setuptools_dso package needs to be installed when manually invoking setup.py scripts in dependent packages. eg. to generate source tars with setup.py sdist.

Developers wishing to work with an in-place build will need to use the added build_dso command, which functions like the build_ext command.

python setup.py build_dso -i
python setup.py build_ext -i

NUM_JOBS¶

Use the $NUM_JOBS environment variable to override the default concurrency when compiling object files for DSOs. eg. export NUM_JOBS=1 for a sequential build.

Applying to your package¶

The example/ demonstrates building a non-python library, and linking it with a python extension module.

To properly support pip install ..., it is recommended to include a pyproject.toml file containing at least:

[build-system]
requires = ["setuptools", "wheel", "setuptools_dso"]

This ensures that setuptools_dso is available to be imported by setup.py.

Add a MANIFEST.in to ensure that setup.py sdist includes everything necessary for a successful source build.

include pyproject.toml
recursive-include src *.h *.c *.cpp

Using of auditwheel with wheel builds from setuptools_dso can cause problems as auditwheel has no concept of non-python libraries in python packages.

If SETUPTOOLS_DSO_PLAT_NAME contains a manylinux name, setuptools_dso will inject this platform name when building wheels. Currently PIP version <=21 will pass this environment variable through to PEP 517 builds. eg.

SETUPTOOLS_DSO_PLAT_NAME=manylinux1_x86_64 pip wheel -w dist .

Building a DSO¶

setuptools_dso.setup(..., x_dsos=[DSO(...)])¶

Wrapper around setuptools.setup() which injects extra Commands needed to build DSOs. Unknown keywords are passed through.

Parameters:: x_dsos – None, a list of DSO instances, or a callable with one argument returning such a list.

class setuptools_dso.DSO(name, sources, ..., dsos=[DSO(...)], soversion=None)¶

A Dynamic Shared Object to be built. Accepts the same options as Extension with additions:

Parameters:

dsos – A list of DSO s which this DSO will be linked against.
soversion (str) – None (default) or a string. On supported targerts, this string which will be appended to create the SONAME. eg. soversion="0" will turn libfoo.so into libfoo.so.0.
lang_compile_args (dict) – Language specific (“c” or “c++”) compiler flags. eg. {'c':['-DMAGIC']}
gen_info (str) – Controls generation of “info” module. True (default) uses the conventional filename, False disables generation, or a specific filename string.

The example source files while make up the non-python demo library are: mylib.h, foo.c, bar.cpp. This library will be expressed as a setuptools_dso.DSO object. The first argument is a directory path and library base name encoded like a python module. eg. the result of dsodemo.lib.demo will be eg. dsodemo/lib/libdemo.so or dsodemo\lib\demo.dll installed in the python module tree along-side any other python code or C extensions.

Note that there need not be a dsodemo/lib/__init__.py as dsodemo.lib need not be a python package. However, if this file is present, then the generated dsodemo/lib/demo_dsoinfo.py will be accessible.

from setuptools_dso import DSO, Extension, setup

dso = DSO('dsodemo.lib.demo', ['src/foo.c', 'src/bar.cpp'], ...)

setup(
    ...
    x_dsos = [dso],
    zip_safe = False, # setuptools_dso is not compatible with eggs!
)

The setup(x_dsos=... argument may be either None (default), a list of DSO instances, or a callable returning such a list. A callable will be invoked during the build_dso phase with one argument of distutils.core.Command. (see Probing Toolchain)

The DSO constructor understands the same keyword arguments as setuptools.Extension and distutils.core.Extension, with the addition of dsos=[...], soversion='...', and lang_compile_args={'...':'...'}.

The dsos= argument is a list of other DSO names (eg. 'dsodemo.lib.demo') to allow one DSO to be linked against others.

eg. dsos=['some.lib.foo'] will result in something like gcc ... -L.../some/lib -lfoo.

Building an Extension¶

class setuptools_dso.Extension(name, sources, ..., dsos=[DSO(...)])¶

Wrapper around setuptools.Extension which accepts a list of DSO dependencies.

Parameters:: dsos – A list of DSO s which this Extension will be linked against.

setuptools_dso.Extension is a wrapper around setuptools.Extension which adds the dsos=[...] keyword argument. This allows a C extension module to be linked against a DSO by name. The named DSO may be built by the same setup.py, or may already be present in $PYTHONPATH.

from setuptools_dso import DSO, Extension, setup

ext = Extension('dsodemo.ext.dtest', ['src/extension.cpp'],
    dsos=['dsodemo.lib.demo'],
)

setup(
    ...
    ext_modules = [ext],
    zip_safe = False, # setuptools_dso is not compatible with eggs!
)

Cython¶

setuptools_dso.cythonize(orig, **kws)¶: Wrapper around Cython.Build.cythonize() to correct handling of DSO s and Extension s using them.

Version 1.3 added a setuptools_dso.cythonize() wrapper to correctly handle Extension(dso=...).

Runtime¶

Beginning with setuptools-dso 2.0 a file *_dsoinfo.py will be generated alongside each DSO. eg. dso "mypkg.lib.thelib" will create mypkg/lib/thelib_dsoinfo.py. If mypkg.lib is a valid python packages (contains __init__.py) then setuptools_dso.runtime.import_dsoinfo() may be used to retrieve build time information about the DSO including platform specific filename (eg. thelib.dll vs. libthelib.so).

Beginning with 2.0 the necessary additions to $PATH or calls to os.add_dll_directory() can be made via dylink_prepare_dso().

setuptools_dso.dylink_prepare_dso(dso, package=None)¶

Take steps necessary to allow the named DSO to be loaded implicitly.

eg. On Windows, call os.add_dll_directory() as neeeded.

Parameters:

dso (str) – DSO name string (eg. ‘my.pkg.libs.adso’).
package (str) – Package name to resolve relative imports. cf. importlib.import_module

Returns:

Info module for the named DSO

Prior to 2.0, or if the generated module is not used, some additional runtime preparation is needed in order to find the "dsodemo.lib.demo" library when the dsodemo.ext.dtest Extension is imported on Windows. This could be placed in eg. example/src/dsodemo/__init__.py to ensure it always runs before the extension library is loaded.

# with setuptools_dso >= 2.0a1
#from setuptools_dso import dylink_prepare_dso
#dylink_prepare_dso('..lib.demo')

# or as previously:

import sys, os

def fixpath():
    # If this file is
    #   .../ext/__init__.py
    # DSOs are under
    #   .../lib/
    libdir = os.path.join(os.path.dirname(os.path.dirname(__file__)))

    if hasattr(os, 'add_dll_directory'): # py >= 3.8
        os.add_dll_directory(libdir)
    else:
        path = os.environ.get('PATH', '').split(os.pathsep)
        path.append(libdir)
        os.environ['PATH'] = os.pathsep.join(path)

if sys.platform == "win32":
    fixpath()

Use with ctypes¶

setuptools_dso.find_dso(dso, package=None, so=True)¶

Lookup DSO file name. eg. for use with ctypes

Parameters:

dso (str) – DSO name string (eg. ‘my.pkg.libs.adso’).
package (str) – Package name to resolve relative imports. cf. importlib.import_module
so (bool) – When True (default) return SO qualified name. eg. “libblah.so.0” vs. “libblah.so”. No effect on Windows.

Returns:

Absolute path string of DSO file.

eg.

fname = setuptools_dso.find_dso('my.pkg.libs.adso')
lib = ctypes.CDLL(fname, ctypes.RTLD_GLOBAL)

Info¶

setuptools_dso.runtime.import_dsoinfo(dso, package=None)¶

Import and return “info” module for the named DSO.

Parameters:

dso (str) – DSO name string (eg. ‘my.pkg.libs.adso’).
package (str) – Package name to resolve relative imports. cf. importlib.import_module

Returns:

Info module

For example, on a ELF target the “info” module for a DSO “mypkg.lib.thelib” would contain the attributes:

.dsoname eg. “mypkg.lib.thelib”
.libname eg. “thelib.so”
.soname eg. “thelib.so.0”
.filename eg. “/full/path/to/thelib.so”
.sofilename eg. “/full/path/to/thelib.so.0”

Working with packages using setuptools_dso¶

NUM_JOBS¶

Applying to your package¶

Building a DSO¶

Building an Extension¶

Cython¶

Runtime¶

Use with ctypes¶

Info¶

setuptools_dso

Navigation

Related Topics