What Comes to Mind

Building vcpkg dependencies with project toolchain

Steve Downey — Sat, 27 Dec 2025 17:03:15 GMT

Making sure vcpkg delivers packages built with your toolchain is not hard, but much of the advice on the internet is flat wrong. You need to specify your toolchain both in your project and in the the vcpkg triplet. There's an airgap between your project and the dependency in vcpkg install. The CMake settings can't just flow through.

1. Toolchain is more than just the compiler

Inside a link context everything needs to use the same definitions for everything or your program breaks. If you are lucky it will fail to link. If you are unlucky it will link and explode at runtime. Or you might have the worst kind of undefined behavior–working the way you expect it to. For now.

The One Definition Rule tries to say that all of the definitions in a program must mean the same thing. Using different flags for different translation units is a simple way of giving different meanings to definitions. This can range from preprocessor defines in the flags, to conditional compilation based on the compiler, or standard version, to subtle changes in meaning based on the standard version or other flags, such as the literal encoding to render characters and string constants into.

The toolchain should encompass all of the ABI affecting flags, and most flags affect ABI, or at least codegen, in some way. Differing codegen, while not as dangerous as an ABI break, can still be an unpleasant surprise.

2. Using a toolchain with `vcpkg` in-project

There are several techniques for using vcpkg for dependency management and specifying a toolchain to use for compilation of the artifacts in a project. For an open source project, the problem is generally how to do so without locking everyone into vcpkg. The current theory for CMake is to provide dependency provider hooks via CMAKE_PROJECT_TOP_LEVEL_INCLUDES, which are a list of files cmake will process before anything else, and in particular before the project statement. The Beman Project has a primitive dependency provider, use-fetch-content.cmake, that converts find_package into FetchContent and downloads and builds from Git URLs. This is an alternative to vendoring dependencies in-tree directly using git submodules or git subtrees.

As an aside, git submodules are like vice-grips–the wrong tool for every job.

For vcpkg, you can integrate using the same mechanism, setting CMAKE_PROJECT_TOP_LEVEL_INCLUDES to $(VCPKG_ROOT)/scripts/buildsystems/vcpkg.cmake. This is entirely in place of setting the CMAKE_TOOLCHAIN_FILE to vcpkg.cmake, including vcpkg.cmake in a local toolchain file, or using VCPKG_CHAINLOAD_TOOLCHAIN_FILE to instruct vcpkg.cmake to load a toolchain after processing the triple file.

The initial configure will look something like:

cmake -DCMAKE_PROJECT_TOP_LEVEL_INCLUDES=$(VCPKG_ROOT)/scripts/buildsystems/vcpkg.cmake \
      -DCMAKE_TOOLCHAIN_FILE=./etc/gcc-16-toolchain.cmake \
      -B .build -S .

With the gcc-16 toolchain file looking something like:

include_guard(GLOBAL)

set(CMAKE_C_COMPILER gcc-16)
set(CMAKE_CXX_COMPILER g++-16)
set(GCOV_EXECUTABLE "gcov-16" CACHE STRING "GCOV executable" FORCE)

set(CMAKE_CXX_STANDARD 26)

set(CMAKE_CXX_FLAGS
    "-Wall -Wextra -std=gnu++26 -Wno-maybe-uninitialized"
    CACHE STRING
    "CXX_FLAGS"
    FORCE)

set(CMAKE_CXX_FLAGS_DEBUG
    "-O0 -fno-inline -g3"
    CACHE STRING
    "C++ DEBUG Flags"
    FORCE
)
set(CMAKE_CXX_FLAGS_RELEASE
    "-Ofast -g0 -DNDEBUG"
    CACHE STRING
    "C++ Release Flags"
    FORCE
)
set(CMAKE_CXX_FLAGS_RELWITHDEBINFO
    "-O3 -g -DNDEBUG"
    CACHE STRING
    "C++ RelWithDebInfo Flags"
    FORCE
)
set(CMAKE_CXX_FLAGS_TSAN
    "-O3 -g -fsanitize=thread"
    CACHE STRING
    "C++ TSAN Flags"
    FORCE
)
set(CMAKE_CXX_FLAGS_ASAN
    "-O3 -g -fsanitize=address,undefined,leak"
    CACHE STRING
    "C++ ASAN Flags"
    FORCE
)

set(CMAKE_CXX_FLAGS_GCOV
    "-O0 -fno-default-inline -fno-inline -g --coverage -fprofile-abs-path"
    CACHE STRING
    "C++ GCOV Flags"
    FORCE
)

set(CMAKE_LINKER_FLAGS_GCOV "--coverage" CACHE STRING "Linker GCOV Flags" FORCE)

get_filename_component(RPATH "~/.local/lib64" ABSOLUTE)

set(CMAKE_EXE_LINKER_FLAGS
    "-Wl,-rpath,${RPATH}"
    CACHE STRING
    "CMAKE_EXE_LINKER_FLAGS"
    FORCE
)

This provides CMake build types DEBUG, RELEASE, RELWITHDEBINFO, TSAN, ASAN, and GCOV, overriding the built-in default flags. It exports the coverage processor to use, picked up by my project, and adds rpath to the linked executable to where I have the gcc-16 libstdc++.so.6 installed. GCC recently bumped the GLIBCXX version number to 35, so it is no longer compatible with the system libstdc++.so.6.

Note that it doesn't have any vcpkg variables or includes.

Building with the cmake command line flags above will end up calling vcpkg for any packages mentioned in the vcpkg.json file and providing them to be found by find_package calls. It's not quite how a cmake dependency provider works, but functions very much the same. However, vcpkg will build dependencies the way the system triple defines things, which will be with the system compiler and default flags, and may be entirely incompatible with your project.

3. Building Dependencies with your Toolchain.

The mechanism vcpkg provides for injecting a toolchain into a dependency is a custom triple that provides a VCPKG_CHAINLOAD_TOOLCHAIN_FILE. Because of the intermediate invocation of vcpkg, setting VCPKG_CHAINLOAD_TOOLCHAIN_FILE in your project does not affect how vcpkg builds or provides anything.

We can, however, smuggle the variable we need through the environment.

The custom triple I'm using, x64-linux-custom.cmake, looks like:

set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE static)

set(VCPKG_CMAKE_SYSTEM_NAME Linux)

set(VCPKG_CHAINLOAD_TOOLCHAIN_FILE "$ENV{PROJECT_VCPKG_TOOLCHAIN}")

And I make sure to export PROJECT_VCPKG_TOOLCHAIN into the environment as part of my build. I use a Makefile to drive my project workflow, so it's straightforward to set and export. Other workflow mechanisms are an exercise for the reader.

The cmake invocation picks up flags and environment:

export PROJECT_VCPKG_TOOLCHAIN=$(realpath ./etc/gcc-16-toolchain.cmake)
cmake -DCMAKE_TOOLCHAIN_FILE=$(PROJECT_VCPKG_TOOLCHAIN) \
      -DVCPKG_OVERLAY_TRIPLETS=$(realpath ./cmake) \
      -DVCPKG_TARGET_TRIPLET=x64-linux-custom \
      -B .build -S .

With this, vcpkg will use the toolchain I specify to build dependencies. It also treats the VCPKG_CHAINLOAD_TOOLCHAIN_FILE as ABI significant for its build caching. This means that if you include other files into your toolchain, the contents of those files do not count. A small issue if you are using layered toolchain files, sharing some flags between them such as for gcc-15 vs gcc-16 in e.g. a gcc-flags.cmake file. But I also know switching between many compilers is a very minority use case.

4. Someone is Wrong on the Internet

This took far longer to work out than it should because there is much advice that is flat out wrong.

In particular from AI generated summaries and articles.

Setting VCPKG_CHAINLOAD_TOOLCHAIN_FILE in your project or on the cmake command line cannot affect how vcpkg builds and retrieves dependencies.

You must set a custom triple file that sets VCPKG_CHAINLOAD_TOOLCHAIN_FILE in order to use that CMake toolchain with vcpkg.

Substitution is Sometimes a Failure

Steve Downey — Fri, 28 Nov 2025 17:06:58 GMT

Why are optional::transform and optional::and_then not constrained by invocable?

1. Optional "Monadic" Interface

Monadic operations for std::optional

transform is the c++ spelling for map or fmap
and_then is monadic bind for optional
or_else is dual to and_then

or_else also has:

Constraints: F models invocable and T models {move,copy}_constructible.

transform and and_then do not.

They don't work if you don't give them invocables, and rely on invoke_result_t to compute the result. So why not constrain them?

2. SFINAE is shallow

The problem is if another template has to be instantiated in order to evaluate the template of interest, and that template has an error, you get an error, not a substitution failure. And, it turns out, lambdas are a common source of the problem and a common case in real use.

Consider the code:

void f(int&);

auto l = [](auto& y) {
    f(y);
    return 42;
};

The two important parts are the auto& parameter and the implicit deduced return type.

We can rewrite it, to make things possibly more obvious:

struct Func {
    template <typename T>
    auto operator()(T& t) {
        f(t);
        return 42;
    }
};

and to slightly spoil things, the problem code is effectively:

static_assert(std::invocable<Func, int const&>);

which produces:

<source>: In instantiation of 'auto Func::operator()(T&) [with T = const int]':
type_traits:2565:26:   required by substitution of 'template<class _Fn, class ... _Args> static std::__result_of_success<decltype (declval<_Fn>()((declval<_Args>)()...)), std::__invoke_other> std::__result_of_other_impl::_S_test(int) [with _Fn = Func; _Args = {const int&}]'
type_traits:2576:55:   required from 'struct std::__result_of_impl<false, false, Func, const int&>'
type_traits:3038:12:   recursively required by substitution of 'template<class _Result, class _Ret> struct std::__is_invocable_impl<_Result, _Ret, true, std::__void_t<typename _CTp::type> > [with _Result = std::__invoke_result<Func, const int&>; _Ret = void]'
type_traits:3038:12:   required from 'struct std::is_invocable<Func, const int&>'
type_traits:3286:71:   required from 'constexpr const bool std::is_invocable_v<Func, const int&>'
concepts:336:25:   required from here
<source>:12:10: error: binding reference of type 'int&' to 'const int' discards qualifiers
   12 |         f(t);
      |         ~^~~
<source>:2:12: note:   initializing argument 1 of 'void f(int&)'
    2 |     void f(int&);
      |            ^~~~
Compiler returned: 1

Compiler Explorer

as the compiler is unhappy about trying to call the function f with a const int&.

If the lambda or Func is changed to have a non-deduced return type, the instantiation errors from the check to invocable go away, although you still get an error calling either with a const int.

So why do we run into this with transform if we were to constrain it with invocable ?

The compiler needs to figure out the overload set in order to resolve which one to use from the set. There are four of them two for the l- and r- value category and two for the const overloads.

template<class F> constexpr auto transform(F&& f) &;
template<class F> constexpr auto transform(F&& f) const &;
template<class F> constexpr auto transform(F&& f) &&;
template<class F> constexpr auto transform(F&& f) const &&;

with differing computations of the resulting optional being returned.

using U = invoke_result_t<F, decltype(std::move(*val))>;
//or
using U = remove_cv_t<invoke_result_t<F, decltype(*val)>>;

So, in order to work out what the templated transform's signature really is, it has to compute what the invocable returns, and since the invocable has deduced return type, it needs to instantiate it, and instantiating with const int causes an error.

This is unfortunate.

If we constrain transform we get the same errors as above. See here, with just enough of an optional to compile.

3. Constraints, what are they good for

Not absolutely nothing.

Constraints in the library:

Constraints: the conditions for the function's participation in overload resolution ([over.match]).

[Note 1: Failure to meet such a condition results in the function's silent non-viability. — end note]

[Example 1: An implementation can express such a condition via a constraint-expression ([temp.constr.decl]). — end example]

[structure.specifications] 3.1

Constraints are for making an overload not exist if the constraint isn't met. It's not a way of signaling an error. Those are Mandates:

Mandates: the conditions that, if not met, render the program ill-formed.

[Example 2: An implementation can express such a condition via the constant-expression in a static_assert-declaration ([dcl.pre]). If the diagnostic is to be emitted only after the function has been selected by overload resolution, an implementation can express such a condition via a constraint-expression ([temp.constr.decl]) and also define the function as deleted. — end example]

[structure.specifications] 3.2

Asking to run and_then on a non-invocable probably ought not to say there is no such function, but instead tell you it can't be invoked. I'm now not convinced that or_else should be constrained this way. It's not significantly better for o.or_else(5) to fail to resolve, mentioning invocable, than produce an error that invoke_result_t doesn't work, or that f can't be invoked. The kind of error is a minor detail.

Constraints that let you control the choice of alternatives are wonderful, and requires clauses are normal programmer accessible, unlike SFINAE, or even enable_if. But without an overload set to constrain, there possibly should not be a constraint.

4. Can we do better?

There are some notes in P0798 that suggest that Deducing This might help, P0847.

The idea would to be to NOT have all the value category overloads that need to be checked, but to just have a single one that deduces what this is and provide it as a template parameter for further use. The contained parameter could be forwarded using forward_like<Self>.

P0847 has discussion about how deducing this might be applied to optional. There's also discussion of deducing this and the SFINAE-unfriendly auto at C++23’s Deducing this: what it is, why it is, how to use it.

With the tools we have today, it looks possible, but still slightly messy. I managed to get my implementation of optional to compile and pass its own tests with.

template <class F, class Self>
    requires(
        std::invocable<F,
                       decltype(std::forward_like<Self>(std::declval<T>()))>)
constexpr auto transform(this Self&& self, F&& f)
    -> optional<std::invoke_result_t<
        F,
        decltype(std::forward_like<Self>(std::declval<T>()))>> {
    using U = std::invoke_result_t<F,
                                   decltype(std::forward_like<Self>(
                                       std::declval<T>()))>;
    static_assert(!std::is_array_v<U>);
    static_assert(!std::is_same_v<U, in_place_t>);
    static_assert(!std::is_same_v<U, nullopt_t>);
    static_assert(std::is_object_v<U> || std::is_reference_v<U>);
    if (self.has_value()) {
        return optional<U>{detail::from_function,
                           std::forward<F>(f),
                           std::forward_like<Self>(self.value_)};
    }
    return optional<U>;
}

If there were a std::forward_like_t, it might be possible to reduce some of the noise in computing the value category used for the T. I also have not thought extensively about if the requires clause is truly needed in light of the invoke_result_ that can now be used in the trailing return type.

Nikola Blog Infrastructure

Steve Downey — Mon, 23 Dec 2024 18:15:27 GMT

I've migrated from WordPress to a static blog generator–Nikola. This is how it works.

1. Nikola

Nikola is a static site generator written in python. It's mostly boring, which is a very good thing for software meant for use to be. Its home is https://getnikola.org and the source is on github at https://github.com/getnikola/nikola. It's many years old at this point, but not dead.

The interesting part for me is that someone had already written a plugin that compiled org mode files, my preferred markup language, to the html that Nikola wants to publish. It wasn't quite to my tastes, but the changes were all on the org export side in elisp, and the python invocation of emacs to do the translation didn't need particular changes. That update is at https://github.com/steve-downey/wctm/tree/src/plugins/orgmode.

I've also set up staging at a github pages repo rendered at https://steve-downey.github.io/ since it's a static site. The real version is at https://sdowney.org.

Conversion was straightforward using the tools that Nikola provided and following the directions for importing a WordPress xml backup.

2. Orgmode plugin

Pulling a plugin into my working site was easy

nikola plugin -i orgmode

copied the code from https://github.com/getnikola/plugins/tree/master/v8/orgmode and I just had to update the main configuration file, conf.py, to map org files to the orgmode compiler.

COMPILERS = {
    "rest": ['.txt', '.rst'],
    "markdown": ['.md', '.mdown', '.markdown'],
    "html": ['.html', '.htm'],
    "orgmode": ['.org'],
}

I then updated the source code for the elisp html exporter using code from https://github.com/steve-downey/wg21org/blob/main/emacs.d/init.el which installs org from upstream rather than using the built-in orgmode and configures my preferences. I may end up using more of ox-wg21html.el as a framework for the exported HTML, only updating to use Bootstrap for css classes instead of my own or the org builtin.

3. LaTeX support via KaTeX

I've configure Nikola to support LaTeX via KaTeX rather than MathJax. It is supposed to be a bit faster. This means inline math like $x^2$ or $1 < 2$ or with breaks like: \[ \int_0^\infty e^{-x^2} dx = {{\sqrt{\pi}} \over {2}} \] Or in running text where I want to mention \[ \frac{-b \pm \sqrt{b^2 - 4 a c}}{2a} \] as the quadratic equation. It does mean I don't have access to latex blocks like

\begin{equation}
  n_{i+1} = \frac{n_{i} (d-i) (e-1)}{(i+1)}
\end{equation}

Maybe I won't miss it.

Concept Maps using C++23 Library Tech

Steve Downey — Sun, 19 May 2024 15:51:00 GMT

Abstract

C++0x Concepts had a feature Concept Maps that allowed a set of functions, types, and template definitions to be associated with a concept and the map to be specialized for types that meet the concept.

This allowed open extension of a concept.

A definition could be provided that allows an algorithm to operate in terms of the API a concept presents and the map would define how those operations are implemented for a particular type.

This is similar to how Haskell's typeclass works.

Lost with `Concepts-Lite`

The feature was very general, and lost as part of the Concepts-Lite proposal that was eventually adopted.

This loss of a level of indirection means that the APIs for a concept must be implemented by those names for a type, even when those names are not particularly good choices in the natural domain of a type rather than in the domain as a concept.

The proliferation of transform functions for functorial map is such a problem.

It is also a problem when adapting types that are closed for extension or do not permit member functions.

Why?

Don't know if you should
Need to know if you could first

Alternatives

Virtual Interface
Adapters
Collection of CPOs

Hard to Support

Example from C++0x Concepts

Student Record

class student record {
public:
  string id;
  string name;
  string address;
  bool   id_equal(const student record&);
  bool   name_equal(const student record&);
  bool   address_equal(const student record&);
};

Equality Comparable

concept_map EqualityComparable<student record>{
    bool operator==(const student record& a,
                    const student record& b){
        return a.id_equal(b);
}
};

Allow associated types

Very useful for pointers

concept_map BinaryFunction<int (*)(int, int), int, int>
{
    typedef int result_type;
};

Why Didn't We Get Them?

Let's not go there right now.

State of the Art

Rust Traits

trait PartialEq {
    fn eq(&self, rhs: &Self) -> bool;

    fn ne(&self, rhs: &Self) -> bool {
        !self.eq(rhs)
    }
}

C++ CPOs

Some Concepts and Types

namespace N::hidden {
template <typename T>
concept has_eq = requires(T const& v) {
  { eq(v, v) } -> std::same_as<bool>;
};

struct eq_fn {
  template <has_eq T>
  constexpr bool operator()(T const& x,
                            T const& y) const {
    return eq(x, y);
  }
};

template <has_eq T>
constexpr bool ne(T const& x, T const& y) {
  return not eq(x, y);
}

template <typename T>
concept has_ne = requires(T const& v) {
  { ne(v, v) } -> std::same_as<bool>;
};

struct ne_fn {
  template <has_ne T>
  constexpr bool operator()(T const& x,
                            T const& y) const {
    return ne(x, y);
  }
};
} // namespace N::hidden

See Why tag_invoke is not the solution I want by Barry Revzin https://brevzin.github.io/c++/2020/12/01/tag-invoke/

C++ partial_equality

namespace N {
inline namespace function_objects {
inline constexpr hidden::eq_fn eq{};
inline constexpr hidden::ne_fn ne{};
} // namespace function_objects

template <typename T>
concept partial_equality
  requires(std::remove_reference_t<T> const& t)
{
  eq(t, t);
  ne(t, t);
};
} // namespace N

See Why tag_invoke is not the solution I want by Barry Revzin https://brevzin.github.io/c++/2020/12/01/tag-invoke/

Requirements for Solution

Tied to the type system
Automatable
"zero" overhead
- no virtual calls
- no type erasure

What does typeclass do?

Adds a record to the function that defines the operations for the type.

Can we do that?

Type-based lookup

Templates!

Additional Requirements

Avoid ADL

Object Lookup rather than Overload Lookup

Variable templates

Variable templates have become more powerful

We can have entirely distinct specializations

A Step Towards Implementation

template <class T>
concept partial_equality = requires(
    std::remove_reference_t<T> const& t) {
  {
    partial_eq<T>.eq(t, t)
  } -> std::same_as<bool>;
  {
    partial_eq<T>.ne(t, t)
  } -> std::same_as<bool>;
};

`partial_eq<T>`

An inline variable object

template<class T>
constexpr inline auto partial_eq = hidden::partial_eq_default;

A default implementation

constexpr inline struct partial_eq_default_t {
  constexpr bool
  eq(has_eq auto const& rhs,
     has_eq auto const& lhs) const {
    return (rhs == lhs);
  }
  constexpr bool
  ne(has_eq auto const& rhs,
     has_eq auto const& lhs) const {
    return (lhs != rhs);
  }
} partial_eq_default;

New `has_eq`

template <typename T>
concept has_eq = requires(T const& v) {
  { operator==(v, v) } -> std::same_as<bool>;
};

Will do better

In a bit

Monoid

A little more than you think.

A type
With an associative binary operation
Which is closed
And has an identity element

Maybe not a lot more

Math

$\oplus: M \times M \rightarrow M$
$x \oplus (y \oplus z) = (x \oplus y) \oplus z$
$1_M \in M$ such that $\forall m \in M : (1_M \oplus m) = m = (m \oplus 1_M)$

Function form

$f : M \times M \rightarrow M$
$f(x, f(y, z)) = f(f(x, y), z)$
$1_M \in M$ such that $\forall m \in M : f(1_M, m) = m = f(m, 1_M)$

The similarity to left and right fold is NOT an accident

Core Functions

$empty : m$: $empty = concat \, []$
$concat : [m] \rightarrow m$: $fold \, append \, empty$
$append : m \rightarrow m \rightarrow m$: $op$

Note that it's self-referential

This is common

From Haskell Prelude

class Semigroup a => Monoid a where
  mempty :: a
  mempty = mconcat []

  mappend :: a -> a -> a
  mappend = (<>)

  mconcat :: [a] -> a
  mconcat = foldr mappend mempty

Minimum Set

$empty \, | \, concat$

In C++

template <typename T, typename M>
concept MonoidRequirements =
    requires(T i) {
      { i.identity() } -> std::same_as<M>;
    }
    ||
    requires(T i, std::ranges::empty_view<M> r1) {
      { i.concat(r1) } -> std::same_as<M>;
    };

I am ignoring all sorts of const volatile reference issues here.

Implementing the other side

The Map for a Monoid

template <class Impl>
  requires MonoidRequirements<
      Impl,
      typename Impl::value_type>
struct Monoid : protected Impl {
  auto identity(this auto&& self);

  template <typename Range>
  auto concat(this auto&& self, Range r);

  auto op(this auto&& self, auto a1, auto a2);
};

empty is a terrible name, concat only a little better. empty becomes identity

`identity`

    auto identity(this auto && self) {
        std::puts("Monoid::identity()");
        return self.concat(std::ranges::empty_view<typename Impl::value_type>{});
    }

`concat`

   template<typename Range>
   auto concat(this auto&& self, Range r) {
        std::puts("Monoid::concat()");
        return std::ranges::fold_right(r,
                    self.identity(),
                    [&](auto m1, auto m2){return self.op(m1, m2);});
    }

`op`

   auto op(this auto&& self, auto a1, auto a2) {
        std::puts("Monoid::op");
        return self.op(a1, a2);
    }

Deducing `this` and CRTP

We'll see in a moment, but it's because we want to constraint the required implementation.

We want to use the derived version which has all of the operations.

`Plus`

template <typename M>
class Plus {
public:
  using value_type = M;
  auto identity(this auto&& self) -> M {
    std::puts("Plus::identity()");
    return M{0};
  }

  auto op(this auto&& self, auto s1, auto s2) -> M {
    std::puts("Plus::op()");
    return s1 + s2;
  }
};

`PlusMonoidMap`

template<typename M>
struct PlusMonoidMap : public Monoid<Plus<M>> {
    using Plus<M>::identity;
    using Plus<M>::op;
};

Need to pull the operations from the Monoid instance into the Map, so we get the right ones being used by concat.

This might be simpler if we didn't allow choice of the basis operations, but that's also overly restrictive.

The map instances

template<class T> auto monoid_concept_map = std::false_type{};

template<>
constexpr inline auto monoid_concept_map<int> = PlusMonoidMap<int>{};

template<>
constexpr inline auto monoid_concept_map<long> = PlusMonoidMap<long>{};

template<>
constexpr inline auto monoid_concept_map<char> = PlusMonoidMap<char>{};

Can we `concat` instead?

class StringMonoid {
public:
  using value_type = std::string;

  auto op(this auto&&, auto s1, auto s2) {
    std::puts("StringMonoid::op()");
    return s1 + s2;
  }

  template <typename Range>
  auto concat(this auto&& self, Range r) {
    std::puts("StringMonoid::concat()");
    return std::ranges::fold_right(
        r, std::string{}, [&](auto m1, auto m2) {
          return self.op(m1, m2);
        });
  }
};

No, I'm not properly constraining Range here. No, I'm not actually recommending this as an implementation.

The Map and instance

struct StringMonoidMap : public Monoid<StringMonoid> {
    using StringMonoid::op;
    using StringMonoid::concat;
};

template<>
constexpr inline auto monoid_concept_map<std::string> = StringMonoidMap{};

Some simple use

Exercise the functions

template<typename P>
void testP()
{
    auto d1 = monoid_concept_map<P>;

    auto x = d1.identity();
    assert(P{} == x);

    auto sum = d1.op(x, P{1});
    assert(P{1} == sum);

    std::vector<P> v = {1,2,3,4};
    auto k = d1.concat(v);
    assert(k == 10);
}

Some simple cases

    std::cout << "\ntest int\n";
    testP<int>();

    std::cout << "\ntest long\n";
    testP<long>();

   std::cout << "\ntest char\n";
    testP<char>();

On `std::string`

This will use the StringMonoid we defined a few moments ago.

    auto d2 = monoid_concept_map<std::string>;

    std::cout << "\ntest string\n";
    auto x2 = d2.identity();
    assert(std::string{} == x2);

    auto sum2 = d2.op(x2, "1");
    assert(std::string{"1"} == sum2);

    std::vector<std::string> vs = {"1","2","3","4"};
    auto k2 = d2.concat(vs);
    assert(k2 == std::string{"1234"});

Note that the map type is mostly invisible.

Results

test int

Plus::identity()
Plus::op()
Monoid::concat()
Plus::identity()
Plus::op()
Plus::op()
Plus::op()
Plus::op()

test long

Plus::identity()
Plus::op()
Monoid::concat()
Plus::identity()
Plus::op()
Plus::op()
Plus::op()
Plus::op()

test char

Plus::identity()
Plus::op()
Monoid::concat()
Plus::identity()
Plus::op()
Plus::op()
Plus::op()
Plus::op()

test string

Monoid::identity()
StringMonoid::concat()
StringMonoid::op()
StringMonoid::concat()
StringMonoid::op()
StringMonoid::op()
StringMonoid::op()
StringMonoid::op()

Monoid in Trees

Foldable generalizes

Folding is very much tied to Range like things.

It can, and has, been generalized to things that can be traversed.

monoids are still critical for Traversables.

Summarizing Data in a tree

If the summary type is monoidal, nodes can hold summaries of all the data below them.

`fingertrees`

Much of the flexibility of fingertrees comes from the monoidal tags.

They are also fairly complicated.

Technique can be applied to other, simpler trees.

P3200 (eventually) ((C++29))

fringe-tree

Simplified tree with data at the edges

Code

Show the monoid-map branch of

steve-downey/fringetree.git

Summary for Concept Maps

Tell you what I told you

Variable templates for map lookup
Named operations on the map object
Open for extension
Concept checkable implementations
Decoupled map use and implementation

Questions?

Or comments

Thank You

Slides from C++Now 2023 Async Control Flow

Steve Downey — Sat, 18 May 2024 22:27:00 GMT

Using Sender/Receiver for Async Control Flow

Steve Downey

These are the slides, slightly rerendered, from my presentation at C++Now 2023.

Abstract

How can P2300 Senders be composed using sender adapters and sender factories to provide arbitrary program control flow?

How do I use these things?

Where can I steal from?

`std::execution`

P2300

Recent version at https://isocpp.org/files/papers/P2300R7.html

A self-contained design for a Standard C++ framework for managing asynchronous execution on generic execution resources.

Three Key Abstractions

Schedulers
Senders
Receivers

Schedulers

Responsible for scheduling work on execution resources.

Execution resources are things like threads, GPUs, and so on.

Sends work to be done in a place.

Senders

Senders describe work.

Receivers

Receivers are where work terminates.

Value channel
Error channel
Stopped channel

Work can terminate in three different ways.

Return a value.
Throw an exception
Be canceled

It's been a few minutes. Lets see some simple code.

Hello Async World

 1: 
 2: #include <stdexec/execution.hpp>
 3: #include <exec/static_thread_pool.hpp>
 4: #include <iostream>
 5: 
 6: int main() {
 7:   exec::static_thread_pool pool(8);
 8: 
 9:   stdexec::scheduler auto sch = pool.get_scheduler();
10: 
11:   stdexec::sender auto begin = stdexec::schedule(sch);
12:   stdexec::sender auto hi    = stdexec::then(begin, [] {
13:     std::cout << "Hello world! Have an int.\n";
14:     return 13;
15:   });
16: 
17:   auto add_42 = stdexec::then(hi, [](int arg) { return arg + 42; });
18: 
19:   auto [i] = stdexec::sync_wait(add_42).value();
20: 
21:   std::cout << "The int is " << i << '\n';
22: 
23:   return 0;
24: }
25:

Compiler Explorer

Hello Async World Results

Hello world! Have an int.
The int is 55

When All - Concurent Async

 1: exec::static_thread_pool pool(3);
 2: 
 3: auto sched = pool.get_scheduler();
 4: 
 5: auto fun = [](int i) { return i * i; };
 6: 
 7: auto work = stdexec::when_all(
 8:     stdexec::on(sched, stdexec::just(0) | stdexec::then(fun)),
 9:     stdexec::on(sched, stdexec::just(1) | stdexec::then(fun)),
10:     stdexec::on(sched, stdexec::just(2) | stdexec::then(fun)));
11: 
12: auto [i, j, k] = stdexec::sync_wait(std::move(work)).value();
13: 
14: std::printf("%d %d %d\n", i, j, k);

Describe some work:

Creates 3 sender pipelines that are executed concurrently by passing to `when_all`

Each sender is scheduled on `sched` using `on` and starts with `just(n)` that creates a Sender that just forwards `n` to the next sender.

After `just(n)`, we chain `then(fun)` which invokes `fun` using the value provided from `just()`

Note: No work actually happens here. Everything is lazy and `work` is just an object that statically represents the work to later be executed

When All - Concurent Async - Results

0 1 4

Order of execution is by chance, order of results is determined.

Dynamic Choice of Sender

 1: exec::static_thread_pool pool(3);
 2: 
 3: auto sched = pool.get_scheduler();
 4: 
 5: auto fun = [](int i) -> stdexec::sender auto {
 6:   using namespace std::string_literals;
 7:   if ((i % 2) == 0) {
 8:     return stdexec::just("even"s);
 9:   } else {
10:     return stdexec::just("odd"s);
11:   }
12: };
13: 
14: auto work = stdexec::when_all(
15:     stdexec::on(sched, stdexec::just(0) | stdexec::let_value(fun)),
16:     stdexec::on(sched, stdexec::just(1) | stdexec::let_value(fun)),
17:     stdexec::on(sched, stdexec::just(2) | stdexec::let_value(fun)));
18: 
19: auto [i, j, k] = stdexec::sync_wait(std::move(work)).value();
20: 
21: std::printf("%s %s %s", i.c_str(), j.c_str(), k.c_str());

Compiler Explorer

Enough API to talk about control flow

The minimal set being:

stdexec::on
stdexec::just
stdexec::then
stdexec::let_value
stdexec::sync_wait

I will mostly ignore the error and stop channels

Vigorous Handwaving

Some Theory

Continuation Passing Style

Not At All New

Sussman and Steele in 1975

AI Memo 349: "Scheme: An Interpreter for Extended Lambda Calculus"

Pass a "Continuation"

Where to go next rather than return the value.

add :: Float -> Float -> Float
add a b = a + b

add_cps :: Float -> Float -> (Float -> a) -> a
add_cps a b cont = cont (a + b)

auto add(float a, float b) -> float {
    return a + b;
}

template<typename Cont>
auto add_cps(float a, float b, Cont k) {
    return k(a+b);
}

Inherently a tail call

In continuation passing style we never return.

We send a value to the rest of the program.

Hard to express in C++.

Extra machinery necessary to do the plumbing.

Also, some risk, so we don't always do TCO.

We keep the sender "thunks" live so we don't dangle references.

Intermittently Popular as a Compiler Technique

The transformations of direct functions to CPS are mechanical.

The result is easier to optimize and mechanically reason about.

Equivalent to Single Static Assignment.

Structured Programming can be converted to CPS.

Delimted Continuations

General continuations reified as a function.

Everyone knows that when a process executes a system call like ‘read’, it gets suspended. When the disk delivers the data, the process is resumed. That suspension of a process is its continuation. It is delimited: it is not the check-point of the whole OS, it is the check-point of a process only, from the invocation of main() up to the point main() returns. Normally these suspensions are resumed only once, but can be zero times (exit) or twice (fork).

Oleg Kiselyov Fest2008-talk-notes.pdf

If this qoute reminds you of coroutines, you are paying attention.

Haskell's Cont Type

newtype Cont r a = Cont { runCont :: (a -> r) -> r }

This is roughly equivalent to the sender value channel. A Cont takes a reciever, a function that consumes the value being sent, and produces an r, the result type.

The identity function is often used.

Underlies `std::execution`

The plumbing is hidden.

Senders "send" to their continuations, delimted by the Reciever.

Another Level of Indirection

Solves all problems

Adds two more.

At least

CPS Indirects Function Return

Transform a function

$latex A \rightarrow B $

$latex A \rightarrow B \rightarrow ( B \rightarrow R ) \rightarrow R $

add :: Float -> Float -> Float
add a b = a + b

add_cps :: Float -> Float -> (Float -> A) -> A
add_cps a b cont = cont (a + b)

Sender Closes Over A

$latex B \rightarrow ( B \rightarrow R ) \rightarrow R $

The $LATEX A$ is (mostly) erased from the Sender.

Reciever Is The Transform to Result

$latex ( B \rightarrow R ) \rightarrow R $

Some Pictures

Sender

`just`

stdexec::just(0)

`then`

auto f(A a) -> B;
auto s = stdexec::just(a) | stdexec::then(f);

`let_value`

sender_of<set_value_t(B)> auto snd(A a);
auto s = stdexec::just(a) | stdexec::let_value(snd);

In which we use the M word

Sender is a Monad

(surprise)

(shock, dismay)

Function Composition is the hint

Functions are units of work.

We compose them into programs.

The question is if the rules apply.

Monadic Interface

bind or and_then: $latex M \langle a \rangle \rightarrow (a \rightarrow M \langle b \rangle ) \rightarrow M \langle b \rangle $
fish or kleisli arrow: $latex (a \rightarrow M \langle b \rangle ) \rightarrow (b \rightarrow M \langle c \rangle ) \rightarrow (a \rightarrow M \langle c \rangle ) $
join or flatten or mconcat: $latex M \langle M \langle a \rangle \rangle \rightarrow M \langle a \rangle $

Monad Interface

Applicative and Functor parts

make or pure or return: $latex a \rightarrow M \langle a \rangle $
fmap or transform: $latex (a \rightarrow b) \rightarrow M \langle a \rangle \rightarrow M \langle b \rangle $

Any one of the first three and one of the second two can define the other three

Monad Interface

Monad Laws

left identity: bind(pure(a), h) == h(a)
right identity: bind(m, pure) == m
associativity: bind(bind(m, g), h) == bind(m, bind((\x -> g(x), h))

Monad Laws

Sender is Three Monads in a Trench-coat

Stacked up.

Value
Error
Stopped

The three channels can be crossed, mixed, and remixed. Focus on the value channel for simplicity.

The Three Monadic Parts

`just`

Send a value.

pure

just lifts a value into the monad

`then`

Send a value returned from a function that takes its argument from a Sender.

fmap or transform

then is the functor fmap

`let_value`

Send what is returned by a Sender returned from a function that takes its argument from a Sender.

bind

let value is the monadic bind

Necessary and Sufficient

The monadic bind gives us the runtime choices we need.

Basis of Control

Sequence
Decision
Recursion

Sequence

 1:   stdexec::sender auto work =
 2:       stdexec::schedule(sch)
 3:       | stdexec::then([] {
 4:           std::cout << "Hello world! Have an int.";
 5:           return 13;
 6:       })
 7:       | stdexec::then([](int arg) { return arg + 42; });
 8: 
 9:   auto [i] = stdexec::sync_wait(work).value();
10:

One thing after another.

Decision

 1: exec::static_thread_pool pool(8);
 2: 
 3: stdexec::scheduler auto sch = pool.get_scheduler();
 4: 
 5: stdexec::sender auto begin  = stdexec::schedule(sch);
 6: stdexec::sender auto seven  = stdexec::just(7);
 7: stdexec::sender auto eleven = stdexec::just(11);
 8: 
 9: stdexec::sender auto branch =
10:     begin
11:     | stdexec::then([]() { return std::make_tuple(5, 4); })
12:     | stdexec::let_value(
13:         [=](auto tpl) {
14:         auto const& [i, j] = tpl;
15: 
16:         return tst((i > j),
17:                    seven | stdexec::then([&](int k) noexcept {
18:                        std::cout << "true branch " << k << '\n';
19:                    }),
20:                    eleven | stdexec::then([&](int k) noexcept {
21:                        std::cout << "false branch " << k << '\n';
22:                    }));
23:     });
24: 
25: stdexec::sync_wait(std::move(branch));

true branch 7

Control what sender is sent at rentime depending on the state of the program when the work is executing rather than in the structure of the senders.

`tst` function

 1: inline auto tst = [](bool                 cond,
 2:                      stdexec::sender auto left,
 3:                      stdexec::sender auto right)
 4:     -> exec::variant_sender<decltype(left),
 5:                             decltype(right)> {
 6:   if (cond)
 7:     return left;
 8:   else
 9:     return right;
10: };
11:

Recursion

Simple Recursion

 1: 
 2: using any_int_sender =
 3:     any_sender_of<stdexec::set_value_t(int),
 4:                   stdexec::set_stopped_t(),
 5:                   stdexec::set_error_t(std::exception_ptr)>;
 6: 
 7: auto fac(int n) -> any_int_sender {
 8:     std::cout << "factorial of " << n << "\n";
 9:     if (n == 0)
10:         return stdexec::just(1);
11: 
12:     return stdexec::just(n - 1)
13:         | stdexec::let_value([](int k) { return fac(k); })
14:         | stdexec::then([n](int k) { return k * n; });
15: }
16:

 1: 
 2:     int                  k = 10;
 3:     stdexec::sender auto factorial =
 4:         begin
 5:         | stdexec::then([=]() { return k; })
 6:         | stdexec::let_value([](int k) { return fac(k); });
 7: 
 8:     std::cout << "factorial built\n\n";
 9: 
10:     auto [i] = stdexec::sync_wait(std::move(factorial)).value();
11:     std::cout << "factorial " << k << " = " << i << '\n';
12:

factorial built

factorial of 10
factorial of 9
factorial of 8
factorial of 7
factorial of 6
factorial of 5
factorial of 4
factorial of 3
factorial of 2
factorial of 1
factorial of 0
factorial 10 = 3628800

General Recursion

 1: auto fib(int n) -> any_int_sender {
 2:     if (n == 0)
 3:         return stdexec::on(getDefaultScheduler(),  stdexec::just(0));
 4: 
 5:     if (n == 1)
 6:         return stdexec::on(getDefaultScheduler(), stdexec::just(1));
 7: 
 8:     auto work = stdexec::when_all(
 9:                     stdexec::on(getDefaultScheduler(), stdexec::just(n - 1)) |
10:                         stdexec::let_value([](int k) { return fib(k); }),
11:                     stdexec::on(getDefaultScheduler(), stdexec::just(n - 2)) |
12:                         stdexec::let_value([](int k) { return fib(k); })) |
13:                 stdexec::then([](auto i, auto j) { return i + j; });
14: 
15:     return work;
16: }
17:

 1: 
 2: int                  k = 30;
 3:     stdexec::sender auto fibonacci =
 4:         begin | stdexec::then([=]() { return k; }) |
 5:         stdexec::let_value([](int k) { return fib(k); });
 6: 
 7:     std::cout << "fibonacci built\n";
 8: 
 9:     auto [i] = stdexec::sync_wait(std::move(fibonacci)).value();
10:     std::cout << "fibonacci " << k << " = " << i << '\n';

fibonacci built
fibonacci 30 = 832040
fibonacci 30 = 832040

Fold

 1: 
 2:         if (first == last) {
 3:             return stdexec::just(U{std::move(init)});
 4:         }
 5: 
 6:         auto nxt =
 7:             stdexec::just(std::invoke(f, std::move(init), *first)) |
 8:             stdexec::let_value([this,
 9:                                 first = first,
10:                                 last = last,
11:                                 f = f
12:                                 ](U u) {
13:                 I i = first;
14:                 return (*this)(++i, last, u, f);
15:             });
16:         return std::move(nxt);

 1: 
 2:     auto v = std::ranges::iota_view{1, 10'000};
 3: 
 4:     stdexec::sender auto work =
 5:         begin
 6:         | stdexec::let_value([i = std::ranges::begin(v),
 7:                               s = std::ranges::end(v)]() {
 8:             return fold_left(i, s, 0, [](int i, int j) { return i + j; });
 9:         });
10: 
11:     auto [i] = stdexec::sync_wait(std::move(work)).value();
12:

work  = 49995000

Backtrack

using any_node_sender =
    any_sender_of<stdexec::set_value_t(tree::NodePtr<int>),
                  stdexec::set_stopped_t(),
                  stdexec::set_error_t(std::exception_ptr)>;

auto search_tree(auto                    test,
                 tree::NodePtr<int>      tree,
                 stdexec::scheduler auto sch,
                 any_node_sender&&       fail) -> any_node_sender {
    if (tree == nullptr) {
        return std::move(fail);
    }
    if (test(tree)) {
        return stdexec::just(tree);
    }
    return stdexec::on(sch, stdexec::just()) |
           stdexec::let_value([=, fail = std::move(fail)]() mutable {
               return search_tree(
                   test,
                   tree->left(),
                   sch,
                   stdexec::on(sch, stdexec::just()) |
                       stdexec::let_value(
                           [=, fail = std::move(fail)]() mutable {
                               return search_tree(
                                   test, tree->right(), sch, std::move(fail));
                           }));
           });
    return fail;
}

    tree::NodePtr<int> t;
    for (auto i : std::ranges::views::iota(1, 10'000)) {
        tree::Tree<int>::insert(i, t);
    }

    auto test = [](tree::NodePtr<int> t) -> bool {
        return t ? t->data() == 500 : false;
    };

    auto fail = begin | stdexec::then([]() { return tree::NodePtr<int>{}; });

    stdexec::sender auto work =
        begin | stdexec::let_value([=]() {
            return search_tree(test, t, sch, std::move(fail));
        });

    auto [n] = stdexec::sync_wait(std::move(work)).value();

    std::cout << "work "
              << " = " << n->data() << '\n';

work  = 500

Don't Do That

Can is not Should

Write an Algorithm

Why You Might

Throughput
Interruptable

Thank You

Some Informal Remarks Towards a New Theory of Trait Customization

Steve Downey — Sun, 24 Dec 2023 05:00:00 GMT

A Possible Technique

constexpr bool g(int lhs, int rhs) {
    auto& op = partial_eq<int>;
    return op.ne(lhs, rhs);
}

Compiler Explorer with Supporting Code A trait is defined as a template variable that implements the required operations. Implementation of those operations is possible via a variety of techniques, but existence is concept checkable. It might prove useful to explicitly opt in to a sufficiently generic trait. The technique satisfies the openness requirement, that the trait can be created independently of the type that models the trait. There can still only be one definition, but this enables opting std:: types into new traits, for example. It also doesn't universally grab an operation name. The trait variable is namespaceable. Syntax isn't really awesome, but not utterly unworkable.

Background

Several years ago, Barry Revzin in "Why tag_invoke is not the solution I want" outlined the characteristics that a good customization interface would have. Quoting

The ability to see clearly, in code, what the interface is that can (or needs to) be customized.

The ability to provide default implementations that can be overridden, not just non-defaulted functions.

The ability to opt in explicitly to the interface.

The inability to incorrectly opt in to the interface (for instance, if the interface has a function that takes an int, you cannot opt in by accidentally taking an unsigned int).

The ability to easily invoke the customized implementation. Alternatively, the inability to accidentally invoke the base implementation.

The ability to easily verify that a type implements an interface.

I believe that with some support on the implementation side, and some concept definitions to assert correct usage, having an explicit object that implements the required traits for a concept can satisfy more of the requirements than tag_invoke or std:: customization points. The trade-off is that usage of the trait is explicit and not dependent on arguments to the trait, which means that it is more verbose and possible to get wrong in both subtle and gross ways.

`concept_map`

In the original proposal for C++ concepts, there was a facility called ~concept_map~s where

Concept maps describe how a set of template arguments satisfy the requirements stated in the body of a concept definition.

class student_record {
  public:
    string id;
    string name;
    string address;
};

concept EqualityComparable<typename T> {
    bool operator==(T, T);
}

concept_map EqualityComparable<student_record> {
    bool operator==(const student_record& a, const student_record& b) {
        return a.id == b.id;
    }
};

template<typename T> requires EqualityComparable<T>
void f(T);

f(student_record()); // okay, have concept_map EqualityComparable<student_record>

n2617 This allowed for customizing how the various requirements for a concept were implemented for a particular type. This was lost in Concepts Lite, a.k.a C++20 Concepts. Other generic type systems have implemented something like this feature, as well as definition checking. In particular, Rust Traits are an analagous feature.

Rust Traits

A trait is a collection of methods defined for an unknown type: Self. They can access other methods declared in the same trait.

An example that Revzin mentions, and that my first example alludes to is PartialEq:

pub trait PartialEq<Rhs: ?Sized = Self> {
    /// This method tests for `self` and `other` values to be equal, and is used
    /// by `==`.
    fn eq(&self, other: &Rhs) -> bool;

    /// This method tests for `!=`. The default implementation is almost always
    /// sufficient, and should not be overridden without very good reason.
    fn ne(&self, other: &Rhs) -> bool {
        !self.eq(other)
    }
}

From https://doc.rust-lang.org/src/core/cmp.rs.html#219 In Rust this is built into the language, and operators like == are automatically rewritten into eq and ne. At least that's my understanding. We're not going to get that in C++, ever. With both Rust and Concept Maps, though, we do get new named operations that can be used unqualified in generic code and the compiler will be directed to the correct implementation. Giving up on that is key to a way forward in C++.

A trait object

The technique I'm considering and describing here is modeled loosly after the implementation of Haskell typeclasses in GHC. For a particular instance of a typeclass, a record holding the operations based on the actual type in use is created and made available, and the named operations are lifted into scope and the functions in the record called when used. It is as if a virtual function table was implemented with name lookup rather than index. In C++, particularly in current post-C++20 C++, we can look up an object via a template variable. The implementations of different specializations of a template variable do not need to be connected in any way. We have to provide a definition, since to make it look like a declaration it's necessary to provide some type such as false_type. Alternatively, we could declare it as an int, but mark it as extern and not define it. I'm still researching alternatives.

template<class T> auto someTrait = std::false_type{};

template <typename T>
extern int otherTrait;

These are useful if there is no good generic definition of the trait. If there is a good generic definition of a trait, the trait variable is straightforward:

constexpr inline struct {
    constexpr auto eq(auto rhs, auto lhs) const {return rhs == lhs;}
    constexpr auto ne(auto rhs, auto lhs) const {return !eq(lhs, rhs);}
} partial_eq_default;

template<class T>
constexpr inline auto partial_eq = partial_eq_default;

In this case, though, there probably ought to be an opt in so that the trait can be checked by concept. An opt in mechanism is a bit verbose, but not necessarily complicated:

template<class T> constexpr auto partial_eq_type = false;
template<> constexpr auto partial_eq_type<int> = true;
template<> constexpr auto partial_eq_type<double> = true;

template<typename T>
concept is_partial_eq =
  partial_eq_type<T> &&
    requires(T lhs, T rhs) {
    partial_eq<T>.eq(lhs, rhs);
    partial_eq<T>.ne(lhs, rhs);
};

constexpr bool h(is_partial_eq auto lhs, is_partial_eq auto rhs) {
    return partial_eq<decltype(lhs)>.eq(lhs, rhs);
}

I have not done a good job at allocating names to the various bits and pieces. Please excuse this.

What have I missed?

We've been making variable templates more capable in many ways, and the concept checks to ensure correctness are new, but has anyone else explored this and found insurmountable problems?

Special Blocks and Emacs Org-mode Export

Steve Downey — Fri, 03 Dec 2021 05:00:00 GMT

Document number:
Date: 2021-12-03
Author: Steve Downey <sdowney2@bloomberg.net>, <sdowney@gmail.com>
Audience: WG21 Emacs Authors

Abstract: Making emacs org-mode more usable for writing WG21 papers.

1. Delete blocks and text

{\color{red}\bfseries

Remove this terrible old stuff.

}

And the bad ~~word~~ from this sentence.

2. Insert blocks and text

Add this wonderful new text

And put this cool word in.

3. Comparison Table

Old Busted	New Hotness
int a = 3 + 2 + 1;	int a = cool_sum({1, 2, 3});

A local CMake workflow with Docker

Steve Downey — Sun, 03 Oct 2021 04:00:00 GMT

l#+BLOG: sdowney

An outline of a template that provides an automated workflow driving a CMake project in a docker container.

This post must be read in concert with https://github.com/steve-downey/scratch of which it is part.

Routine process should be automated

Building a project that uses cmake runs through a predictable lifecycle that you should be able to pick up where you left off without remembering, and for which you should be able to state your goal, not the step you are on. make is designed for this, and can drive the processs.

The workflow

Update any submodules
Create a build area specific to the toolchain
Run cmake with that toolchain in the build area
Run the build in the build area
Run tests, either dependent or independent of rebuild
Run the intall
(optionally) Clean the build
(optionally) Clean the configuration

All of the steps have recognizable filesystem artifacts, need to be run in order, and if there are any failures, the process should stop.

So we want a make system on top of our meta-make build system.

The one thing this system does, that plain cmake doesn't automate, is making sure that any changes are incorporated into a build before tests are run. CMake splits these, in order to provide the option of running tests without a recompile. I think that is a mistake to automate, but I do provide a target that just runs ctest.

My normal target is test

make test

This will run through all of the steps, but only those, necessary to compile and run tests. The core commands for the build driver are

compile: Compile all out of date source
install: Install into the INSTALL_PREFIX
ctest: Run the currently build test suite
test: Build and run the test suite
cmake: run cmake again in the build area
clean: Clean the build area
realclean: Delete the build area

There are several makefile variables controlling the details of what toolchain is used and where things are located. By default the build and install are completely out of the source tree, in ../cmake.bld and ../install respectively, with the build directory further refined by the project name, which defaults to the current directory name, and the toolchain if that is overridden.

The build is a Ninja Multi-config build, supporting RelWithDebInfo, Debug, Tsan, and Asan, with the particular flavor being selectable by the CONFIG variable. See targets.mk for the variables and details, such as they are.

Because other tools, unfortunately, rely on a compile_commands.json this system symlinks one from the build area when reconfiguration is done.

default: compile

$(_build_path):
    mkdir -p $(_build_path)

$(_build_path)/CMakeCache.txt: | $(_build_path) .gitmodules
    cd $(_build_path) && $(run_cmake)
    -rm compile_commands.json
    ln -s $(_build_path)/compile_commands.json

compile: $(_build_path)/CMakeCache.txt ## Compile the project
    cmake --build $(_build_path)  --config $(CONFIG) --target all -- -k 0

install: $(_build_path)/CMakeCache.txt ## Install the project
    DESTDIR=$(abspath $(DEST)) ninja -C $(_build_path) -k 0  install

ctest: $(_build_path)/CMakeCache.txt ## Run CTest on current build
    cd $(_build_path) && ctest

ctest_ : compile
    cd $(_build_path) && ctest

test: ctest_ ## Rebuild and run tests

cmake: |  $(_build_path)
    cd $(_build_path) && ${run_cmake}

clean: $(_build_path)/CMakeCache.txt ## Clean the build artifacts
    cmake --build $(_build_path)  --config $(CONFIG) --target clean

realclean: ## Delete the build directory
    rm -rf $(_build_path)

To Docker or Not to Docker

The reason for the separate targets.mk file is to simplify running the build purely locally in the host, or in a docker containter. The structure of the build is the same either way. In fact, before I dockerized this template project, there was a single makefile which was mostly the current contents of targets.mk. However, splitting does make the template easier, as project specific targets can naturally be placed in targets.

Tha outer Makefile is responsible for checking if Docker has been requested and for making sure the container is ready. The makefile has a handful of targets of its own, but otherwide defers everything to targets.mk.

use-docker: set a flag file, USE_DOCKER_FILE, indicating to forward to docker
remove-docker: remove the flag file
docker-rebuild: rebuild the docker image
docker-clean: Clean volumes and rebuild image
docker-shell: Shell in the docker container

The docker container is build via docker-compose with the configuration docker-compose.yml. It uses the Dockerfile which uses steve-downey/cxx-dev:latest as the base image, and mounts the current source directory as a bind mount and a volume for ../cmake.bld.

I don't publish steve-downey/cxx-dev:latest and you should build your own BASE. I do provide the recipe for the base image as a subprojct in docker-inf/docker-cxx-dev.

You running unknown things as root scares me.

The image is assumed to provide current version of gcc and clang as c++ or gcc, or clang++ respectively.

The intent of the image is to provide compilation services and operate as an lsp server using clangd. Mine doesn't provide X, editors, IDEs, etc. The intent isn't a VM, it's a controlled compiler installation.

Compiler installations bleed in to each other. Mutliple compilers installed onto the same base system can't be assumed to behave the same way as a compier installed as the only compiler. The ABI libraries vary, as do the standard libaries. Deployment just makes this all an even worse problem. As a Rule I use for production Red Hat's DTS compilers and only deploy on later OSs than I've built on, with strict controls on OS deployments and statically linking everything I possibly can.

The base image I am using here, steve-downey/cxx-dev, works for me, and is avaiable at https://github.com/steve-downey/docker-cxx-dev as a definition as well.

It is based on current Ubuntu (jammy), installs gcc-12 from the ubuntu repositories, adds the LLVM repos and installs clang-14 from them based on how https://apt.llvm.org/llvm.sh does.

It then installs the current release of cmake from https://apt.kitware.com/ubuntu/ because using out of date build tools is a bad idea all around.

I also configure it to run as USER 1000, because running everything as root is strictly worse, and 1000 is a 99.99 percent solution/

.update-submodules:
    git submodule update --init --recursive
    touch .update-submodules

.gitmodules: .update-submodules

.PHONY: use-docker
use-docker: ## Create docker switch file so that subsequent `make` commands run inside docker container.
    touch $(USE_DOCKER_FILE)

.PHONY: remove-docker
remove-docker: ## Remove docker switch file so that subsequent `make` commands run locally.
    $(RM) $(USE_DOCKER_FILE)

.PHONY: docker-rebuild
docker-rebuild: ## Rebuilds the docker file using the latest base image.
    docker-compose build

.PHONY: docker-clean
docker-clean: ## Clean up the docker volumes and rebuilds the image from scratch.
    docker-compose down -v
    docker-compose build

.PHONY: docker-shell
docker-shell: ## Shell in container
    docker-compose run --rm dev

Work In Progress

I expect I will make many changes to all of this. I'm providing no facilities for you to pick them up. Sorry.

Please consider this as an exhibition of techniques rather than as a solution.

std::execution, Sender/Receiver, and the Continuation Monad

Steve Downey — Sun, 03 Oct 2021 04:00:00 GMT

Some thoughts on the std::execution proposal and my understanding of the underlying theory.

What's proposed

From the paper's Introduction

This paper proposes a self-contained design for a Standard C++ framework for managing asynchronous execution on generic execution contexts. It is based on the ideas in [P0443R14] and its companion papers.

Which doesn't tell you much.

It proposes a framework where the principle abstractions are Senders, Receivers, and Schedulers.

Sender: A composable unit of work.
Receiver: Delimits work, handling completion, exceptions, or cancellation.
Schedulers: Arranges for the context work is done in.

The primary user facing concept is the sender. Values and functions can be lifted directly into senders. Senders can be stacked together, with a sender passing its value on to another function. Or stacking exception or cancellation handling the same way.

Receivers handle the three ways a sender can complete, by returning a value, throwing an exception, or being canceled. As described, receivers are most likely to be implemented within particular algorithms that combine senders, such as `then` or `retry`.

Schedulers provide access to execution contexts. Like inline, single thread, a thread pool, a GPU, and so on, would all have schedulers that provide for putting a sender into the context they manage.

There's a fairly large API surface being proposed. But there's an underlying theory about this, governing what algorithms need to be there and how the pieces fit together.

Continuation Passing Style and the Continuation Monad

Continuation passing style is a transformation from a normal function and a call stack to a direction to send the result to the "continuation" without returning. This means the functions context can be cleaned up. Delimited continuations are a slight variation, where instead of an unbounded "rest of the program", the continuation has an end point and a value. It's essentially a function, and can be handled as such. There is a purely mechanical method for converting all of the lambda calculus transforms into CPS form, and this can be profitable for compilers based on lambda, or related logics, like system F.

The mechanical transformation also means that all the control structures, like loops, gotos, coroutines, exceptions, have CPS equivalents.

CPS is tedious, though. Having to explicitly add a continuation to everything is complicated.

However, there's also a typeclass, or concept, that allows you to convert regular functions into continuation passing style, automatically. It's then rather straightforward to involve concerns like where work is being run for something that wraps up the entire work. Even being able to switch back and forth between contexts. That's the continuation monad.

And unfortunately monads became an organizing principle in programming language theory one or two decades after most CS programs were standardized. So it's all complicated and involves things we weren't trained on. Fitting it into C++ has been an ongoing challenge, and until we had generic lambda was neither reasonbly concise nor idiomatic.

See, however, the new monadic interface additions for std::optional for why you want this. Or Ranges, which are solidly based in the 'list' or non-deterministic monad.

We have a poor relationship with Theory

There is no satisfactory PL theory for object oriented programming. There's lots of work, but it mostly ends up describing something that OO programmers don't think is quite the same as what they do. Even the ones who spend a lot of time doing theory.

Yet OO was, and is, a successful discipline. Working with identity, behavior, and state has produced remarkable results. Temporal calculi, not so much.

For a long while, we as a discipline thought that multi-threading was similar. There was poor theory, but we had hardware, and libraries that let us use that hardware to do concurrent work correctly.

That turned out not to be the case.

Concurrency can't be just a library, unfortunately. Concurrency models that hardware vendors will commit to won't promise not to violate causality. That makes producing a programming model programmers can use frighteningly difficult.

Which is why having a sound theory for std::execution is a good thing, even if the theory is unfamiliar.

But as a group, we learned the wrong lessons from the 80s and thought it was a researcher's job to take the successes of practitioners and put a sound basis to them. Ignoring that it is a feedback loop. In the 60s and 70s, those researchers were also the practitioners. It's not wrong to get out ahead of theory, but we do need to check back.

p2300 std::execution

Senders, via the Decorator pattern, lift ordinary functions into the continuation passing style. People writing functions only need to be concerned with handling the arguments they are passed, without concern for execution context or continuations. Functions used by senders act like, and are, normal functions.

Senders manage a bundle of channels, representing normal return of a value, throwing an exception, or an error channel to handle cancellation, or other errors not within the bound of ordinary functions. All of these channels can be composed taking the result to another function, or monadically with a function returning a sender, where that function can determine the kind of sender based on the values of the arguments. The channels can be combined or rerouted, connecting one to another, or presenting a variant containing either result, exception, and/or error to the continuation function.

Although senders form a logical graph of units of work, the physical type model is containment, much like expression templates. The result of binding senders together via an algorithm is a sender that contains the bound together senders. There are no nodes or allocations inherent to the model, just function calls.

C++ coroutines fit into this model. C++ coroutines are, from the outside, functions with rules about the interaction patterns with the returned value. Making a coroutine owning type a sender, and a sender co_awaitable, is possible and has been demonstrated.

std::execution takes the Continuation Monad and fits it to C++ control flow, return or exception, and adds cancellation, which incidentally allows a channel for failures from execution contexts. The thread pool can potentially signal failure via the error channel, without aliasing problems from application function code. However, for advanced users, these can be folded back into the normal function arguments and handled by application code. Policy decisions are not burned into the ROM of std::execution, but there are defaults that can be provided by application infrastructure authors.

Those infrastructure authors do not have to be std library vendors. The protocols, rendered as concepts, are available to normal users.

Network TS

Eppur si muove: And yet it moves

I do not believe ASIO's model is a firm foundation for all async programming. However, it is well proven, and exists. It works.

And …

I have confidence that a networking library can and will be built using p2300. I am less confident that can be done in the timeframe for C++26. I do not believe for a moment we could have one for C++23, even with an existence proof a networking library appearing now. It's simply too late to review and agree. We're in the same place as coroutines. We can have the machinery, but without all of the application user facing infrastructure we should have.

I think this was the right choice with coroutines, and I think providing the machinery for general continuation based async in the standard library so that we can build on top of it is the right choice. The authors have committed to making sure all the facilities are available for programmers, in particular the pipe syntax (an issue for ranges) as well as providing bases or adapters for coroutine promises and typed senders. We can experiment and add existing practice as we go.

Disclaimer

This is all my personal opinion, based on my own understanding. I've been in the meetings, I've been in discussions, asked questions. But if I'm wrong about some aspect of the proposal, that's on me. Certainly not a formal opinion of Bloomberg, where I work. While we do lots of network services, and async programming, this isn't what our tech looks like at all. Getting from here to there is an open question, but it would be for ASIO, too.

At least it isn't CORBA.

Source For Blog

Standard Vocabulary for Algorithms

Steve Downey — Sat, 16 Jan 2021 23:22:19 GMT

This is feedback after considering A Plan for C++23 Ranges Disclosure: I voted in favor of this. It does not suggest work on views::maybe [P1255R6]. I'm fine with that priority.

Vocabulary is not just Types between Components

There's broad agreement that 'vocabulary types' belong in the standard. Domain independent types that can be used between otherwise uncoupled facilities. They each depend on facilities provided by Standard C++ and can thus be developed independently. Each component can rely on the API of a core vocabulary type not changing. This allows facilities to communicate effectively. A component can have std::string or std::vector<int> in its interface with little concern. The STL parts of the standard library have always relied on concepts for constraining template parameters implicitly, requiring an Iterator of particular category in places, or specializing algorithms based on such. Composition of container types has also been natural. No one is confused by a std::unordered_map<std::string, std::vector<my::person>>. The new C++20 range facilities allow composition of algorithms, which means that the names of algorithms will, and must, become part of the vocabulary of C++ programmers. Algorithms, and the names for them, are far broader than the linear containers supported by C++ today. It is a particular problem because C++ has named core algorithms badly, as they were not, until recently, first class entities.

Algorithms are now first class entities

Functions are a terribly overloaded term in C++. Overloaded is an overloaded term. With respect to ranges and views, however, it should be clear that the adaptors, transformers, and other entities we apply to ranges and views to produce new ranges, views, and values, are now first class entities in their own right. The important thing, for programmers, is that moving up the ladder of abstraction should not be conceptually more difficult than dealing with existing libraries of functions. We have algorithms that deal with generic types rather than functions that deal with concrete types, but keeping them in mind should not be fundamentally more difficult. Naming is therefore critical. As are chosing the algorithms that are primitive, that is, the ones that compose into others in useful ways. I have some concerns about providing fused algorithms that a sufficiently smart compiler could optimize. On the other hand, I doubt how imminent sufficiently smart compilers are.

Why Haskell keeps coming up

Python was the answer to "Why can't I just write pseudo-code?" I have text books where the authors seriously discussed writing out your algorithm into pseudo code, and then translating it into actual programming language. Programming languages were too high ceremony to discuss what you wanted to do, while normal English was not structured enough. "Scripting" languages gave precise higher level semantics that were implicit in the pseudo code, or it would have been impossible to translate by hand to other computer languages. Haskell was designed to do that for algorithm research. The primary goal was to be able to give direct executable form to a research paper. This was early replicable research. The paper provided, on its own, the mechanism to test and reproduce the result. Haskell source code can be embedded in LaTeX and given directly to a Haskell compiler. Every interesting algorithms paper for the last 30 years or so has implicitly included the prelude. Haskell itself is not a terribly complex language. Much of it is specified as sugar where a form is translated into a more primitive expression, as range for loops are in C++. Its power for researchers is in its inherent laziness, that an expression is not evaluated unless it is required, and its purity, that values are immutable. These properties make it straightforward to reason with and about. The core property of lazy evaluation has forced it to be pure, not allowing side-effects. This has forced it to be exemplary, and evolutionary, in supporting value oriented programming. These factors are why we end up often looking to Haskell. Value oriented programming has been an emerging style in modern C++. The new Range facilities embody the style. Being able to crib proveably correct facilities from modern programming language research is a benefit. There was a long period, in which C++ grew up, that academic research lagged behind the pragmatic craft of programming. This has, in retrospect, not been an unqualified success. Parallelism and concurrency were widely implemented without theoretic underpnnings. We are still trying to put C++'s memory model on firm footing. Many constructs regularly used in the past are now understood to be unsound, in explicable ways. Multi-threading should not be an experimental science. Object orientation is an area which still does not have satisfying theoretic backing. There are various theories that provide many of the same benefits, allowing abstraction and encapsulation. In OO, however, objects have identity and state, which implies that equational reasoning can fail, as two equivalent object may have distinct identity and then different behavior. Temporal calculus formalisms are still open research. The manifest success of OO{A,D,P} with limited underlying theory has allowed many practitioners to believe that academic research is irrelevant. To the extent of dismissing rigor. Jokes about the "M-word" are emblematic.

Sidebar: Monads are boring

A monad is a generic type with a few operations, some of which can be expressed in terms of the others, that follow a few fundamental and fairly trivial rules. They allow predictable composition of entities that model monad. It is not about `then`. Although `then` does tend to fall out. The interesting thing is that algorithms can be built that take generic types that follow some "Laws", and they will do, something, correctly. For some kinds of generic type, what happens may be surprising, even interesting. The surprises will, however, not break anything. That's the point of the semantic guarantees surrounding the operations. C++ has philosophically adopted semantics surrounding Concepts. A Concept is entitled to demand behavior and laws in addition to the checkable syntax that can be enforced. The utility for an algorithm writer is that reasoning can be applied at a much higher level. Even beyond the abstraction that this is a Container, or Range, or View, being able reason about the operations for anything that provides a set of well defined operations means that the algorithm is correct, even if the details when applied to a new kind are obscure. Iterators provide a way for algorithms to work with many containers. Those same, correct, algorithms should be able to work with things that aren't Iterable, or Containers. It turns out having a named way of converting a Type<Type<Value>> to a Type<Value> opens up many algorithms to broader use. Flattening a vector of vectors into a linear vector is not the only use of a `concat`.

Fold: In Which I Become Cranky About Names

One of the most primitive and yet powerful algorithms is `fold`. Fold takes a binary operation and interposes it between elements in an iterable data type. It replaces the commas {a, b, c, d, …} with `op`, resulting in {a op b op c op d … } Except that is slightly ambiguous. Is that (a op (b op (c op (d)))) or ((((a) op b) op c) op d) Right or left evaluation of the terms. Folding right is strictly more general, at least in the lambda calculus. Left fold can be implemented as a right fold, but not vice-versa. However, in C++, the normal implementation of folds is not as general as it might be, because C++ is strict in evaluating function arguments. This means that even though a fold whose operation might not need to evaluate one of its arguments in order to return a value, will still do so, because the argument is evaluated before being passed to the operation. This mean, in particular, that we can't simply undo fold by using a `cons` operation, reconstructing the list. The entire, indeterminate and possibly infinite, list must be evaluated even though all that is necesary is the first element. Left folding is tail recursive, and so can be converted to a loop. This is why it seems natural in C++.

T fold_left(Range&& r, T init, BinaryOperation op) {
    range_iterator_t<Range> b = begin(r);
    range_sentinel_t<Range> e = end(r);
    for (; b != e; ++b) {
        init = op(std::move(init), *b);
    }
    return init;

Right fold is not generally tail recursive, although for finite sequences in C++, using reverse iterators, foldr is also a loop.

auto fold_right_loop(Range&& r, T init, BinaryOperation op) -> result_of<>{
    range_iterator_t<Range> b = begin(r);
    range_sentinel_t<Range> e = end(r);
    for (; b != e; ++b) {
        init = op(std::move(init), *b);
    }
    return init;

To make the diagrams below pretty, we assume that foldl over a sequence of T takes a single value of type Z and an op of Z -> T -> Z, while foldr over a sequence of T takes an op of type T -> Z -> Z and a value of type Z. For a left fold, the value is given to the op that takes the first element in the sequence, while for a right fold, it is given to the op that takes the last element in the sequence.

result = op(op(op(op(op(op(op(op(z, a), b), c), d), e), f), g), h);

result = op(a, op(b, op(c, op(d, op(e, op(f, op(g, op(h, z))))))));

So, left fold piles up the ops on the left and right fold piles them up on the right. The 'zero' or init value is with the last value for a right fold, and the first on a left fold. In converting to iteration, which is important, left fold is in the natural direction, and right fold is reversed, which means it can't be used that way on some sequences. So which one should be the fold. It's an important question because short names are implicitly privileged. Programmers assume that the short name is the best one, and longer names indicate more specialized usage. This isn't true for C++, but not really on purpose. We did think that `map` was a good choice. The programmers should probably reach for `unordered_map` instead is unfortunate, but at this point unfixable. Since C++ is strict, it would seem that left fold is the fold you mostly want, so it would be OK to call it fold. Haskell, as lazy as it is, has issues with folds on ranges with indeterminate bounds. foldl vs foldr vs foldl' is a constant question. Foldr Foldl Foldl' Memory consumption of either stack or unevaluated expressions will surprise programmers. But in the literature, fold is right fold. In the pure lambda calculus, right fold is strictly more expressive. Tail recursion is a subset of primitive recursion, and left is tail recursive while right is not. If we call our left fold, fold, we will confuse people. Further, the forward iteration is somewhat balanced by the fact that the normal addition to a sequence is emplace_back or push_back, which means that left fold will tend to reverse lists if the operation is accumulation into a sequence, if for example the op were [](auto t){v.emplace_back(t);}. More, we have started to introduce lazy features into the language, with coroutines. Although we won't get them soon, the C++ standard library should eventually have good lazy algorithms. It would be unfortunate if we have to teach that you can't use co_fold in place of fold because they are opposite, or that co_fold_left is how you spell the coroutine fold. [Note co_ prefixes are mildly sarcastic.] C++ has a long history of setting defaults and realizing later that they are a problem. I would much prefer we not make a choice about which fold is the fold, and just spell them as fold_left and fold_right. We should all be able to get behind not calling it accumulate, even if we must live with transform.

Monoid

Some other time I'll vent about the various fold operations and how they get extended if your type is a monoid. See Chris Di Bella's work such as A Concept Design for the Numeric Algorithms, although this rant suggests that algorithms that operate on different Concepts should have different names. Overloading on concepts with semantic differences can be confusing.

What Comes to Mind

Building vcpkg dependencies with project toolchain

1. Toolchain is more than just the compiler

2. Using a toolchain with vcpkg in-project

3. Building Dependencies with your Toolchain.

4. Someone is Wrong on the Internet

Substitution is Sometimes a Failure

1. Optional "Monadic" Interface

2. SFINAE is shallow

3. Constraints, what are they good for

4. Can we do better?

Nikola Blog Infrastructure

1. Nikola

2. Orgmode plugin

3. LaTeX support via KaTeX

Concept Maps using C++23 Library Tech

Abstract

Lost with Concepts-Lite

Why?

Alternatives

Hard to Support

Example from C++0x Concepts

Student Record

Equality Comparable

Allow associated types

Why Didn't We Get Them?

State of the Art

Rust Traits

C++ CPOs

Some Concepts and Types

C++ partial_equality

Requirements for Solution

What does typeclass do?

Type-based lookup

Additional Requirements

Variable templates

A Step Towards Implementation

partial_eq<T>

An inline variable object

A default implementation

New has_eq

Will do better

Monoid

Maybe not a lot more

Math

Function form

Core Functions

From Haskell Prelude

Minimum Set

In C++

Implementing the other side

The Map for a Monoid

identity

concat

op

Deducing this and CRTP

Plus

PlusMonoidMap

The map instances

Can we concat instead?

The Map and instance

Some simple use

Exercise the functions

Some simple cases

On std::string

Results

test int

test long

test char

test string

Monoid in Trees

Foldable generalizes

Summarizing Data in a tree

fingertrees

fringe-tree

Code

Summary for Concept Maps

Questions?

Thank You

Slides from C++Now 2023 Async Control Flow

2. Using a toolchain with `vcpkg` in-project

Lost with `Concepts-Lite`

`partial_eq<T>`

New `has_eq`

`identity`

`concat`

`op`

Deducing `this` and CRTP

`Plus`

`PlusMonoidMap`

Can we `concat` instead?

On `std::string`

`fingertrees`

`std::execution`

Underlies `std::execution`

`just`

`then`

`let_value`

`just`

`then`

`let_value`

`tst` function

`concept_map`