CEP XXXX: Improving dependency export infrastructure

[h-vetinari]

First draft after discussion in #77. Does not contain (much) specification yet, because I'm unsure how to go about changing the schema of v1 recipes (does it need a bump in the schema version, or do we specify build tools must translate between them?), and how to deal with the repodata side of things. This is my first CEP, please excuse my lack of experience with a lot of the underlying details.

Help on these questions would be much appreciated! I decided to write up the design in more comprehensive form than originally in this comment though, in order to hopefully facilitate more effective discussion of how to solve the transition issues posed by the new design.

Closes #77 (eventually)

[h-vetinari]

pre-commit.ci autofix

[h-vetinari]

pre-commit.ci autofix

[h-vetinari]

pre-commit.ci autofix

[h-vetinari]

because I'm unsure how to go about changing the schema of v1 recipes (does it need a bump in the schema version, or do we specify build tools must translate between them?)

Wolf mentioned in private that we don't necessarily have to go to a v2 schema over this, because despite being renamed semantically, the new keys would just be extending the v1 schema, not breaking it. Of course, we'd then have to mandate mutual exclusivity between exports: and run_exports: etc., but I think that's probably a gentler approach to this than taking this comparatively minor issue as cause for "recipe v2".

The same approach (consider the new keys if present, error if not mutually exclusive with the old way) could even by used by conda-build to support the CEP¹, which would be great because a lot of our compiler feedstocks that would need this the most are not necessarily ready to be ported to v1 yet. :)

If people are in agreement over this approach, I can try to write up specification for it.

In any case, I think this is in a good enough state to ask for a first round of feedback; I'd be very curious to hear the thoughts of
@chenghlee @isuruf @jezdez @msarahan @wolfv, people from mamba, prefix, cf/core, and anyone else interested in this!

Footnotes

1. somewhat informally perhaps, since it's currently formulated atop of the v1 format. ↩

[cbouss]

I read the whole thread and think the proposed solution is a good addition. (From the perspective of writing recipes.)

[h-vetinari]

I read the whole thread and think the proposed solution is a good addition. (From the perspective of writing recipes.)

Thanks a lot for the feedback @cbouss, I appreciate you taking the time!

That reminds me that I wanted to post a comment¹ from a recent discussion on zulip which covers an open point of the design here: how to deal with host_to_run: exports of packages that were themselves injected via build_to_host: (i.e. not explicitly named in the consuming recipe).

Concretely, if we have

# output: a_complicated_package
requirements:
  exports:
    build_to_host:
      - some_package_with_a_run_export

and

# output: some_package_with_a_run_export
requirements:
  exports:
    host_to_run:
      - the_export_in_question

and then consume it

# output: mypkg
requirements:
  build:
    - a_complicated_package
  host:
    # from a_complicated_package's build_to_host
    # - some_package_with_a_run_export
  run:
    # ...should the run-export of some_package_with_a_run_export get triggered here?!
    # - the_export_in_question

the question is whether, why and how we trigger an export of a package that's not explicitly named in the recipe. Obviously this is quite an impactful question, and we certainly do not want to trigger all exports of packages that happen to transitively make it into an environment.

On zulip, I discussed this in the context of how injected exports could/would work. Despite not wanting to cover self-exports in this CEP, I've used the motivating use-case for them as an example here, because it illustrates the situation more concretely than just some abstract arithmetic between environments.

As I note at the end, this CEP could go either way (i.e. whether mypkg gets a run-requirement of the_export_in_question or not). However, I'm trying to take a wider view of the design space here, and to surface potential problems early. The flip side is that it's possible to go too far ahead in the "what if" exercise and get bogged down without ever achieving the first step. In any case, I'd be interested what people think.

---

Let's take the clearest case that I'm aware of that's been in contention around self-exports vs. conditional dependencies. I'm rephrasing here for consistent language: libB depends on libA as usual, but compiling against libB needs the specific version of libA used to build libB, e.g. due to the way ABI and headers between libA and libB interact.

In other words, if libB 1.0.0 *_1 is built against libA=1 and libB 1.0.0 *_2 is built against libA=2, we need to match the libA version constraint of mypkg that (generically) depends onlibB with the specific constraints on the artefact of libB that was in host: at build time.

Let's look at the recipes:

# output: libA
requirements:
  [...]  # regular build: / host: / run:
  exports:
    host_to_run:
      - ${{ pin_compatible("libA") }}

That's the easy part, which we already know as run_export: today. For libB, the recipe would use conditional dependencies

# output: libB
requirements:
  [...]  # regular build:
  host:
    - libA 
  run:
    # regular run-export from libA
    # - libA >={{ver_A}},<{{next_ver_A}}
    # additional requirement when compiling against libB; same effect as host_to_host
    - ${{ pin_compatible("libA") }}; if env.HOST  # how to spell this is TBD!
  exports:
    host_to_run:
      - ${{ pin_compatible("libB") }}

Now, whether as a conditional dependency or a host_to_host: export, using libB necessarily needs to inject a constraint on libA that's not present in the consuming recipe

# output: mypkg
requirements:
  [...]  # regular build:
  host:
    - libB
    # injected!
    # - libA  # matches libA-constraint of libB
  run:
    # regular run-export from libB
    # - libB >={{ver_B}},<{{next_ver_B}}
    # run-export from injected libA!
    # - libA >={{ver_A}},<{{next_ver_A}}
    - some_regular_dep

In my mental model, the process is as follows:

• Solve build: env
• determine exports: for packages that have been named or injected in build:, collect any build_to_host: or build_to_run:, call them B2H, B2R
• Add B2H to host:, solve
• determine exports: for packages that have been named or injected in host:, collect host_to_run:, call them H2R
• Save run: + B2R + H2R as run deps of the resulting artefact.

The "or injected" part is the most open aspect of the design space, but I believe the example with libA/libB above shows that we cannot rely only on named packages in such cases. So my proposed rule would be: "we apply exports from packages that are either explicitly named in the environment, or have been injected via exports or conditional dependencies". The second part of this rule only exists because there are cases that cannot be solved with just the first part.

Coming back to your example [for automatically injecting ${{ stdlib("c") }}], this could be written as

# mycompiler
requirements:
  run:
    - ${{ stdlib("c") }}
    # conditional dependency when in build, which then participates in run-exports; same as build_to_build
    - ${{ stdlib("c") }}; if env.BUILD

This would work with the "or injected" scheme above, but as you can see, it's neither very elegant nor obvious why you'd have to repeat the same dependency for mycompiler, and how that triggers the export from ${{ stdlib("c") }} in mypkg [from using mycompiler].

IMO that's because conditional dependencies and self-exports are features that are only truly needed for niche cases; any feature can be misused, so any additional expressivity needs to be guarded (e.g. by the linter etc.). As a consequence, I'm convinced that we shouldn't abuse this mechanism for something which every compiled recipe needs. In other words, ${{ stdlib("c") }} is too large a use-case to fit into this niche, and the consequences of trying to hide it are not worth the benefits.

This comment is already too long, so I'll just note that a more restricted form of [this PR] without the "or injected" is possible, and that this would already solve a lot of the cases where we need host-exports, e.g. the ABI of C++/Fortran modules. It's only when we get to stuff like the libA/libB case above that we really need to go beyond the "only named packages can contribute exports"; though I do think (compare the list of steps above) that the same mechanism would also be quite natural for chaining e.g. build_to_host: with host_to_run: exports.

Footnotes

1. I've edited the comment to leave out the bits not relevant here. ↩

[h-vetinari]

Obviously this is quite an impactful question, and we certainly do not want to trigger all exports of packages that happen to transitively make it into an environment.

One thing that gnawed at me since posting that comment on zulip is that I don't like the relationship "conditional dependencies participate in run-exports", which seems too magical, and is by far not explicit enough in the recipe IMO.

However, I just had an idea that might solve this case: we can use export: host_to_host: ... as the syntax to indicate this unusual mechanism, but implement it using conditional dependencies. IOW, libB from above would look like

# output: libB
requirements:
  exports:
    host_to_run:
      - ${{ pin_compatible("libB") }}
    host_to_host:
      # implemented not as an export, but as a conditional dependency of libB when it appears in `host:`!
      - ${{ pin_compatible("libA") }}

That would IMO be the best of both worlds: explicit syntax for the most unusual case, as well as a sane environment resolution process, and conditional dependencies don't have to be imbued with some magical pixie dust (pun intended 😉). This would also simplify the rule I had posited above to "we apply exports from packages that are either explicitly named in the environment, or have been injected via exports or conditional dependencies".

This would then give the following dependencies between the different proposals

---
config:
  securityLevel: loose
---
flowchart TB
    this["this CEP"]-->se["self exports CEP"]
    cond["conditional dependencies CEP"]-->se

Loading

[isuruf]

Nice that we are finding paths forward! Is this similar to the solution I proposed in #129 (comment), or does it differ in some way? If that’s the case, can you elaborate how it’s different?

[h-vetinari]

Nice that we are finding paths forward!

Glad to hear it. It's not for lack of wanting to solve the issue that I had descoped self-exports; now I'm beginning to see a path that allows the various pieces to work together in a non-hacky way (where before I couldn't see it at all).

Is this similar to the solution I proposed in #129 (comment), or does it differ in some way? If that’s the case, can you elaborate how it’s different?

It's different in that we do not have to add a condition like "build_to_build: has to match run:", but we still maintain the fact that it doesn't require multiple solves.

I've been getting down into the nitty gritty details at least one level deeper (e.g. @jaimergp opened another rabbit hole under my feet about the mechanics of ignore_exports:, and I had an illuminating chat with @baszalmstra about the steps that (can) happen between the metadata and the resolver); I'm planning to expand the CEP with the design conclusions from those discussions, which should hopefully tell the whole story better than yet another very long comment.

[h-vetinari]

I've written an update based on the various discussions. Since I'm trying to design things holistically (despite the fact that I want to scope this CEP to a manageable size without external dependencies), I've added a draft of the "self-exports" CEP for now. This goes into much more detail about how things should work in practice. I had a long discussion with @baszalmstra about the list of concrete steps, but I'll be the first to admit that I'm not very familiar with that part of the process, and any mistakes are almost certainly my own.

The procedure for self-exports is a superset of steps compared to the "regular" cross-exports, so describing the more complicated workflow should immediately show how the simpler procedure works (i.e. by dropping the steps marked [self]), and thus allow for unified discussions. Once the design has survived validation by various parties, I'll split off self-exports into a separate PR.

The key point relevant for self-exports that requires conditional exports (and avoids multiple solves) is that there is a necessary pre-processing step between fetching the package metadata and feeding it to the SAT solver. This preprocessing step is where we can apply self-exports (and any matching ignore_exports: rules).

In fact, conditional dependencies as implemented by resolvo are way more powerful than what we need here. Resolvo can even handle conditional dependencies that depend dynamically on other packages in the resolution, whereas everything we need here is statically known (i.e. "are we solving for the host: environment?" If so, all we need to do is turn host_to_host: exports into regular dependencies before handing to the SAT solver. This works even for transitive self-exports, i.e. X_to_X: chained with X_to_X: both get applied in the same solve).

Nice that we are finding paths forward! Is this similar to the solution I proposed in #129 (comment), or does it differ in some way? If that’s the case, can you elaborate how it’s different?

See 2f3d23f (though I recommend reading the updates to the main exports CEP first)