Sensible default for --inspect-channels

[ericdill]

If --upload-channels is specified but --inspect-channels is not, all recipes will be built and at the end of the recipe building, the following message will be printed out:

Assuming the distribution we've just built and the one on lightsource2/main are the same.

It seems like it would be sensible for --inspect-channels to default to --upload-channels if upload is present but inspect is not. Thoughts?

[pelson]

It seems like it would be sensible for --inspect-channels to default to --upload-channels if upload is present but inspect is not. Thoughts?

They are definitely things I'd like to control independently (sometimes you really do want to re-build everything, but only upload the things that aren't already available on the upload channel. Do you have a proposal for the interface if we change that default such that the existing behaviour can be maintained?

[ericdill]

They are definitely things I'd like to control independently (sometimes you really do want to re-build
everything, but only upload the things that aren't already available on the upload channel.

Why would you want to build everything even if it is already available on the upload channel?

Do you have a proposal for the interface if we change that default such that the existing
behaviour can be maintained?

Well, since this behavior is useful to you, I'll wait to hear why this is useful to you before I decide if I still think that it is sensible for --inspect-channels to default to --upload-channels.

[pelson]

Why would you want to build everything even if it is already available on the upload channel?

In order to assert that changes you've made to a repo result in recipes still building. e.g. you change a CI config, you update conda/conda-build, you don't know if a recent build of a dependency results in the build still succeeding.

Essentially, there is a huge gaping hole in staged-recipes and all of its forebears which means that I can't have confidence that a recipe continues to build.

[ericdill]

I can't have confidence that a recipe continues to build.

Fair point. Do you have any tooling to parse the output of conda-build-all or is this something that you inspect manually?

[pelson]

Fair point. Do you have any tooling to parse the output of conda-build-all or is this something that you inspect manually?

Sorry for not responding on this. No tools for inspecting the output (grep aside). It was also kind of asked for in #8.

[ericdill]

Sorry for not responding on this.

No worries!

No tools for inspecting the output (grep aside). It was also kind of asked for in #8.

Alright, thanks for the info. For firewall reasons, I'm using conda-build-all in a cron job to build tagged versions of our stack and so am working on some codeto parse the output of conda-build-all /path/to/recipes > log. I'd obviously be happy to share it once it's in more of a working state. Are you interested in potentially shipping a 'turn this log file into python objects' module with conda-build-all

[pelson]

Alright, thanks for the info. For firewall reasons, I'm using conda-build-all in a cron job to build tagged versions of our stack

I'm doing the same kind of thing with it. We should share notes! 😄

am working on some codeto parse the output

If you could just do:

conda-build-all /path/to/recipes --json-dry-run

Would that help?

[ericdill]

Reopening this issue since we still seem to have an active discussion on it.

[ericdill]

conda-build-all /path/to/recipes --json-dry-run

--json-dry-run would only be able to give the plan (which would still be quite useful!). I also need to be notified of build/test/upload failures so that I can fix the build script.

I think what I need to harden my build scripts is some sort of monitoring. The first thing that comes to mind is this workflow:

1. kick off build script that dumps output to a log file
2. parse log file for build failures, test failures and upload failures and turn that into a short summary
3. email that summary to myself.

I am quite close to having this above work flow functional, but I definitely do not like it primarily because I dislike parsing log files.

What would be really neat would be if there was a callback registry for before and after the each 'section' of conda-build, so pre-build, post-build, pre-test, post-test, pre-upload, post-upload, etc... Then you could hook in some live monitoring tools like a small flask app and I wouldn't have to parse logs any more 😃 But adding a callback registry to conda-build seems like a reasonably large undertaking, so I'll probably just stick with parsing logs and emailing them to myself.

In case you are curious, I've got a bash build script that currently builds tagged recipes with conda-build-all and master branch recipes with just calls to conda-build for each recipe and each python version.

I'm working on replacing the straight calls to conda-build with a python builld script for master branch builds that will check to see if the master branch build is already on the anaconda channel, but it is still a WIP. I also need to revisit my log parser and get that usable so I stop getting megabyte sized logs in my email...

Anyway this has just sort of turned in to a brain dump, sorry about that. I guess I have two main questions for you.

1. What are your feelings on parsing logs vs adding callbacks to conda-build or conda-build-all?
2. Would you be open to reworking conda-build-all to be able to handle master branch recipes?

[pelson]

What are your feelings on parsing logs vs adding callbacks to conda-build or conda-build-all?

I definitely agree that parsing the logs is less than ideal. Putting hooks into conda-build-all is doable, you'll notice that https://github.com/SciTools/conda-build-all/blob/master/conda_build_all/build.py#L19 splits the build and test phase, so conda-build-all has that scope. If I were implementing it, I'd be tempted to learn a bit more about entry points to see if they are a silver bullet which gives me the ability to allow user hooks. Alternatively, it is entirely possible to mock out some of conda-build-all's internals to get the hooks you need right now...

Would you be open to reworking conda-build-all to be able to handle master branch recipes?

I guess the problem is that we can't know if something needs building until we have got its source (and maybe even built it) - which doesn't sit well with me. How would you envisage it working?