All Systems operational.

Systems

Incidents

2017-01-20 23:28:33 UTC resolved staged-recipes
staged-recipes blocked

ATM some recipes may be blocked in staged-recipes. Details and discussion can be redirected to the linked issue.

xref: https://github.com/conda-forge/staged-recipes/issues/2259

Update 2017-01-21 02:45:21 UTC

The cause of the issue has been narrowed down and a workaround is being put in place.

xref: https://github.com/conda-forge/staged-recipes/pull/2261

Update 2017-01-21 02:57:18 UTC

Reopening until we actually clear staged-recipes.

Update 2017-01-21 18:17:29 UTC

staged-recipes is now cleared thanks to the workaround in place.

2016-12-07 15:24:01 UTC resolved CircleCI
PR Builds not starting

Appears CircleCI builds for PRs are not starting. No known workaround at this time.

Update 2016-12-07 15:29:16 UTC

Have emailed support.

Update 2016-12-07 15:32:13 UTC

cc @conda-forge/core

Update 2016-12-07 16:05:18 UTC

I have receive a reply. Please see below.

We corrected a bug where we were allowing forked PR builds to run on Open Source projects even if that feature was disabled.

To go back to the functionality you are used to, please enable Permissive building of fork pull requests in the advanced settings for the project on CircleCI.

In short, it sounds like we were relying on a bug for PRs to work at all.

Update 2016-12-07 16:15:06 UTC

Should note that under this setting I see the following text.

There are serious security concerns with this setting (see the documentation for details.) If you have SSH keys, sensitive env vars or AWS credentials stored in your project settings and untrusted forks can make pull requests against your repo, then this option isn't for you!

Not something that inspires confidence. 😞 This may mean things deploy prematurely from PRs if these environment variables are exposed. Am asking CircleCI about this.

Update 2016-12-07 16:28:17 UTC

I do not think there are any security concerns for us. According to that document:

The variables configured via the circle.yml will be available in the fork PR builds (as they have access to the same circle.yml file); the values configured through the UI will not be visible to the fork PRs

I believe that the anaconda.org upload token is stored as a secure variable via the UI and therefore is not visible in PRs. This would be the only variable that I can think of that needs to be protected.

Update 2016-12-07 16:35:17 UTC

Going to Settings (the gear icon), Advanced Settings and switching the Permissive building of fork pull requests setting to On does in fact re-enable Circle CI build on pull requests. I turned this on in the python-feedstock and the PR I submitted is building on Circle (conda-forge/python-feedstock#96).

I think this will need to be done for all feedstocks. It might be worthwhile to ask Circle CI if they can do this for all the conda-forge repositories as manually updating 1000+ settings will be a pain.

Update 2016-12-07 16:49:05 UTC

I do not think there are any security concerns for us.

I hope you are right. Which document are you referring too?

It might be worthwhile to ask Circle CI if they can do this for all the conda-forge repositories as manually updating 1000+ settings will be a pain.

Also am asking about that too. It seems they don't have an API for us to do it. So even if we fix all of our repos, we will be stuck doing this manually when it comes to new repos.

Update 2016-12-07 16:52:50 UTC

I'm looking at the Building Pull Requests from forks document. The odd part is that according to that document pull request builds should be turned on for public GitHub repositories (including conda-forge) by default. I would be interested in seeing if Permissive building of fork pull requests is turned on for a newly created feedstock or not.

Update 2016-12-07 17:08:41 UTC

Should be easy to find out. 😉

Update 2016-12-07 17:18:48 UTC

Thanks for looking into this @jakirkham. I have some day job work that needs attention but will be back tonight to keep the ball moving towards conda build 2!

Update 2016-12-07 20:01:45 UTC

So apparently turning on Permissive build of fork pull requests also exposes the Environment Variables for the repository to pull requests which results the package being uploaded from the pull request's Circle CI builds rather than after merging, for example in this glib build.

That is a serious security concern as packages are getting uploaded without review and the upload token could can be exposed by a malicious PR.

Update 2016-12-07 20:04:06 UTC

Thanks for confirming this. Will ask what we can do about it.

Update 2016-12-07 20:16:39 UTC

Ahh wait. Apparently this behavior is what is expected when the Project fork pull requests toggle is set. So we should definitely NOT be doing that, sorry I suggested it.

Still does not explain why pull requests were not triggering builds.

Update 2016-12-07 20:32:46 UTC

Still does not explain why pull requests were not triggering builds.

The explanation is we were relying on a "bug" to get pull requests to work before. Yesterday they "fixed" the "bug". Though IMHO it was a feature and they just broke it. 😢

Update 2016-12-07 20:37:32 UTC

Though IMHO it was a feature and they just broke it.

Yes, and a very important feature at that. It looks like the same thing is happening with the Circle CI build on SciPy. I've opened an issue (scipy/scipy#6839) to track the issue there as well.

Update 2016-12-08 06:39:59 UTC

CircleCI said they are looking into it. Let's wait and see what comes out of it.

Update 2016-12-12 22:42:30 UTC

What are CircleCI thinking?!? Why would you ever want to expose your secrets through forked PRs? If you do actually want them exposed, you don't need to call them secret because anybody can echo them... it is a completely illogical move for open source, and a huge bug in their design.

Do you have a GitHub issue that we can xref?

Update 2016-12-16 18:28:08 UTC

Well, they have added a setting since that appears to disable exposing your secrets in a forked PR build. FWIW that appears to be off. I might try experimenting with this in one feedstock and see how it goes.

Update 2016-12-22 07:50:09 UTC

The current workaround is to enable CircleCI on one's own fork of the feedstock or staged-recipes and push new changes to get it to trigger with the other CIs.

Update 2016-12-24 02:29:29 UTC

Should add that another consequence of this problem is that sometimes recipes pending conversion in staged-recipes will be included in CircleCI builds as we are no longer able to do proper builds of PRs, but are instead having users build on their own fork. Fortunately it should not take away from the org queue on CircleCI though.

xref: https://github.com/conda-forge/staged-recipes/pull/1373 xref: https://github.com/conda-forge/staged-recipes/pull/1486

Update 2017-01-03 04:21:05 UTC

For details on how to enable CircleCI on your own fork, please see this doc.

Update 2017-01-04 18:38:46 UTC

Was able to enable CircleCI builds on staged-recipes. To do this generally on a feedstock, one has to navigate to the repo settings on CircleCI (gear in the upper right hand corner). Then select "Advanced Settings" in the left panel. Finally enable "Build forked pull requests" only. After doing this one must close and reopen the PR. This likely requires administrative privileges on CircleCI. Screenshot below.

screen shot 2017-01-04 at 13 35 42

Update 2017-01-08 17:25:30 UTC

You can enable this setting for all feedstocks by capturing the request for one feedstock for changing the setting in your browser (A request to https://circleci.com/api/v1.1/project/github/conda-forge/mpc-feedstock/settings) and then replaying it after changing the repo name in the url and payload.

Update 2017-01-08 21:05:30 UTC

Yeah, what I would really like to do is get this added to conda-smithy. Partly so that we can roll out this change on all feedstocks. Also partly to make sure new feedstocks have the right settings.

Guessing I should try using Selenium for implementing this, but maybe that is too much for clicking one radio button. Other suggestions welcome.

On a finer grained detail, I had looked at the source before and found even though I could identify the radio button. The source did not make it very clear which radio button belonged to which setting. My concern here is if I set everything up with say radio button 9 now and they change the settings latter, we could be getting the totally wrong behavior. So it would be nice if there was a good way of finding the setting name and then the radio button next to that setting. Any suggestions on this would be appreciated.

Update 2017-01-09 03:09:00 UTC

I checked and the following works. curl 'https://circleci.com/api/v1.1/project/github/conda-forge/mpc-feedstock/settings?circle-token=mytoken' -X PUT -H 'Content-Type: application/json; charset=UTF-8' -H 'Accept: application/json' --data-binary '{"feature_flags":{"build-fork-prs":true}}' Here mytoken needs to be replaced by a valid token. Implemented here, conda-forge/conda-smithy/pull/420

Update 2017-01-09 05:44:12 UTC

After working with @isuruf on his proposal for conda-smithy, I think we have a working fix. Have used it to enable CircleCI builds for PRs on all current feedstocks. Please take a look at the PR if you are interested.

IMHO we should get this CircleCI fix in, the re-rendering index fixes, and conda 4.2 fixes in. Then we should be in good shape for a healthy release.

Update 2017-01-10 23:34:42 UTC

Reopening until we actually release the new conda-smithy that will fix this issue.

Update 2017-01-11 21:17:05 UTC

Changed to degraded performance as many feedstocks are functioning at this point.

Update 2017-01-13 18:35:11 UTC

conda-smithy 2.0.0 is released and in action at staged-recipes. This release includes an important fix by @isuruf that enables CircleCI builds of PRs from forks on new feedstocks. So all new feedstocks should not have this issue. Also have applied this fix to all existing feedstocks. Therefore this should no longer be a problem for any feedstocks. Will close this out.

Fine print: We are using a non-public API to do this as there is no public API for it. So it may be changed in the future without warning.

Update 2017-01-13 19:09:03 UTC

Should add that I have asked CircleCI to make this part of the Public API. We'll see what they say.

2016-10-12 23:59:08 UTC resolved Anaconda.org
Download issues with Anaconda.org/conda-forge on CIs

Seeing intermittent download issues from conda-forge on CIs. Please restart builds if you encounter this issue. Have filed a bug report upstream to track this.

xref: https://github.com/Anaconda-Platform/support/issues/78

Update 2016-11-17 15:32:01 UTC

The primary issue seems to have been resolved with a new release of anaconda.org. There are still some linger issues, but they do not occur with near the same frequency and are actively being worked on by upstream as mentioned in the xref'd issue. Considering this resolved. Closing...

2016-08-24 01:59:49 UTC resolved Travis CI
Travis CI OS X Backlog

Seems that there is some serious issue affecting Travis CI builds and causing them to wait for long periods on the queue. Have raised this issue upstream as ( https://github.com/travis-ci/travis-ci/issues/6515 ) to see if they have any ideas as to what is going on and if we can get it fixed.

For the meantime, I'd like to ask people to nice builds. If you see multiple builds for a PR at staged-recipes, please cancel all older version of it. If you are planning any large running builds, please hold off on them until we can track down this issue. It does seem some builds are getting processed, but it appears to be very slowly and long after they were initiated.

Please bear with us as we try to sort out this issue.

Update 2016-08-24 03:12:44 UTC

Reducing to degraded as we seem to actually have some things running on the queue now. Also looks like the number of jobs running is starting to fall off. Will continue to monitor the situation.

Update 2016-08-24 18:47:04 UTC

Travis CI has now announced that they are stopping all OS X builds due to instability in their OS X infrastructure. Upgrading this to major outage.

Update 2016-08-25 03:12:53 UTC

Adding staged-recipes to this outage as conversions to feedstocks occur on Travis CI. So they will similarly be blocked until Travis CI comes back online.

Update 2016-08-25 18:58:09 UTC

It appears Travis CI is re-enabling OS X builds and are starting to work through the backlog. Downgrading to degraded performance. Will provide more updates as this progresses.

Update 2016-08-26 05:59:34 UTC

Have removed staged-recipes as they have cleared. Feel free to close this when you are comfortable with Travis CI's progress.

Update 2016-08-27 15:16:39 UTC

This seems to be resolved now. Going to go ahead and close this out. If people are still having issues, please let us know.

Update 2017-02-05 18:07:55 UTC

...there seem to be a long backlog on travis of staged-recipes again.

This status issue is on a different problem with Travis CI a long time ago. If there are new issues, I would recommend raising one. Though we are certainly aware of backlog issues with Travis CI that have been occurring recently.

...how do I cancel the expired PRs? There is no such option from my travis login.

Travis CI does not extend that privilege to anyone that does not have write access to the repo. I had put together a script ( https://github.com/conda-forge/conda-forge-build-setup-feedstock/pull/52 ) that would terminate old PR builds early and could be used at staged-recipes ( https://github.com/conda-forge/staged-recipes/pull/2264 ) or feedstocks (once the dust settles on the first two). However, it seems to be stuck in discussion ATM. Feel free to nudge it along if you think it is important.

Update 2017-02-06 04:42:12 UTC

Well this problem seems to show up on a weekly basis during the work week. So IMHO this is a big problem as it mucks with people's ability to get work done. That said, you are right it goes away on the weekend.

There are pros and cons to enabling on your fork. Pro being it is on your own queue vs conda-forge's. Con it still adds to the overall Travis CI queue and it fails to do things like test on the merge commit. Probably not worthwhile to enable Travis CI on your fork in the end.