So, here’s the thing - I posted a comment around this at the original site / post but it looks as if it’s been deleted.
The author basically is taking a big old poop all over CircleCI because of token / javascript bleed. Fine, that’s a reasonable potshot to make - BUT what’s not clear to me, and what I asked in my question - is has the author actually done the work to compare and contrast against another hosted CI provider?
This feels like a lazy smear to me. There are trade offs to be made when you choose to trust your sources to a hosted provider like this, and I am not convinced that CircleCI is doing anything at all untoward here other than needing to do a better job of communicating its dependencies to its users.
I tend to agree, but in a way you have to question the judgement of the developers when they include so many trackers. I don’t think it’s surprising that someone has latched onto this issue and it can be raised with many products that have a business model that doesn’t rely on advertising. Launch Darkly might actually be functional compared to the others listed by the author. I see Facebook references blocked by EFF’s Privacy Badger when I log into the product.
I wonder if these trackers are toggled off when I use a paid version of their product? I think that’s a possible trade-off since they are letting you test the product for free. Toggling these off might result in a loss of analytics data that they wouldn’t want, so there is a vicious circle.
I used CircleCI in a past job, and it does something pretty unique among SaS CI vendors - it lets you use an arbitrary number of VMs to run your unit tests.
So, we were able to halve the amount of time it took our unit tests to run by increasing the number of simultaneous servers running them. When you’re an org dealing with a legacy code base where the test corpus is taking HOURS and HOURS to run, that is some serious bottom line ROI right there.
They just need to update their comms to VERY CAREFULLY indicate what’s happening and everybody can choose to use them or not. What bugs me about the original post is the slapdash nature of the accusations levied and the lack of any kind of even handedness.
Also, when I posted my question to the original article, my comment was deleted. Their bat and ball, their rules, but if we’re gonna question judgement or motivation I think we can point the flying fickle finger of fate at this post’s author as well.
the slapdash nature of the accusations levied and the lack of any kind of even handedness.
I don’t understand this. What was I supposed to do differently be “even-handed”? Companies imbue CircleCI with an enormous amount of trust, and they do an incredibly risky thing with that trust. “If I am going to pay you thousands of dollars a month, please don’t build a dashboard where my source code gets stolen if someone hacks Quora.js” seems like a reasonable request. I suppose I could have said “By loading third party Javascript in a secure environment, CircleCI is picking up pennies in front of a steamroller, but in fairness, they do have the pennies.”
I posted my question to the original article, my comment was deleted.
I didn’t delete your comment; I manually approve all comments that appear on my website, there should have been a notice above the post that said “Comments are heavily moderated.”
I haven’t looked at your comment and couldn’t say, but generally I get low quality comments on posts and approving them isn’t really a priority for me.
Thanks for the clarification. Have you considered simply disabling comments for your blog in that case? Offering them but leaving them in limbo seems like a questionable practice. Your bat and ball, etc.
Thanks. Occasionally I get a good comment, one that does not describe hours of research and writing as a “lazy smear,” for that reason I like having the option to approve them.
Would you disagree that calling a particular vendor out for particular problems might warrant citing similar issues with other vendors in the name of even handedness?
You called my article a “lazy smear” because there are, supposedly, other CI companies that let arbitrary third party Javascript run in a trusted environment and access CSRF tokens/create API tokens that could result in data compromise, but you’ve provided zero evidence that such companies exist. I can, however, cite many CI tools that do not let arbitrary 3rd party Javascript run in a trusted environment, as Circle does: Phabricator, Jenkins, Gitlab, Travis.
Even if they did exist, no, I am under no obligation to cite them. The fact that many people drive drunk is not an excuse for your own decision to drink and drive. You are welcome to do your own research and post your own findings about other companies.
I wonder if these trackers are toggled off when I use a paid version of their product?
No, they are not, everyone gets them, even if you pay them thousands of dollars per month. You can collect analytics data on the server side, and there’s no way a compromise of your server side analytics provider affects the security of my source code.
This is like complaining that arresting someone for drunk driving is unfair because other people also drive drunk and did not get caught. Across the entire industry companies don’t care enough about third party javascript that runs on secure pages (dashboards, credit card input forms, API token creation, more). It’s a dangerous situation and consumers should demand better. I understand there are tradeoffs to be made letting a third party run CI, but I don’t think that “outsource my company’s source code security to Quora.js” is a reasonable one. I also think those companies overstate the benefit compared to the risks, the benefits are small and immediate, the risks are larger and unquantifiable.
There are steps they could take to secure important fields - for example, use a different domain for marketing/dashboards, require an HMAC token that the third party Javascripts don’t have access to, but they did not take those steps.
That depends a lot on your needs. For some people spinning up a Jenkins is far too heavyweight. Having experienced the joy of having to debug an awful Jenkins plugin to make it do basic things, I can tell you that Jenkins is not a sure fire win for everyone.
I agree. and they have in the past year or so gone to a much better security posture, and are much more proactive around security issues. They are still releasing security patches/updates regularly, but I imagine over time they will finally get most of that sorted out, and security updates/patches can become a dull-roar instead of a constant. They are very good about letting you know however, and they provide packages for most OS’s so it’s not difficult to apply, it’s basically just a restart. Overall very happy with them. I think there is a reason most large OSS projects use them (Debian, FreeBSD, etc.)
We use buildkite at my company. One nice aspect is that we get an agent to run on /our/ “hardware” (we just use large vm instances). It works pretty well.
It’s probably worth mentioning here that GitLab offers similar functionality with their GitLab CI offering. You can use their infrastructure or install runners (their equivalent of agents) on as many machines as you like. Disclaimer: I haven’t used either yet but attended a meetup event where somebody praised them highly and ditched their Atlassian stack for that single reason.
Their website looks intriguing could you elaborate on their security posture? Is it just an artifact of the on-premise build agent, or is there more to it than that?
If you happen to run on Heroku, Heroku-CI works quite well. You don’t wait in a queue—we just launch a new dyno for every CI run, which happens while you blink. It’s definitely not as full features as Circle, or even Travis, but it’s typically good enough.
At $WORK we run some things on Heroku but we can’t or don’t want to for most things — it’s either too expensive or the workload isn’t really well-suited for it.
What do you need? I like Travis, they also get vastly better when you actually use the paid offering and they offer on-premise should you actually need it.
But, just to be clear: your builds take 8-14 minutes. What takes time for you is the low concurrency settings on travis public/free infrastructure. It’s a shared resource, you only get so many parallel builds. That’s precisely why I referred to their paid offering: travis is a vastly different beast when using the commercial infrastructure.
I also recommend not running the full matrix for every pull request, but just the stuff that frequently catches errors.
You were asking in a public forum. I didn’t ask you to rebut or debate my experiences with TravisCI. https://github.com/cmhamill their email is on their GitHub profile if you’d like to speak with them without anyone one else chiming in. I’m relating an objection that is tied to real time lost on my part and that of other maintainers. It is a persistent complaint of other people I work with in OSS. I’m glad TravisCI’s free offering exists but I am not under the illusion that the value they’re providing was brought into existence ex nihilo with zero value derived from OSS.
It’s a shared resource, you only get so many parallel builds. That’s precisely why I referred to their paid offering: travis is a vastly different beast when using the commercial infrastructure.
We use commercial TravisCI at work. It’s better than CircleCI or Travis’ public offering but still not close to running a CI service on a dedis (singular or plural).
I had to aggressively cache (multiple gigabytes) the build for Bloodhound before it stopped timing out. I’m glad their caching layer can tolerate something that fat but I wish it wasn’t necessary just to keep my builds working period.
That combined with how unresponsive TravisCI has been in general leaves a sour taste. If there was a better open source CI option than something like DroneCI I’d probably have rented a dedi for the projects I work on already.
Would you rather pay for a licensed software distribution that you drop in a fast dedicated computer you’ve bought and it turns that computer into a node in a CI cluster that can be used like Travis?
Would you rather pay for a service just like Travis but more expensive and running on latest-and-greatest CPUs and such?
Would you rather pay for a licensed software distribution that you drop in a fast dedicated computer you’ve bought and it turns that computer into a node in a CI cluster that can be used like Travis?
If it actually worked well and I could test it before committing to a purchase, probably yes I would prefer that to losing control of my hardware or committing to a SAAS treadmill but businesses loooooooove recurring revenue and I can’t blame them.
Would you rather pay for a service just like Travis but more expensive and running on latest-and-greatest CPUs and such?
That seems like a more likely stop-gap as nobody seems to want to sell software OTS anymore. Note: it’s not really just CPUs, it’s tenancy. I’d rather pay SAAS service premium + actual-cost-of-leasing-hardware and get fast builds than the “maybe pay us extra, maybe get faster builds” games that most CI services play. Tell me what hardware I’m actually running on and with what tenancy so I don’t waste my time.
Has anyone done this kind of dependency scan on Travis that this guy did on CircleCI? I suspect you will see much the same.
Travis does have one clear advantage here in that it’s OSS so you can SEE its dependencies and make your own decisions. See my note about CircleCI needing to be better about communication above.
Well… “scan”. They posted a screenshot of their network debugger tab :).
Travis (.org) uses Pusher, but not their tracking scripts. It integrates Google Analytics and as such, communicates with it. ga.js is loaded from google.
The page connects to:
api.travis-ci.org
cdn.travis-ci.org (which ends up being fast.ly)
gravatar.com (loading avatar images)
statuspage.io (loading some status information as JSON)
fonts.googleapis.com (loading the used fonts)
ws.pusherapp.com
All in all, it is considerably less messy then circle-ci’s frontend.
CircleCI’s only sin here is one of a lack of communication. There is nothing actually wrong with any of the callouts the article mentions, they just need to be VERY sure that their users are aware of exactly who is seeing the source code they upload. This should be an object lesson for anyone running a SaS company, ESPECIALLY if said SaS company caters to developers.
This is not an apples to apples comparison, in my post I cited Javascripts only (which can make AJAX requests and extract source code), @skade cites that Travis loads fonts, images, and CSS from third party domains, which don’t have those properties; a compromise in CSS might change the appearance of a page but generally can’t result in your source code/API tokens being leaked to a third party.
As far as I follow the only external Javascript run by Travis CI is Pusher. So, no, it has not proven your point perfectly, in fact it demonstrates the opposite.
It’s really 2 companies in terms of analytics…
Optimizely is A/B testing as well.
So, here’s the thing - I posted a comment around this at the original site / post but it looks as if it’s been deleted.
The author basically is taking a big old poop all over CircleCI because of token / javascript bleed. Fine, that’s a reasonable potshot to make - BUT what’s not clear to me, and what I asked in my question - is has the author actually done the work to compare and contrast against another hosted CI provider?
This feels like a lazy smear to me. There are trade offs to be made when you choose to trust your sources to a hosted provider like this, and I am not convinced that CircleCI is doing anything at all untoward here other than needing to do a better job of communicating its dependencies to its users.
I tend to agree, but in a way you have to question the judgement of the developers when they include so many trackers. I don’t think it’s surprising that someone has latched onto this issue and it can be raised with many products that have a business model that doesn’t rely on advertising. Launch Darkly might actually be functional compared to the others listed by the author. I see Facebook references blocked by EFF’s Privacy Badger when I log into the product.
I wonder if these trackers are toggled off when I use a paid version of their product? I think that’s a possible trade-off since they are letting you test the product for free. Toggling these off might result in a loss of analytics data that they wouldn’t want, so there is a vicious circle.
I used CircleCI in a past job, and it does something pretty unique among SaS CI vendors - it lets you use an arbitrary number of VMs to run your unit tests.
So, we were able to halve the amount of time it took our unit tests to run by increasing the number of simultaneous servers running them. When you’re an org dealing with a legacy code base where the test corpus is taking HOURS and HOURS to run, that is some serious bottom line ROI right there.
They just need to update their comms to VERY CAREFULLY indicate what’s happening and everybody can choose to use them or not. What bugs me about the original post is the slapdash nature of the accusations levied and the lack of any kind of even handedness.
Also, when I posted my question to the original article, my comment was deleted. Their bat and ball, their rules, but if we’re gonna question judgement or motivation I think we can point the flying fickle finger of fate at this post’s author as well.
I don’t understand this. What was I supposed to do differently be “even-handed”? Companies imbue CircleCI with an enormous amount of trust, and they do an incredibly risky thing with that trust. “If I am going to pay you thousands of dollars a month, please don’t build a dashboard where my source code gets stolen if someone hacks Quora.js” seems like a reasonable request. I suppose I could have said “By loading third party Javascript in a secure environment, CircleCI is picking up pennies in front of a steamroller, but in fairness, they do have the pennies.”
I didn’t delete your comment; I manually approve all comments that appear on my website, there should have been a notice above the post that said “Comments are heavily moderated.”
I haven’t looked at your comment and couldn’t say, but generally I get low quality comments on posts and approving them isn’t really a priority for me.
Thanks for the clarification. Have you considered simply disabling comments for your blog in that case? Offering them but leaving them in limbo seems like a questionable practice. Your bat and ball, etc.
Thanks. Occasionally I get a good comment, one that does not describe hours of research and writing as a “lazy smear,” for that reason I like having the option to approve them.
Would you disagree that calling a particular vendor out for particular problems might warrant citing similar issues with other vendors in the name of even handedness?
You called my article a “lazy smear” because there are, supposedly, other CI companies that let arbitrary third party Javascript run in a trusted environment and access CSRF tokens/create API tokens that could result in data compromise, but you’ve provided zero evidence that such companies exist. I can, however, cite many CI tools that do not let arbitrary 3rd party Javascript run in a trusted environment, as Circle does: Phabricator, Jenkins, Gitlab, Travis.
Even if they did exist, no, I am under no obligation to cite them. The fact that many people drive drunk is not an excuse for your own decision to drink and drive. You are welcome to do your own research and post your own findings about other companies.
I would disagree. Someone pointed out problems, why should we demand extra work? We should be thankful that someone has pointed out problems.
No, they are not, everyone gets them, even if you pay them thousands of dollars per month. You can collect analytics data on the server side, and there’s no way a compromise of your server side analytics provider affects the security of my source code.
This is like complaining that arresting someone for drunk driving is unfair because other people also drive drunk and did not get caught. Across the entire industry companies don’t care enough about third party javascript that runs on secure pages (dashboards, credit card input forms, API token creation, more). It’s a dangerous situation and consumers should demand better. I understand there are tradeoffs to be made letting a third party run CI, but I don’t think that “outsource my company’s source code security to Quora.js” is a reasonable one. I also think those companies overstate the benefit compared to the risks, the benefits are small and immediate, the risks are larger and unquantifiable.
There are steps they could take to secure important fields - for example, use a different domain for marketing/dashboards, require an HMAC token that the third party Javascripts don’t have access to, but they did not take those steps.
I actually think the most sane solution at the moment is to use a self-hosted solution like Jenkins. It’s open and can do just about anything.
That depends a lot on your needs. For some people spinning up a Jenkins is far too heavyweight. Having experienced the joy of having to debug an awful Jenkins plugin to make it do basic things, I can tell you that Jenkins is not a sure fire win for everyone.
I agree. and they have in the past year or so gone to a much better security posture, and are much more proactive around security issues. They are still releasing security patches/updates regularly, but I imagine over time they will finally get most of that sorted out, and security updates/patches can become a dull-roar instead of a constant. They are very good about letting you know however, and they provide packages for most OS’s so it’s not difficult to apply, it’s basically just a restart. Overall very happy with them. I think there is a reason most large OSS projects use them (Debian, FreeBSD, etc.)
Here’s a question for the ages: are there any actually-existing good hosted CI providers out there?
Not if you need speed: http://bitemyapp.com/posts/2016-03-28-speeding-up-builds.html
I would honestly pay good money for reliable, tested deployment automation that stood things like CI up.
Who’d you end up going with for the dedicated server / what are the specs on that machine like?
Approximately this with NVMe RAID: https://www.ovh.com/us/dedicated-servers/infra/173eg1.xml
tbqh, most the time we saved on compilation was lost to the GHCJS build later on. I was very sad.
We use buildkite at my company. One nice aspect is that we get an agent to run on /our/ “hardware” (we just use large vm instances). It works pretty well.
Another vote for buildkite here - their security posture is markedly better and you have much more control over performance.
It’s probably worth mentioning here that GitLab offers similar functionality with their GitLab CI offering. You can use their infrastructure or install runners (their equivalent of agents) on as many machines as you like. Disclaimer: I haven’t used either yet but attended a meetup event where somebody praised them highly and ditched their Atlassian stack for that single reason.
Their website looks intriguing could you elaborate on their security posture? Is it just an artifact of the on-premise build agent, or is there more to it than that?
If you happen to run on Heroku, Heroku-CI works quite well. You don’t wait in a queue—we just launch a new dyno for every CI run, which happens while you blink. It’s definitely not as full features as Circle, or even Travis, but it’s typically good enough.
At
$WORKwe run some things on Heroku but we can’t or don’t want to for most things — it’s either too expensive or the workload isn’t really well-suited for it.What do you need? I like Travis, they also get vastly better when you actually use the paid offering and they offer on-premise should you actually need it.
I need builds to not take 25-30 minutes.
Bloodhound averages 25 minutes right now on TravisCI and that’s after I did a lot of aggressive caching: https://travis-ci.org/bitemyapp/bloodhound/builds/286053172?utm_source=github_status&utm_medium=notification
Gross.
I was asking cmhamill.
But, just to be clear: your builds take 8-14 minutes. What takes time for you is the low concurrency settings on travis public/free infrastructure. It’s a shared resource, you only get so many parallel builds. That’s precisely why I referred to their paid offering: travis is a vastly different beast when using the commercial infrastructure.
I also recommend not running the full matrix for every pull request, but just the stuff that frequently catches errors.
You were asking in a public forum. I didn’t ask you to rebut or debate my experiences with TravisCI. https://github.com/cmhamill their email is on their GitHub profile if you’d like to speak with them without anyone one else chiming in. I’m relating an objection that is tied to real time lost on my part and that of other maintainers. It is a persistent complaint of other people I work with in OSS. I’m glad TravisCI’s free offering exists but I am not under the illusion that the value they’re providing was brought into existence ex nihilo with zero value derived from OSS.
We use commercial TravisCI at work. It’s better than CircleCI or Travis’ public offering but still not close to running a CI service on a dedis (singular or plural).
I had to aggressively cache (multiple gigabytes) the build for Bloodhound before it stopped timing out. I’m glad their caching layer can tolerate something that fat but I wish it wasn’t necessary just to keep my builds working period.
That combined with how unresponsive TravisCI has been in general leaves a sour taste. If there was a better open source CI option than something like DroneCI I’d probably have rented a dedi for the projects I work on already.
You posted in a public forum and received some valid feedback based on the little context of your post ;)
How long does it take on your local machine as a point of comparison?
https://mail.haskell.org/pipermail/ghc-devs/2017-May/014200.html
That’s just build, doesn’t include test suite, but the tests are a couple more minutes.
Hm, that’s roughly the time your travis needs, too?
https://travis-ci.org/bitemyapp/bloodhound/jobs/286053181#L539 -> 120.87s seconds
Nope, the mailing list numbers do not include
--fastand that makes a huge difference.You are off your rocker if you think the EC2 machines Travis uses are going to get close to what my workstation can do.
Would you rather pay for a licensed software distribution that you drop in a fast dedicated computer you’ve bought and it turns that computer into a node in a CI cluster that can be used like Travis?
Would you rather pay for a service just like Travis but more expensive and running on latest-and-greatest CPUs and such?
If it actually worked well and I could test it before committing to a purchase, probably yes I would prefer that to losing control of my hardware or committing to a SAAS treadmill but businesses loooooooove recurring revenue and I can’t blame them.
That seems like a more likely stop-gap as nobody seems to want to sell software OTS anymore. Note: it’s not really just CPUs, it’s tenancy. I’d rather pay SAAS service premium + actual-cost-of-leasing-hardware and get fast builds than the “maybe pay us extra, maybe get faster builds” games that most CI services play. Tell me what hardware I’m actually running on and with what tenancy so I don’t waste my time.
Has anyone done this kind of dependency scan on Travis that this guy did on CircleCI? I suspect you will see much the same.
Travis does have one clear advantage here in that it’s OSS so you can SEE its dependencies and make your own decisions. See my note about CircleCI needing to be better about communication above.
Well… “scan”. They posted a screenshot of their network debugger tab :).
Travis (.org) uses Pusher, but not their tracking scripts. It integrates Google Analytics and as such, communicates with it. ga.js is loaded from google.
The page connects to:
All in all, it is considerably less messy then circle-ci’s frontend.
Also, Travis does not have your tokens or code in their web frontend, code is on Github, tokens should be encrypted using the encrypted environment: https://docs.travis-ci.com/user/environment-variables#Defining-encrypted-variables-in-.travis.yml
You have proven my point perfectly.
CircleCI’s only sin here is one of a lack of communication. There is nothing actually wrong with any of the callouts the article mentions, they just need to be VERY sure that their users are aware of exactly who is seeing the source code they upload. This should be an object lesson for anyone running a SaS company, ESPECIALLY if said SaS company caters to developers.
This is not an apples to apples comparison, in my post I cited Javascripts only (which can make AJAX requests and extract source code), @skade cites that Travis loads fonts, images, and CSS from third party domains, which don’t have those properties; a compromise in CSS might change the appearance of a page but generally can’t result in your source code/API tokens being leaked to a third party.
As far as I follow the only external Javascript run by Travis CI is Pusher. So, no, it has not proven your point perfectly, in fact it demonstrates the opposite.