-
arikfr
8 months ago
|
link
| on:
Continuous integration services for open-source projects not requiring config in root of repository?
You can try SemaphoreCI – they have a free plan and don’t require a configuration file.
-
Indeed, SemaphoreCI seems to support customizing build commands w/o putting any stuff in the repo, they even recognize possible need to add custom configuration files that shouldn’t be part of the repo.
Their C/C++ toolchain is dated, though. GCC 4.9 is the latest one there.
Thanks!
-
It is also the most beautifully designed from what I’ve seen <3
-
SemaphoreCI is the first service, where you can choose how you to connect to GitHub:
- Only Public repos,
- All repos, Public & Private.
I don’t have private repos on GitHub, but I value greatly that kind of choice. I’m assuming the first option below.
GitHub
Semaphore by renderedtext wants to access your account
Repositories
Public onlyThis application will be able to read and write all public repository data. This includes the following:
- Code
- Issues
- Pull requests
- Wikis
- Settings
- Webhooks and services
- Deploy keys
BitBucket
Semaphore is requesting access to the following:
- Read your account information
- Read and modify your repositories and their pull requests
- Administer your repositories
- Read and modify your repositories’ webhooks
BB permissions aren’t as bloated as in case of Codeship, and are generally more slim than those in CircleCI, but there is one important exception, SemaphoreCI wants to be able to modify pull requests, while CircleCI is fine with only reading them.
-
-
Dramatiq is licensed under the AGPL
Now I have three options:
- Make the codebase at work opensource (lol)
- Violate AGPL on purpose
- Use any of the other Redis-based task queues
I really don’t get what the author is trying to achieve with choosing a license like this.
-
You could also buy a license.
-
-
-
You are right of course. Other message frameworks like sidekiq seem to do alright: https://github.com/mperham/sidekiq
The challenge here is that Celery is in pretty great shape for a free solution. On the other hand Python’s support for high concurrency is changing rapidly so who knows maybe there’s room for a new player in this market.
-
-
It works like a charm with RabbitMQ as a backend. The rest is pretty experimental and breaks, especially at high volume. (I’ve been using Celery for >5 years)
-
I’ve been using Celery professionally for about 3 years and dramatiq tries to solve many of the issues I’ve encountered using it. Some stuff that immediately springs to mind:
- Celery doesn’t support task prioritization. You have to deploy multiple sets of workers in order to prioritize queues.
- Celery has poor support for delayed tasks. Delayed tasks go on the same queue that normal tasks go on and they’re simply pulled into worker memory until they can be executed. This makes it hard to autoscale workers by queue size.
- Celery acks tasks as soon as they’re pulled by a worker by default. This is easy to change, but a bad default. Dramatiq doesn’t let you change this: tasks are only ever acked when they’re done processing.
- Celery tasks are not retried on error by default.
- Celery’s not well suited for integration testing. You’re expected to unit test tasks and to turn eager evaluation on for integration tests, but even then task exceptions will be swallowed by default. Dramatiq provides an in-memory stub broker specifically for this use case.
- The source code is spread across 3 different projects (celery, billiard and kombu) and it’s impenetrable. Its usage of runtime stack frame manipulation leads to heisenbugs.
- It’s easy for some of its more advanced “canvas” features to drop tasks.
All of the above are things that are first-class in dramatiq and there are definitely other things I’m not thinking of right now. That’s not to say that celery is bad, but I think we can do better and that’s why I made dramatiq. :D
-
-
I don’t have much experience with RQ since it is Redis-only and I’ve generally preferred to use RabbitMQ as a message broker. However, a few things that seem like disadvantages to me with RQ are:
- Messages are pickled so it’s strictly limited to Python and pickled messages are potentially exploitable. This also means you may sometimes send bigger messages than you intended over the network purely by accident.
- Queue prioritisation is handled like it is in Celery: you have to spawn different sets of workers.
- It forks for every job, so it’s slightly slower and forks that are killed b/c they’ve surpassed their time limits can leak DB connections if you’re not careful. I understand this may be swappable behaviour, however.
- Similar to Celery, there isn’t a good integration testing story for RQ.
Because I’ve criticised both Celery and RQ at this point, I feel it’s important that I mention a couple areas where they’re both currently better than dramatiq:
- the obvious one: it’s newer than either of those and is less likely to be familiar to users. The extension ecosystem for dramatiq is nonexistent (though I will be releasing integration packages for Django and Flask soon!)
- dramatiq doesn’t store task results and doesn’t offer a way to retrieve them. Adding that sort of functionality is trivial using middleware, but it’s not there ootb so if you absolutely need something like that and you don’t care about the things I have mentioned so far then you should look at Celery or RQ instead.
-
Thank you for taking the time to post this!
There are two other areas that bother me personally:
- Python 3 only. While I would love to switch to Python 3, still need to maintain a large project in Python 2.
- The AGPL license. The above project is open source too, but I want to keep it BSD licensed to stay “friendly” towards potential users. Ironically, for a commercial project I would worry less about your license of choice, as I wouldn’t mind buying the commercial license when needed.
I share @jscn’s sentiment about Celery. I I was wondering if RQ, despite the above disadvantages might be more stable. At least their codebase should easier to grok (single repo)…
-
Python 3 only. While I would love to switch to Python 3, still need to maintain a large project in Python 2.
I’m considering adding Python 2 support, but it’s a hard thing to balance what with 2.x getting EOL’d in a little less than 2 and a half years.
The AGPL license. The above project is open source too, but I want to keep it BSD licensed to stay “friendly” towards potential users. Ironically, for a commercial project I would worry less about your license of choice, as I wouldn’t mind buying the commercial license when needed.
Understandable.
-
-
Sure, that’s true. Did you ever look at https://github.com/RichardKnop/machinery that project is still really early. Probably much easier to compete with.
-
-
-
-
-
So “use any other queue thing” will definitely be a very popular option :)
That’s perfectly fine! I just want those people that get value out of my work to contribute back in some way. If someone makes a cost-benefit analysis and decides that they’d rather use celery over dramatiq because they prefer the cheaper option (although it’s worth mentioning that I give out free comm. licenses for one year per company) then that’s their prerogative. I’ll still be around a year later when they realise their mistake ;).
-
-
-
Trying to achieve you not using this at work? That’s usually what I’m going for when I choose AGPL
https://redash.io/ Redash is a very similar feature set (if I understand correctly) and is open source, if anyone wants something hostable / ownable.
(disclaimer: Redash author)
Thank you for mentioning Redash. Actually Redash’s features extend beyond QueryClips, as it allows to visualize the data, create dashboards, alerts and more.
@dpaola2 on the website you mentioned that you created QueryClips after being frustrated with the lack of simple, easy-to-use tools. I created Redash for the same reasons ;-) I just wonder, did you have the chance to evaluate Redash before creating QueryClips? :-)