It was also interesting to see the reaction from open source developers to unsolicited pull requests from what looks like a bot (really a bot found the problem and made the PR, but really a human developer at Code Review Doctor did triage the issue before the PR was raised). After creating 69 pull requests the reaction ranged from:
I wonder if you’d get better reactions if a human made the PR and didn’t say it came from a bot.
That’s exactly the quote that prompted me to share the article. I think there was recently a case in the linux kernel community that some university group was submitting (arguably bad) patches that were generated by a tool – it didn’t went well if I recall correctly. Maybe initial reactions would be better, but long term, if the project finds out, it would lead to loss of trust.
A flipside of that is that you might expect better analysis from a human if it had been filed under a human’s name. These are clearly mostly auto-generated bug reports, and a number of false positives were filed, despite the triaging (from just spot-checking: 1, 2). So filing them under a bot’s name is maybe more honest to manage expectations.
This PR got closed as invalid. It looks like one of the two fixes involved a format string which was passed as a function argument. It would be good if each fix was a separate PR, or some human grouped fixes into PRs based on the area of the code.
On a related note (I can’t post another comment): Tried running this on a repo I’m working on, and it returned a blank page (I tried reloading). I guess that means there were no issues, but it would be good to get an indication of that.
“good bot” so funny
I wonder if you’d get better reactions if a human made the PR and didn’t say it came from a bot.
That’s exactly the quote that prompted me to share the article. I think there was recently a case in the linux kernel community that some university group was submitting (arguably bad) patches that were generated by a tool – it didn’t went well if I recall correctly. Maybe initial reactions would be better, but long term, if the project finds out, it would lead to loss of trust.
It was the University of Minnesota and they got their entire university banned from submitting anything to the Linux kernel.
The biggest argument against stuff like this I saw was that the heads of the groups being tested against had not accepted to participate in the study.
A flipside of that is that you might expect better analysis from a human if it had been filed under a human’s name. These are clearly mostly auto-generated bug reports, and a number of false positives were filed, despite the triaging (from just spot-checking: 1, 2). So filing them under a bot’s name is maybe more honest to manage expectations.
This PR got closed as invalid. It looks like one of the two fixes involved a format string which was passed as a function argument. It would be good if each fix was a separate PR, or some human grouped fixes into PRs based on the area of the code.
On a related note (I can’t post another comment): Tried running this on a repo I’m working on, and it returned a blank page (I tried reloading). I guess that means there were no issues, but it would be good to get an indication of that.