I think more programmers should be using secret scanners but there weren’t any “no-brainer” solutions I could find, so I decided to build a new one. The core of secret scanning is running regex against a large number of files, and it turns out this is something ripgrep is excellent at. By leveraging the ripgrep library effectively secrets is able to scan files roughly 100x faster than other solutions I tested. This is my first Rust project and I was impressed with how quickly I was able to put something together that is also really fast. Let me know if you have any feedback!
I appreciate that you put links to other similar projects in the README! It’s a small thing but really helps to encourage adoption of the idea, even if the implementation doesn’t meet specific requirements. That being said, this tool looks good for my use case and I’m definitely going to try it.
I like the secretsignore feature. Sometimes you want things that look like secrets in your tests, and not being able to accommodate that has made me avoid similar tools in the past.
There’s also the git-secrets project (from AWS, first released in 2015) that’s also designed as a pre-commit hook.
(I used to work for AWS and used git-secrets, but never worked on git-secrets.)
A while back I accidentally committed (a not particularly important) secret to a personal project and had to clean it up manually, so I will definitely be checking this out.
One suggestion from a quick look at the code: I’d probably try to avoid handling paths as Strings, instead favouring OsString/OsStr and creating Paths/PathBuf from them. File systems often don’t enforce UTF-8, which String requires but OsString does not.
Seconded; I once had a corrupted hard drive that turned a bunch of directory names into garbage and it was hard as hell to figure out how to actually get a tool to delete them.
That’s good to know, thanks for the tip!
Is it really that hard to not commit secrets? I mean I just don’t ever put them in with code… Never use git add .
I guess it just seems like a heavy handed solution to a problem that’s barely there. Unless this happens a ton elsewhere. I am astounded when I hear the statistics about how many secrets are committed in GitHub, but I wonder if it has more to do with a lack of understanding than just a git flub/accident.
git add .
I think it’s a coding practices thing. Like Nathaniel Borenstein said, “No ethically-trained software engineer would ever consent to write a DestroyBaghdad procedure. Basic professional ethics would instead require him to write a DestroyCity procedure, to which Baghdad could be given as a parameter.” I never hard code secrets because secrets always come in as environmental variables or other parameters, but I think for people who just want to get something done quickly, hard coding seems like the fastest way to do things.
I think in a perfect world you’re right, however a lot of exploits that cause users data to be exposed are caused in part by people checking in secrets into source control. My goal with this project was to create something as lightweight and quiet as possible such that you can mostly forget that it’s installed and get the protection with very little downside. Also if you work at a company on a web service the security team might mandate using a security scanner as part of your pre-commit, in which case it’s nice to have a very fast and lightweight option.
i like the concept of having a company where the security team might mandate a tool like this
the point i was making is that if someone is committing secrets, they probably don’t realize what they’re doing, and in those cases, they probably won’t understand the need to add this tool to their git pre-commit hooks
in other words, if they were to fully understand what this tool is used for, then the usefulness is probably greatly reduced
which leads me to feel like ultimately the education is what is important… but the security team mandate thing is good, having this run in CI before in a main branch is good too, depending on your repo setup this may make overwriting the history in the repo easier
but best to have it checked before ever committed, which is the point of the tool
It’s easy to not commit secrets! That’s why we should make a machine automate it for us.
I’m also not sure what’s hard about it.
Never use git add .
Never use git add .
Don’t even need to avoid any commands - instead, just don’t have secrets in the repository directory at all. It’s easy and also completely foolproof.
Which breaks down for infrastructure repos that are mostly secrets :P
i work every day in infrastructure repos and don’t have any secrets committed, so that just sounds like someone’s doing something wrong
You can try to argue with reality or just accept it. There are setups that are a lot older than e.g. Vault has been around, or anything else based on tokens.
I don’t think this is the place for judgment of practices, and I’m not even involved in this game anymore, so don’t read this as defending myself.
oh, not arguing at all, it just seemed like your statement was a “this won’t work for infrastructure repos”… which was, at least I think, the only logical conclusion to take from your statement, because i think it’s pretty obvious this isn’t going to work for repos that are literally designed to store secrets
then again, maybe the :P invalidates any attempt at a logical conclusion
Never use git add . or git commit -a is good advice for other reasons though.