Hi @adamshaylor, this product is open-source, licensed under Apache-2.0, and can be self-hosted for free with no strings attached. There is also a managed cloud-hosted instance that we (project maintainers) operate which is free for individual use. Does this address your concern?
Not really. It’s still commercial software, even if it is open source and self-hostable. But it’s not your business model I’m objecting to. Most content here is articles. In that context, the current title and tags are misleading. A post titled, “A better alternative to snapshot testing” tagged “practices” implies that this is an article. It’s not. It’s a link to a software product. Posts announcing software products should include the name of the product and be tagged “releases” rather than “practices”.
There’s a pretty strict no-spam policy here. Self-promotion in a first-time post is generally frowned upon. I personally have a higher tolerance for posts about self-promoting commercial software than some folks here. I just wasn’t expecting a software product when I clicked on your link. You can see spam posts being removed in the moderation log. I can’t speak for our moderator, but based on precedent, I’m guessing you’re in a bit of a gray area. As it’s titled and tagged currently, it’s leaning toward spam. Titling and tagging this correctly will reduce the chance of this article being flagged and culled as such.
I chose this title because this project introduces a different approach to testing that I am most interested to hear feedback about. But I am new here so I trust you on this. Could you help me find how to edit the tags or change the title? I don’t seem to find how to do so.
There’s a suggest link for users other than the OP. The logic around that link is a little vague to me, but I used it to suggest replacing the practices tag with releases. The title could be prepended with the name of the product (i.e. Touca: a better alternative to snapshot testing) to both preserve your intent and reduce confusion. I think there’s a UI to accept suggested changes. If you don’t see it, perhaps @pushcx could help.
Thank you for your friendly suggestions, @adamshaylor. It looks like editing is not available to the OP but I agree with your proposed changes. I am all for removing potential confusion.
I’ve been using snapshot testing in data engineering work for a while and it works well. I like to think of it as automating manual testing. You do a lot of manual testing to make code work the first time, and then the snapshot testing tool verifies that it continues to work the same.
There’s a versioning tie in. In the system I created, we used it to detect potentially breaking changes, so that downstream consumers can get involved appropriately before it’s released. @pejman are you planning on supporting this use case at all? It’s potentially quite powerful.
I’m glad to see something like this. It was a huge headache to build for myself, I’d rather use someone else’s implementation.
Hi @kellogh, thanks for sharing your experience! Yes, this use-case is already supported. Version management is one of the core features I designed the product around. The test results for each version are submitted and stored on a remote server that automatically compares them against a baseline version and visualizes the differences. This lets all team members:
See the differences between any two version and work together to decide whether they are intended.
Write comments and share notes about the test results.
Subscribe to any suite to receive an email with a summary of differences in each new version.
If the differences in a given version are intended, promote that version as the new baseline.
Is this in-line with what you had in mind? Also, Is the system you built still around and public, by any chance? Would love to learn more about and get inspirations from it.
Hi everyone, My first post here. Nice to meet you all and thanks to @ocramz for inviting me.
This project has been in the making for a long time. It includes tooling and infrastructure to help developers write high-level tests for complex software workflows that are not easy to unit test. I wanted to take ideas from visual regression testing, snapshot testing, and property-based testing and build a general-purpose regression testing system that developers can use to find the unintended side-effects of their day-to-day code changes during the development stage. I wrote the first line of code in 2018 and left my job 2 years ago to work on it full-time (i.e. all the time). I am posting it here because I want to hear your unvarnished thoughts and feedback about its latest v2.0 release, a milestone version that hopes to be useful to small and large teams alike. This version comes with:
An easy to self-host server that stores test results for new versions of your software workflows, automatically compares them against a previous baseline version, and reports any differences in behavior or performance.
A CLI that enables snapshot testing without using snapshot files. It lets you capture the actual output of your software and remotely compare it against a previous version without having to write code or to locally store the previous output.
4 SDKs in Python, C++, Java, JavaScript that let you write high-level tests to capture values of variables and runtime of functions for different test cases and submit them to the Touca server.
Test runner and GitHub action plugins that help you continuously run your tests as part of the CI and find breaking changes before merging PRs.
I would really appreciate your honest feedback, positive or negative, about Touca. Do you find this useful? Would love to hear your thoughts and happy to answer any questions.
I haven’t really felt the pain of making snapshot tests scale (currently working with only about a 100 snapshots), and the Website / Docs don’t really make it clear to me what problems Touca solves that I might face in the future.
I’m definitely interested in a UI for visual regression testing as I’ve struggled to find good tooling for that, but looking at the screenshots it doesn’t seem like a big focus for Touca.
Thanks for sharing your thoughts. I’m sorry that you didn’t find the docs clear enough. I’ve tried to briefly explain the differences between Touca and snapshot testing here: https://touca.io/docs/guides/vs-snapshot/. As outlined in that document, one primary difference is that Touca removes the need for storing snapshot files in version control and moves the storage, comparison, visualization, and reporting of test results to a remote server. In this new model, instead of git committing snapshot files with differences, you’d use the server to promote a new version as baseline.
You are right that visual regression testing of web UI is not a focus of this project. I believe there are many good solutions in the market for web apps. We focus on software that may not have a web interface, like API endpoints, data pipelines, machine learning algorithms, command-line tools. We want to make it easy to test these types of software with various inputs without writing extensive unit tests and integration tests.
I found that page and got the difference, but I really like version control and its benefits. Focusing a little bit on the benefits a centralized server brings in comparison to the usual approach would be the interesting part for me :)
When visiting https://touca.io/ as someone who already understands snapshot testing, it was difficult to find how Touca was different from normal snapshot testing. The only obvious difference was that it has a web app interface instead of my local Git diff interface, but that on its own doesn’t sound like a desirable feature for an app that already has a CI pipeline. I think your pitch about “removes the need for storing snapshot files in version control” should be more visible. I still have no need for the feature – I struggle to imagine a case where snapshot files in nested folders would not be easy enough to manage – but at least that information would have made it clearer that the software is targeting problems I don’t have, so I don’t need to read more of the site.
On the home page, one heading is relevant to that question, “Snapshot testing without snapshot files”. However, the rest of that block doesn’t elaborate on how snapshot testing could possibly work without snapshot files or clarify whether “snapshot files” are the generated snapshots or source code files with snapshot tests. The next sentence, “Remotely compare the your software output against a previous baseline version”, sounds like it could equally apply to traditional snapshot testing. I think the word “remotely” in that sentence was also meant to imply “without snapshot files”, but I just interpreted it as “diff snapshot files on our remote server instead of locally”. (Also, there’s a typo: “the your” should be “the”.) The final part of that block, “brew install touca”, is not at all connected to the previous two sentences and seems out of place. Without reorganizing the whole page, that brew command might seem more relevant if a “Try it:” label were before it.
After I first was confused by the home page, when I clicked Docs, I just saw another summary of snapshot testing in general followed by a bunch of links, none of which sounded like they would explain how Touca is different.
I saw in the sidebar that I was in the Getting Started section. When I skimmed the next page, Concepts, it looked like just another explanation of snapshot tests in general.
Okay, so I went to the next page, Writing Tests. Great, examples of how to actually use this. Except… after reading the whole page it was hard to understand the expected input and output of each test. There is “alice”, “bob”, and “charlie”, but is the input literally those strings? How can that be when the assertions mention student.gpa? And where is the expected output saved – why wasn’t it mentioned on this page? If it’s not saved to a file, it must be saved to a server, but I had trouble imagining at this point why that would be better. At that point I gave up on understanding Touca. Only later did I come back to these comments and see your link.
I think the Writing Tests page is too abstract. The JavaScript code sample calls code_under_test.find_student(username), but that’s not defined anywhere, so I struggled to imagine how it’s connected to the rest of the code. Maybe include a short stub definition in your examples, like this?
async function find_student(username) { // the code under test
if (username === "alice") {
return {name: "Alice", gpa: 4.0}
} else if (username === "bob") {
// etc.
}
}
And the next line, touca.check("gpa", student.gpa), didn’t give any hint as to what expected value Touca would compare the GPA against. Maybe add a comment to that line: // compare student.gpa to Touca’s saved 'gpa' value in the scope of this testcase. That comment may be inaccurate; you can correct it.
Hi @roryokane, Thank you so much for this thorough feedback. I really appreciate you taking the time. I am going to read it again and again tomorrow and apply your suggestions both to the landing page and to the docs.
I think your pitch about “removes the need for storing snapshot files in version control” should be more visible.
Agreed. Will try to clarify.
I struggle to imagine a case where snapshot files in nested folders would not be easy enough to manage
You mentioned you are familiar with snapshot testing. Could you share more about your typical use-case?
I think the Writing Tests page is too abstract.
Based on your suggestion about this page and the “Concepts” page, I’m tempted to just rewrite the entire “Getting Started” section.
This is exactly the type of feedback that I was looking for. Thank you!
This sounds like an interesting project, but I expected based on the title and tags to find an article rather than a link to commercial software.
Hi @adamshaylor, this product is open-source, licensed under Apache-2.0, and can be self-hosted for free with no strings attached. There is also a managed cloud-hosted instance that we (project maintainers) operate which is free for individual use. Does this address your concern?
Not really. It’s still commercial software, even if it is open source and self-hostable. But it’s not your business model I’m objecting to. Most content here is articles. In that context, the current title and tags are misleading. A post titled, “A better alternative to snapshot testing” tagged “practices” implies that this is an article. It’s not. It’s a link to a software product. Posts announcing software products should include the name of the product and be tagged “releases” rather than “practices”.
There’s a pretty strict no-spam policy here. Self-promotion in a first-time post is generally frowned upon. I personally have a higher tolerance for posts about self-promoting commercial software than some folks here. I just wasn’t expecting a software product when I clicked on your link. You can see spam posts being removed in the moderation log. I can’t speak for our moderator, but based on precedent, I’m guessing you’re in a bit of a gray area. As it’s titled and tagged currently, it’s leaning toward spam. Titling and tagging this correctly will reduce the chance of this article being flagged and culled as such.
I chose this title because this project introduces a different approach to testing that I am most interested to hear feedback about. But I am new here so I trust you on this. Could you help me find how to edit the tags or change the title? I don’t seem to find how to do so.
There’s a suggest link for users other than the OP. The logic around that link is a little vague to me, but I used it to suggest replacing the practices tag with releases. The title could be prepended with the name of the product (i.e. Touca: a better alternative to snapshot testing) to both preserve your intent and reduce confusion. I think there’s a UI to accept suggested changes. If you don’t see it, perhaps @pushcx could help.
Thank you for your friendly suggestions, @adamshaylor. It looks like editing is not available to the OP but I agree with your proposed changes. I am all for removing potential confusion.
I’ve been using snapshot testing in data engineering work for a while and it works well. I like to think of it as automating manual testing. You do a lot of manual testing to make code work the first time, and then the snapshot testing tool verifies that it continues to work the same.
There’s a versioning tie in. In the system I created, we used it to detect potentially breaking changes, so that downstream consumers can get involved appropriately before it’s released. @pejman are you planning on supporting this use case at all? It’s potentially quite powerful.
I’m glad to see something like this. It was a huge headache to build for myself, I’d rather use someone else’s implementation.
Hi @kellogh, thanks for sharing your experience! Yes, this use-case is already supported. Version management is one of the core features I designed the product around. The test results for each version are submitted and stored on a remote server that automatically compares them against a baseline version and visualizes the differences. This lets all team members:
Is this in-line with what you had in mind? Also, Is the system you built still around and public, by any chance? Would love to learn more about and get inspirations from it.
Hi everyone, My first post here. Nice to meet you all and thanks to @ocramz for inviting me.
This project has been in the making for a long time. It includes tooling and infrastructure to help developers write high-level tests for complex software workflows that are not easy to unit test. I wanted to take ideas from visual regression testing, snapshot testing, and property-based testing and build a general-purpose regression testing system that developers can use to find the unintended side-effects of their day-to-day code changes during the development stage. I wrote the first line of code in 2018 and left my job 2 years ago to work on it full-time (i.e. all the time). I am posting it here because I want to hear your unvarnished thoughts and feedback about its latest v2.0 release, a milestone version that hopes to be useful to small and large teams alike. This version comes with:
An easy to self-host server that stores test results for new versions of your software workflows, automatically compares them against a previous baseline version, and reports any differences in behavior or performance.
A CLI that enables snapshot testing without using snapshot files. It lets you capture the actual output of your software and remotely compare it against a previous version without having to write code or to locally store the previous output.
4 SDKs in Python, C++, Java, JavaScript that let you write high-level tests to capture values of variables and runtime of functions for different test cases and submit them to the Touca server.
Test runner and GitHub action plugins that help you continuously run your tests as part of the CI and find breaking changes before merging PRs.
I would really appreciate your honest feedback, positive or negative, about Touca. Do you find this useful? Would love to hear your thoughts and happy to answer any questions.
Congrats on the release!
I haven’t really felt the pain of making snapshot tests scale (currently working with only about a 100 snapshots), and the Website / Docs don’t really make it clear to me what problems Touca solves that I might face in the future.
I’m definitely interested in a UI for visual regression testing as I’ve struggled to find good tooling for that, but looking at the screenshots it doesn’t seem like a big focus for Touca.
Thanks for sharing your thoughts. I’m sorry that you didn’t find the docs clear enough. I’ve tried to briefly explain the differences between Touca and snapshot testing here: https://touca.io/docs/guides/vs-snapshot/. As outlined in that document, one primary difference is that Touca removes the need for storing snapshot files in version control and moves the storage, comparison, visualization, and reporting of test results to a remote server. In this new model, instead of git committing snapshot files with differences, you’d use the server to promote a new version as baseline.
You are right that visual regression testing of web UI is not a focus of this project. I believe there are many good solutions in the market for web apps. We focus on software that may not have a web interface, like API endpoints, data pipelines, machine learning algorithms, command-line tools. We want to make it easy to test these types of software with various inputs without writing extensive unit tests and integration tests.
I found that page and got the difference, but I really like version control and its benefits. Focusing a little bit on the benefits a centralized server brings in comparison to the usual approach would be the interesting part for me :)
When visiting https://touca.io/ as someone who already understands snapshot testing, it was difficult to find how Touca was different from normal snapshot testing. The only obvious difference was that it has a web app interface instead of my local Git diff interface, but that on its own doesn’t sound like a desirable feature for an app that already has a CI pipeline. I think your pitch about “removes the need for storing snapshot files in version control” should be more visible. I still have no need for the feature – I struggle to imagine a case where snapshot files in nested folders would not be easy enough to manage – but at least that information would have made it clearer that the software is targeting problems I don’t have, so I don’t need to read more of the site.
On the home page, one heading is relevant to that question, “Snapshot testing without snapshot files”. However, the rest of that block doesn’t elaborate on how snapshot testing could possibly work without snapshot files or clarify whether “snapshot files” are the generated snapshots or source code files with snapshot tests. The next sentence, “Remotely compare the your software output against a previous baseline version”, sounds like it could equally apply to traditional snapshot testing. I think the word “remotely” in that sentence was also meant to imply “without snapshot files”, but I just interpreted it as “diff snapshot files on our remote server instead of locally”. (Also, there’s a typo: “the your” should be “the”.) The final part of that block, “
brew install touca”, is not at all connected to the previous two sentences and seems out of place. Without reorganizing the whole page, thatbrewcommand might seem more relevant if a “Try it:” label were before it.After I first was confused by the home page, when I clicked Docs, I just saw another summary of snapshot testing in general followed by a bunch of links, none of which sounded like they would explain how Touca is different.
I saw in the sidebar that I was in the Getting Started section. When I skimmed the next page, Concepts, it looked like just another explanation of snapshot tests in general.
Okay, so I went to the next page, Writing Tests. Great, examples of how to actually use this. Except… after reading the whole page it was hard to understand the expected input and output of each test. There is “alice”, “bob”, and “charlie”, but is the input literally those strings? How can that be when the assertions mention
student.gpa? And where is the expected output saved – why wasn’t it mentioned on this page? If it’s not saved to a file, it must be saved to a server, but I had trouble imagining at this point why that would be better. At that point I gave up on understanding Touca. Only later did I come back to these comments and see your link.I think the Writing Tests page is too abstract. The JavaScript code sample calls
code_under_test.find_student(username), but that’s not defined anywhere, so I struggled to imagine how it’s connected to the rest of the code. Maybe include a short stub definition in your examples, like this?And the next line,
touca.check("gpa", student.gpa), didn’t give any hint as to what expected value Touca would compare the GPA against. Maybe add a comment to that line:// compare student.gpa to Touca’s saved 'gpa' value in the scope of this testcase. That comment may be inaccurate; you can correct it.Hi @roryokane, Thank you so much for this thorough feedback. I really appreciate you taking the time. I am going to read it again and again tomorrow and apply your suggestions both to the landing page and to the docs.
Agreed. Will try to clarify.
You mentioned you are familiar with snapshot testing. Could you share more about your typical use-case?
Based on your suggestion about this page and the “Concepts” page, I’m tempted to just rewrite the entire “Getting Started” section. This is exactly the type of feedback that I was looking for. Thank you!