“Not only that, any code examples in rustdoc are treated like tests and get executed during compilation!”
This is brilliant. First time I’ve heard of it. ZeD on Hacker News said Python can do it, too, with seangransee offering this example.
Aside from a good idea, it looks like it should also be a new requirement for high-assurance systems where all examples of system usage are executable. It’s often implicit where you’re already supposed to have both tests of all functions and consistency with documentation. At the least, it could be an extra check on these.
I’m over the moon about doctests. Elixir has them too, that’s where I saw the light. In the past, I’ve gone to annoying lengths to ensure that code examples in documentation don’t go stale; it’s a great feeling to have first-class support for that workflow.
I’ve found that in Elixir having both doctests and normal tests is quite handy–one as a quick sanity check and a way of demonstrating an API, and the other as a full way of confirming behavior. The use of tags to turn on and off different sets of tests is also not well supported (to my knowledge) with doctests.
AFAIK turning Elixir doctests on and off is not supported. That bothers me just a bit, because there are times when I’d like to show some non-testable code (e.g., demonstrating a request to an external network service) in the exact same syntax that my doctests use (iex>, etc).
I think the first place I saw that was R, where it’s used pervasively in the package repository (CRAN). In fact the output of example REPL sessions is generated by running the code, so besides being used as tests that are flagged if they fail to run entirely, it also keeps the examples up to date with minor changes to output format, etc., which can otherwise get out of date even when the code continues to work.
I’ve found that in practice I write far fewer testable Go examples than I do in Rust code. In Rust, I just drop into Markdown, write the code, and I’m done. It’s incredibly low friction.
D achieves the same the other way round. Instead of treating documentation like code, D has unit test code blocks and can treat them as documentation. This makes it easier for the tooling, since syntax highlighting etc applies to the unit tests as well.
This is not a really critical difference, but if you design a new language, I would suggest the D approach as slightly better.
I’ve generated documentation from tests before (in Ruby’s rspec). Just thinking about it, isn’t D’s approach prone to the same weakness as the comments that they use to generate the docs? That is, when you use comments or non-executable code (in my case it was string descriptors) to generate docs, they can get out of date with the actual code, whereas if you write tests inside documentation, the docs may grow stale but the tests will break.
This also got another round of counters on how difficult the borrow-checker is vs C with one person referencing the post on linked lists. I keep trying to put that apples to oranges comparison into perspective by showing what it would’ve taken to do the same thing in C with separation logic. The borrow checker is so much easier by comparison. Here’s my latest take on that reposted from Hacker News in case anyone needs something to fire back at those false comparisons:
“Wait until you see what it took to do that in C with the formally-checked, safety guarantees of Rust around the time Rust was being developed:
Ok, so author needed to know C extremely well (i.e. “language lawyer”), comfortable with separation logic, able to turn informal requirements into specifications, use of ghost code, and have a SMT solver. The solvers often fail on some stuff that requires formal proof by rare experts. However, that expert was able to automatically verify a singly-linked list with just 113 ghost statements for the 90 statements in C. That’s more specs than code! Industrial case studies in other article give more examples. More importantly, when making the C code as verifiably safe as Rust, the effort required had developers cranking it out at rate of two lines of code per hour.
Both manual and automated proof with separation logic has a lot more automation since then. Yet, I think programmers will more easily learn to intuitively and heuristically structure their programs to pass Rust’s automated, formal checker than learning expert-level C, separation logic, and/or formal proof. And Rust can still do reference counting or unsafe operation in event it can’t check something.”
“Not only that, any code examples in rustdoc are treated like tests and get executed during compilation!”
This is brilliant. First time I’ve heard of it. ZeD on Hacker News said Python can do it, too, with seangransee offering this example.
Aside from a good idea, it looks like it should also be a new requirement for high-assurance systems where all examples of system usage are executable. It’s often implicit where you’re already supposed to have both tests of all functions and consistency with documentation. At the least, it could be an extra check on these.
I’m over the moon about doctests. Elixir has them too, that’s where I saw the light. In the past, I’ve gone to annoying lengths to ensure that code examples in documentation don’t go stale; it’s a great feeling to have first-class support for that workflow.
I’ve found that in Elixir having both doctests and normal tests is quite handy–one as a quick sanity check and a way of demonstrating an API, and the other as a full way of confirming behavior. The use of tags to turn on and off different sets of tests is also not well supported (to my knowledge) with doctests.
AFAIK turning Elixir doctests on and off is not supported. That bothers me just a bit, because there are times when I’d like to show some non-testable code (e.g., demonstrating a request to an external network service) in the exact same syntax that my doctests use (
iex>, etc).I think the first place I saw that was R, where it’s used pervasively in the package repository (CRAN). In fact the output of example REPL sessions is generated by running the code, so besides being used as tests that are flagged if they fail to run entirely, it also keeps the examples up to date with minor changes to output format, etc., which can otherwise get out of date even when the code continues to work.
Go has this too: https://blog.golang.org/examples
I’ve found that in practice I write far fewer testable Go examples than I do in Rust code. In Rust, I just drop into Markdown, write the code, and I’m done. It’s incredibly low friction.
D achieves the same the other way round. Instead of treating documentation like code, D has unit test code blocks and can treat them as documentation. This makes it easier for the tooling, since syntax highlighting etc applies to the unit tests as well.
This is not a really critical difference, but if you design a new language, I would suggest the D approach as slightly better.
I’ve generated documentation from tests before (in Ruby’s rspec). Just thinking about it, isn’t D’s approach prone to the same weakness as the comments that they use to generate the docs? That is, when you use comments or non-executable code (in my case it was string descriptors) to generate docs, they can get out of date with the actual code, whereas if you write tests inside documentation, the docs may grow stale but the tests will break.
This also got another round of counters on how difficult the borrow-checker is vs C with one person referencing the post on linked lists. I keep trying to put that apples to oranges comparison into perspective by showing what it would’ve taken to do the same thing in C with separation logic. The borrow checker is so much easier by comparison. Here’s my latest take on that reposted from Hacker News in case anyone needs something to fire back at those false comparisons:
“Wait until you see what it took to do that in C with the formally-checked, safety guarantees of Rust around the time Rust was being developed:
The VeriFast Program Verifier
Software Verification with VeriFast: Industrial, Case Studies
Ok, so author needed to know C extremely well (i.e. “language lawyer”), comfortable with separation logic, able to turn informal requirements into specifications, use of ghost code, and have a SMT solver. The solvers often fail on some stuff that requires formal proof by rare experts. However, that expert was able to automatically verify a singly-linked list with just 113 ghost statements for the 90 statements in C. That’s more specs than code! Industrial case studies in other article give more examples. More importantly, when making the C code as verifiably safe as Rust, the effort required had developers cranking it out at rate of two lines of code per hour.
Both manual and automated proof with separation logic has a lot more automation since then. Yet, I think programmers will more easily learn to intuitively and heuristically structure their programs to pass Rust’s automated, formal checker than learning expert-level C, separation logic, and/or formal proof. And Rust can still do reference counting or unsafe operation in event it can’t check something.”