Thanks! Some time ago I wrote an arrticle about git notes too - https://github.com/ligurio/testres/wiki/Using-GIT-as-a-storage. It describes using notes as a storage fir testing results. Moreover, I’ve sent a patch for cgit that shows these testing results in web ui.
The Sail project generated formal specifications of PowerPC by parsing the pseudocode in the PDF architecture references. I think that they found that the x86 docs at the time were too limited (inconsistent and incomplete) to do the same. I’m curious how well this project manages it. Arm eventually moved to auto-generating their pseudocode from their internal Architecture Specification Language (ASL) and there’s a tool that will transform ASL to Sail. The RISC-V official spec now uses Sail as well.
Seems you are talking about this project, - https://github.com/rems-project/sail.
Haven’t heard about it. I didn’t get - is possible to translate assembler instructions to SMT representation (smt-lib2) using Sail?
I didn’t get - is possible to translate assembler instructions to SMT representation (smt-lib2) using Sail?
It can prove SMT properties inline and can export to a variety of theorem provers. We used this to prove monotonicity properties about CHERI capabilities, for example.
ccache is not a single way to speedup building a project. There are many other possible ways to speedup both building and testing. Most of them we implemented for Tarantool CI and reduced a time required for a feedback.
If someone interested, I’ve prepared a list of such improvements in CI.
Does anyone here use tarantool? Is there a smooth migration path from Postgres?
I ask because I’m currently implementing cubejs as a database cache/acceleration layer. The performance is decent BUT it’s a solution to poll the database for updates and rebuild the cache rather than a write-through solution.
To be honest, Tarantool is not a drop-in replacement for PostgreSQL. Typical Tarantool usecases are cache, queue, sophisticated business logic on top of DB engine. Regarding Postgres - there is a connector to Postgres.
I would propose to take a look on CTest. It is a part of CMake and allows running any tests. CTest is a test framework-agnostic tool. CTest allows running tests in parallel, rerun tests failed last time, filter tests by label etc.
If our history contains many transactions, knossos will be unable to verify linearizability.
It is worth to say that Elle is much better than Knossos, it allows checking longer histories.
By the way, it is not mandatory to use Jepsen for writing consistency tests. It is possible to make tests in any other programming language, produce a history in Jepsen-compatible and check it with elle-cli, a command-line frontend to transactional consistency checkers for black-box databases.
One thing the author seemingly hasn’t tried is using a manufacturer-independent gateway for Zigbee like https://www.zigbee2mqtt.io. This approach uses a USB-to-Zigbee adapter and makes all Zigbee devices from all vendors talk to each other. Run on a Raspberry Pi together with HomeAssistant (general-purpose home automation) or Homebridge (to bridge to Apple HomeKit), this setup provides a great user experience (for me).
It also opens two common and documented APIs for interfacing with the devices: HomeAssistant (high-level) and MQTT (lower-level, still easy).
I personally run Zigbee2mqtt with HomeBridge and control everything from the Apple ecosystem. It works flawlessly for me and my family. From the nerd-side, I stream all MQTT messages as JSON to a Postgres database and use Grafana to plot various metrics from sensors and devices.
Personally, I don’t like zigbee2mqtt. However, it is the best solution for vendor-independent gateway.
I’m looking on zigbee-lua that uses Lua instead of Javascript and other alternatives.
I agree - the amount of code and complexity of the Javascript in zigbee2mqtt is staggering. Thanks for pointing me to zigbee-lua - looks great.
In an ideal world, there would be a standardised mqtt protocol definition with zigbee2mqtt and others implementing that protocol. From a short look it’s not entirely clear if zigbee-lua tries to be wire-compatible with zigbee2mqtt.
As far as I get it right, the main complexity of zigbee2mqtt are quirks for different Zigbee devices that doesn’t conform to Zigbee specification. For interaction with Zigbee devices, z2m uses zigbee-herdsman-converters to parse messages to and from devices. Adding support of a new Zigbee device to z2m is actually an implementation of a new converter that understands and processes messages from the new device.
There is alternative of zigbee-herdsman-converters written in Python - zha-device-handlers. It uses zigpy for access to Zigbee messages, and it is used by Zigbee plugin for Home Assistant. zha-device-handlers contains a huge number of quirks for Zigbee devices, see subdirectories in the zhaquirks directory. zha-device-handlers has a great explanation of quirks for Zigbee devices and it is worth to read it - https://github.com/zigpy/zha-device-handlers#what-the-heck-is-a-quirk
I’d never heard of zigbee-lua, but it looks pretty stale compared
to
zigbee2mqtt. I don’t find the implementation language anything
more than
a implementation detail, especially not when typically running in
a
container.
I’ve used z2m on a raspberry pi and homeassistant on another. I
moved to
ZHA (the native Home assistant implementation) only by accident
and was
too lazy to start over again. But I’ll go back to z2m again after
moving
houses soon.
I have tried using a USB-to-zigbee adapter, but with custom software and not zigbee2mqtt.
Maybe the experience would have been better with zigbee2mqtt, but I generally like building my own stuff. From that perspective, the zigbee stack is not great, and my USB-to-zigbee interfered with the IKEA tradfri gateway pairing process.
I hope more modern smart home standards result in better ecosystems, but I’ll stick to the vendor gateways for now :)
The software mine came with was absolutely horrendous - which is usually the case for such systems. I updated the stick’s software and then uninstalled it :-)
The great thing is, that Zigbee2mqtt makes building your own stuff incredibly nice - you just start at a different level - MQTT instead of the stick’s serial interface. Zigbee2mqtt is the gateway and provides an API (via MQTT) to control devices in your Zigbee network.
That it will interfere with other gateway’s when both are open for devices to join is expected - but that’s expected. Multiple separate zigbee networks in the same house work fine. Devices just get confused which network to join when multiple gateways are in pairing mode.
In my use case the only gateway used is the usb stick used by zigbee2mqtt, which removes the need for all other Gateways, thus freeing you from having to use a different gateway for each vendor, and putting you much more in control of the stack.
Well, there’s always Matter which should come out sometime this
year. I’ll keep on using Home Assistant until Matter makes sense to
switch to - and then maybe. Home Assistant is pretty nice, not having to
care what ecosystem my stuff is in and just interact with all of them
instead.
I plan to make a first release of elle-cli, command line transactional safety checker based on Jepsen, Elle and Knossos. I would like to hear any feedback from everyone who involved in testing of distributed systems or DBMS.
Thanks, didn’t know about it. I made unreliablefs agnostic for operating systems and it has a single dependence - the FUSE library, that exists almost everywhere.
To be honest, this is not the first FUSE-based filesystem with fault injection, but other implementations do not match my requirements. I wanted an easy way to manage configuration at runtime, so I added an INI-like config file that is re-read on every change, and some specific error injection, such as setting real errors for the file operations being performed.
There is an article, if you are really curious about speeding up your CI.
There’s some good general info there, but the FactoryBot-specific optimizations in this post are still more relevant.
It is funny that popular test “bank” was not used for Datomic Pro used in NuBank :)
Thanks! Some time ago I wrote an arrticle about
git notestoo - https://github.com/ligurio/testres/wiki/Using-GIT-as-a-storage. It describes using notes as a storage fir testing results. Moreover, I’ve sent a patch for cgit that shows these testing results in web ui.The Sail project generated formal specifications of PowerPC by parsing the pseudocode in the PDF architecture references. I think that they found that the x86 docs at the time were too limited (inconsistent and incomplete) to do the same. I’m curious how well this project manages it. Arm eventually moved to auto-generating their pseudocode from their internal Architecture Specification Language (ASL) and there’s a tool that will transform ASL to Sail. The RISC-V official spec now uses Sail as well.
Seems you are talking about this project, - https://github.com/rems-project/sail. Haven’t heard about it. I didn’t get - is possible to translate assembler instructions to SMT representation (smt-lib2) using Sail?
Yup.
It can prove SMT properties inline and can export to a variety of theorem provers. We used this to prove monotonicity properties about CHERI capabilities, for example.
See also: ValidAlloy: A tool for validating a git alloy specification using testcase generation. It is an impressive work where model refined using real testcases for real Git implementation.
Oh wow! Thank you for sharing this. I’m gonna have to read it carefully
ccache is not a single way to speedup building a project. There are many other possible ways to speedup both building and testing. Most of them we implemented for Tarantool CI and reduced a time required for a feedback. If someone interested, I’ve prepared a list of such improvements in CI.
What would you use from that page to speed up a kernel build, except
ccache?paths-ignorethat skips running CI for changes in READMEs and other files that are not checked by jobs in CIThe only thing applicable for building Linux is parallel builds, and those are already implemented.
Test impact analysis could significantly reduce a total time required for testing a kernel, and it is not implemented.
Because there are no tests ;)
It is actually not so. See https://linux.kernelci.org/job/
Those aren’t used for local development.
Does anyone here use tarantool? Is there a smooth migration path from Postgres?
I ask because I’m currently implementing cubejs as a database cache/acceleration layer. The performance is decent BUT it’s a solution to poll the database for updates and rebuild the cache rather than a write-through solution.
I’m not using Tarantool, but I’m building it.
To be honest, Tarantool is not a drop-in replacement for PostgreSQL. Typical Tarantool usecases are cache, queue, sophisticated business logic on top of DB engine. Regarding Postgres - there is a connector to Postgres.
You have any case studies?
I would recommend joining telegram chat and ask questions there.
I’m polishing my native coverage-guided libFuzzer-based fuzzer for Lua before publishing source code.
I would propose to take a look on CTest. It is a part of CMake and allows running any tests. CTest is a test framework-agnostic tool. CTest allows running tests in parallel, rerun tests failed last time, filter tests by label etc.
See documentation for ctest.1 and for add_test.
Feel free to ask questions about integration with CTest if you are interested in it.
Valgrind is a nice tool, but AddressSanitizer is 10 times faster than Valgrind.
I’m surprised this is so new, as Lua is now on version 5.4.
Lua 5.1 is actively used, for example LuaJIT still supports 5.1+ partially. See also results of survey - https://docs.google.com/forms/d/1yacI6dj0d9OD8P5nhNp3647amjeVHauCvgVw9K3otFM/viewanalytics
Feel free to create manual pages missed for newer Lua versions.
It is worth to say that Elle is much better than Knossos, it allows checking longer histories.
By the way, it is not mandatory to use Jepsen for writing consistency tests. It is possible to make tests in any other programming language, produce a history in Jepsen-compatible and check it with elle-cli, a command-line frontend to transactional consistency checkers for black-box databases.
One thing the author seemingly hasn’t tried is using a manufacturer-independent gateway for Zigbee like https://www.zigbee2mqtt.io. This approach uses a USB-to-Zigbee adapter and makes all Zigbee devices from all vendors talk to each other. Run on a Raspberry Pi together with HomeAssistant (general-purpose home automation) or Homebridge (to bridge to Apple HomeKit), this setup provides a great user experience (for me).
It also opens two common and documented APIs for interfacing with the devices: HomeAssistant (high-level) and MQTT (lower-level, still easy).
I personally run Zigbee2mqtt with HomeBridge and control everything from the Apple ecosystem. It works flawlessly for me and my family. From the nerd-side, I stream all MQTT messages as JSON to a Postgres database and use Grafana to plot various metrics from sensors and devices.
Personally, I don’t like zigbee2mqtt. However, it is the best solution for vendor-independent gateway. I’m looking on zigbee-lua that uses Lua instead of Javascript and other alternatives.
I agree - the amount of code and complexity of the Javascript in zigbee2mqtt is staggering. Thanks for pointing me to zigbee-lua - looks great.
In an ideal world, there would be a standardised mqtt protocol definition with zigbee2mqtt and others implementing that protocol. From a short look it’s not entirely clear if zigbee-lua tries to be wire-compatible with zigbee2mqtt.
Absolutely agree.
As far as I get it right, the main complexity of zigbee2mqtt are quirks for different Zigbee devices that doesn’t conform to Zigbee specification. For interaction with Zigbee devices, z2m uses zigbee-herdsman-converters to parse messages to and from devices. Adding support of a new Zigbee device to z2m is actually an implementation of a new converter that understands and processes messages from the new device.
There is alternative of zigbee-herdsman-converters written in Python - zha-device-handlers. It uses zigpy for access to Zigbee messages, and it is used by Zigbee plugin for Home Assistant. zha-device-handlers contains a huge number of quirks for Zigbee devices, see subdirectories in the zhaquirks directory. zha-device-handlers has a great explanation of quirks for Zigbee devices and it is worth to read it - https://github.com/zigpy/zha-device-handlers#what-the-heck-is-a-quirk
I’d never heard of zigbee-lua, but it looks pretty stale compared to zigbee2mqtt. I don’t find the implementation language anything more than a implementation detail, especially not when typically running in a container.
I’ve used z2m on a raspberry pi and homeassistant on another. I moved to ZHA (the native Home assistant implementation) only by accident and was too lazy to start over again. But I’ll go back to z2m again after moving houses soon.
I have tried using a USB-to-zigbee adapter, but with custom software and not zigbee2mqtt.
Maybe the experience would have been better with zigbee2mqtt, but I generally like building my own stuff. From that perspective, the zigbee stack is not great, and my USB-to-zigbee interfered with the IKEA tradfri gateway pairing process.
I hope more modern smart home standards result in better ecosystems, but I’ll stick to the vendor gateways for now :)
The software mine came with was absolutely horrendous - which is usually the case for such systems. I updated the stick’s software and then uninstalled it :-)
The great thing is, that Zigbee2mqtt makes building your own stuff incredibly nice - you just start at a different level - MQTT instead of the stick’s serial interface. Zigbee2mqtt is the gateway and provides an API (via MQTT) to control devices in your Zigbee network.
That it will interfere with other gateway’s when both are open for devices to join is expected - but that’s expected. Multiple separate zigbee networks in the same house work fine. Devices just get confused which network to join when multiple gateways are in pairing mode.
In my use case the only gateway used is the usb stick used by zigbee2mqtt, which removes the need for all other Gateways, thus freeing you from having to use a different gateway for each vendor, and putting you much more in control of the stack.
Well, there’s always Matter which should come out sometime this year. I’ll keep on using Home Assistant until Matter makes sense to switch to - and then maybe. Home Assistant is pretty nice, not having to care what ecosystem my stuff is in and just interact with all of them instead.
I’ll discuss GSoC task with a student in a mentor role. We plan to implement a fuzzer for Lua interpreters (especially LuaJIT).
it’s worth use an existed fault injection filesystem like unreliablefs.
its easily possible to simulate faulty disks using dm-setup on linux, see: https://abbbi.github.io/dd/
Right. Except dd(1) and unreliablefs there are many other tools for injecting faults. See a diagram.
Thanks! Added it to the list with ASCII games.
I plan to make a first release of elle-cli, command line transactional safety checker based on Jepsen, Elle and Knossos. I would like to hear any feedback from everyone who involved in testing of distributed systems or DBMS.
I like the idea! I am not sure how to use it.
Reminds me of FreeBSD’s gnop(8).
Thanks, didn’t know about it. I made unreliablefs agnostic for operating systems and it has a single dependence - the FUSE library, that exists almost everywhere.
To be honest, this is not the first FUSE-based filesystem with fault injection, but other implementations do not match my requirements. I wanted an easy way to manage configuration at runtime, so I added an INI-like config file that is re-read on every change, and some specific error injection, such as setting real errors for the file operations being performed.