Amazon has a distributed transactional journal service they use internally for some projects like MemoryDB (discussed previously) which sounds more featureful - in particular the fencing API. Were they to expose/sell that service it would probably compete directly with S2 and maybe also Warpstream
Sorry I misunderstood your comment. I see you are suggesting the Journal if it launches as a service, would be a competitive threat. This could be true. We intend to be multi-cloud, which I don’t think the Journal will be ;-)
They’re aiming for 5ms latency in the future, but the current p99 latency appears to be ~400ms (standard) / ~28ms (express). I’m not really a cloud guy, so maybe someone can help alleviate my ignorance: who is this for? Especially the standard (~400 ms p99 latency) seems to be way too long to fit into a request-response cycle that a human is waiting for.
(S2 Founder) Yes, Standard is not appropriate for online usage, but those latencies work for high volume use cases like observability data and activity streams which are typically shipped asynchronously. The cost savings will be very worth it.
You may be surprised to learn that our 50ms end-to-end p99 SLO for Express is better than typical latencies for AWS Kinesis or GCP PubSub, and in the ballpark of Confluent’s managed Kafka offering called Kora (https://www.confluent.io/blog/kafka-vs-kora-latency-comparison/). A lot of online use cases will be fine with this kind of latency, but we definitely want to be able to offer better.
Object storage latencies in general are high, even with S3 Express (at least we don’t see the promised tail latency of single-digit millisecond, tends to be a bit higher) - and pricing incentivizes larger writes, so accumulating them up adds additional latency. The sketch of our 5 ms storage class involves introducing a disaggregated disk service into the system (so no longer just object storage, as is the case right now).
(S2 Engineer) Yup, the data will live in S2’s AWS. Not yet a BYOC but we want to explore this option in the future. Moreover, we do intend to be multi-cloud as well. We just started with AWS.
Amazon has a distributed transactional journal service they use internally for some projects like MemoryDB (discussed previously) which sounds more featureful - in particular the fencing API. Were they to expose/sell that service it would probably compete directly with S2 and maybe also Warpstream
I think you will be happy to learn, we have a fencing API! https://s2.dev/docs/stream#concurrency-control
Sorry I misunderstood your comment. I see you are suggesting the Journal if it launches as a service, would be a competitive threat. This could be true. We intend to be multi-cloud, which I don’t think the Journal will be ;-)
They’re aiming for 5ms latency in the future, but the current p99 latency appears to be ~400ms (standard) / ~28ms (express). I’m not really a cloud guy, so maybe someone can help alleviate my ignorance: who is this for? Especially the standard (~400 ms p99 latency) seems to be way too long to fit into a request-response cycle that a human is waiting for.
(S2 Founder) Yes, Standard is not appropriate for online usage, but those latencies work for high volume use cases like observability data and activity streams which are typically shipped asynchronously. The cost savings will be very worth it.
You may be surprised to learn that our 50ms end-to-end p99 SLO for Express is better than typical latencies for AWS Kinesis or GCP PubSub, and in the ballpark of Confluent’s managed Kafka offering called Kora (https://www.confluent.io/blog/kafka-vs-kora-latency-comparison/). A lot of online use cases will be fine with this kind of latency, but we definitely want to be able to offer better.
Object storage latencies in general are high, even with S3 Express (at least we don’t see the promised tail latency of single-digit millisecond, tends to be a bit higher) - and pricing incentivizes larger writes, so accumulating them up adds additional latency. The sketch of our 5 ms storage class involves introducing a disaggregated disk service into the system (so no longer just object storage, as is the case right now).
Thanks for your response!
Just to be clear, the data will sit on S2’s AWS account right? This is not (yet?) a BYOC kind of thing.
(S2 Engineer) Yup, the data will live in S2’s AWS. Not yet a BYOC but we want to explore this option in the future. Moreover, we do intend to be multi-cloud as well. We just started with AWS.