Does ingest require a predefined schema? If not, how do you handle converting the schemaless data into Parquet?
I’m working on something similar (log-store.com) and built a database to get around the issue of most databases and file formats requiring a schema… that and I like databases :-)
Heh I’m also working on a schemaless log ingest and search tool written in Rust, which is why I’m asking.
There is a MAP type and also a JSON/BSON type that would help. Maybe that’s what they use?