1. 3

Hello, I see sometimes on twitter #AWSWishList.

Here is mine (extracted from https://medium.com/appaloosa-store-engineering/from-mongodb-to-aws-redshift-a-practical-guide-5ec8ee8fb147)

  • Kinesis : We should be able to more easily validate the uniqueness of the Kinesis output or Kinesis consumer input.
  • Lambda timeouts: Have more infos about lambda timeout
  • Cloudwatch alarms: actually it’s not possible to get alarms that combine multiple metrics. I would like to report all alarm for a group of lambda (per environment).
  • Lambda errors reporting: You have to build your own system if you want to be notified when a lambda fails. It still misses a dedicated errors reporting system to have the complete stacktrace of a failing lambda. Also it should split the errors into two groups (timeouts and errors) that occurred inside the lambda. Right now, lambda errors are all grouped together, which is not ideal.
  • Redshift: Some useful SQL commands are missing from the syntax (DISTINCT ON, GENERATE_SERIES()..).

  2. 2

    On the input/output uniqueness validation, the canonical way to do this is to create and maintain your own bloom filter; the precise method depends pretty heavily on what your stack and your use case looks like, but, e.g., https://github.com/Baqend/Orestes-Bloomfilter might give you some ideas/pointers/clues.

    On the Lambda timeouts, what specifically are you looking to understand? https://aws.amazon.com/lambda/faqs/ tells us that there’s a 3 second execution timeout that you can configure up to 300 seconds if you need. If you want to understand why you’re timing out, then that’s an application level concern that you need to instrument through application level logging/tracing/analysis libraries.

    On Lambda error reporting, generally the best practice is to take the logs and put them somewhere as a unified whole, and then run whatever processing on them you believe may be necessary, rather than split them at the application level and then try to have multiple handling paths. This is especially true in distributed environments where there might be hundreds or thousands of workers running on different machines and you may have a variety of questions that might have correlated answers, like timeouts and such (e.g. a centralized db table is missing an index and the query planner gave up).

    On Redshift, there are probably many SQL commands “missing”; the world of SQL database command implementation is one of thousands of overlapping venn diagrams, none of which are quite alike. In some cases the underlying technical implementation may make certain commands impossible or incredibly expensive; in others, it may be that there hasn’t been enough obvious demand yet to implement the functionality. For DISTINCT ON, you might try something like this: https://gist.github.com/jmindek/62c50dd766556b7b16d6 and for GENERATE_SERIES(), you might try the ‘old school’ way: https://www.periscopedata.com/blog/generate-series-in-redshift-and-mysql.html .

    (disclaimer: I work for Amazon, but not in AWS)

    1. 1

      Thanks a lot for your answer. I didn’t know about bloomfilter. Will take a look.

      For lambda timeouts it’s only because I like to understand what’s happen and if I can do something about it. The only way I found it’s to add logs on a every step of the lambda to check where it stops.

      Yes you’re correct. I just thought that maybe AWS could split in two type of errors (lambda error and timeout). I saw that people were using elasticsearch and kibana or iopipe but we are a small team and we can spent some times on this for the moment. For the moment with the number of lambda we have cloudwatch is nearly enough.

      It’s funny because last comment it’s me :) Thanks for the other link.