1. 4

    We’ve been discussing potential solutions, and came up with three avenues, listing their pros and cons. We’re leaning towards a “Sandbox accounts in production” approach at the moment, but recognize that none of the solutions are free, and whatever we decide on will have a big impact going forward.

    Production-like environment

    + Clear, separate environment
    + No impact on production data / users (events, emails)
    + Not a problem if we mess up data without cleaning up afterwards
    + Runs the same code as on production
    + Requires very little changes to existing code
    - non-production code can (and will be!) deployed, making it no longer "production-like"
    - data needs to be kept in sync, to be usable by clients
    - every service needs to run both a production instance, and a production-like instance
    - all external services need to work with this production-like setup (payments, emails, etc...)
    - higher ongoing maintenance overhead
    - requires "manual agreements" on how to manage/manipulate this environment

    Individual temporary environments

    + all the pros of the "Production-like environment"
    + even more isolated, higher guarantee of your expected state of the environment
    - requires a lot of CI/operational changes
    - requires ongoing maintenance to keep working
    - requires "operational" knowledge to make new changes work with this setup
    - requires more syncing of data
    - higher costs of running

    Sandbox accounts with special “capability” flags, in production

    + All test-data is scoped to a (sandbox) user account, single "source of truth" on wether some piece of data is test-data
    + One single environment to maintain, no divergence
    + Data is always the same as production
    + Whatever code runs on production, is what you test
    + Because you want to test your changes, you automatically make sure your new code works with sandbox accounts / capability flags
    + You deploy your pre-production changes behind a capability flag, for testing
    + A preference page allows you to enable/disable certain capabilities to enable real/fake PSP environments, pre-release functionality, etc...
    - Much more complex to realise
    - Only works (well) for user-scoped test-data
    - Might result in higher learning curve to ship a feature that works with sandbox+capability flags
    - Production data can be changed accidentally, there's nothing but our own code between test and production data
    - Other systems need to be able to handle (and/or ignore) test data
    - Only suitable to the specific use-case of testing user-flows, not for testing f.e. if a library upgrade broke anything