1. 38
  1.  

  2. 21

    For such a use case I’d really recommend tqdm, which does basically all of that (and more) automatically.

    https://tqdm.github.io/

    1. 4

      A progress library with custom merch. This is where we’re going?

      1. 3

        I recommend this lib in general too! Main problem is just having the foresight to have it installed in whatever env I’m working with. Of coures tqdm is Python-specific but you can replicate stuff like that in 20 lines or so if you want to.

        1. 1

          Yes, I think especially because it’s a nice-to-have dependency it’s often not included in a dependency list or environment explicitly. It also “feels” a bit heavy-weight, although it hardly adds a performance penalty to the execution itself.

          While tqdm is python-specific, if you go via the command line, then you can use tqdm as an executable, eg from the website:

          seq 9999999 | tqdm --bytes | wc -l
          

          Otherwise the stuff you outlines in the post is more generally applicable.

        2. 2

          There is also the language agnostic pv, but it’s limited in what it can output.

        3. 20

          I usually prefer to use \r (carriage return), without a newline: this preserves any text before. You’ll need to explicitly flush the output, though, since it’s usually line buffered:

          print(f"Processing {line}...", end="\r", flush=True)

          https://asciinema.org/a/CMgdWJCK5gT0DGfRrWd2z04pP

          1. 4

            oh that’s actually a really nice way of doing this!

            Of course for counting that works as-is, but I think with this you gotta actually print over all the output since you’re not clearing the results.

            In [3]: print("Processing 10000...\rProcessing 1..")
            Processing 1..00...
            
            1. 8

              Yes. Tracking the line length sounds good, but sometimes people type stuff on accident into the terminal, so unless you’re also disabling echo (or processing it) you might prefer end="\x1b[K\r"

              In general, before doing anything like this, I check if stdout is a tty (i.e. sys.stdout.isatty() in python) and then use fun graphics if they are, but just regular newlines if they’re not. This way the user can save your log output in a file “just” by piping it or redirecting the output and do other things with the log data instead of just watching it fly by.

            2. 3

              I think it’s slightly preferable to put the CR on the front so that the text always starts in the right place, even if something else has interfered with the terminal in the meantime.

            3. 6

              It is a neat trick, but I would not dare to use it. What if I accidentally erase an important clue that could lead me to the cause of the problem? An overload of data can simply be solved by piping the output into a file and then grepping it, or opening it in an editor.

              If simple print debugging does not work and some complexity is needed, I rather just go all the way and start an interactive debugger.

              1. 2

                especially that “rate limiting” idea is weird. What if the problem occured in one of the suppressed lines? That makes not much sense to me. If you go print debugging, then dump as much data as possible and use the good ol’ unix toolbox to filter it down later

              2. 1

                won’t time.time() call end up taking more time than the more important function call in a tight loop? I don’t like this method.