1. 17

Hi everyone, this is srv, a minimalist http server and file browser. I felt like lobsters readers would be interested in this kind of software.

I made it as a much lighter and faster alternative to python -m http.server. Benchmarks can be found here.

If you look at the source code, I went to some pretty great lengths to really make things as fast and simple as possible. Golang made it really easy to write something like this, I like the language a lot.

Please open issues if you’d like to request features or report bugs. Though I probably won’t accept most features (depends on how much complexity they’d introduce), I’d be interested in hearing people’s use cases.

  1.  

  2. 2
    1. 2

      Oh this is real nice, didn’t know about this one. It feels very similar to me in spirit. Reminder to self to match darkhttpd in logging info.

    2. 2

      I have something like this called quickserv that I recently ported to Rust from Go. I use this to host a production-facing website at work and for my various eBook projects.

      1. 2

        How does this compare to https://github.com/svenstaro/miniserve?

        1. 2

          Thanks for mentioning miniserve :D I’m currently rewriting it to work on stable rust and to upgrade its version of actix-web.

          1. 1

            I looked at that one while researching prior art, it seems good. More featureful than my personal liking though, for example, it lets you download a tarball which is nice in principle, but I’d just use the CLI for the rare occasion I’d personally need that, rather than embed all that extra code + complexity into srv.

            1. 1

              For science, I used ab -n 1000 -c 10 http://localhost:${PORT}/ against the latest miniserve from Homebrew and srv from GH Releases. I used default options for each serving up the content of my ~/Downloads directory.

              miniserve:

              Server Software:
              Server Hostname:        localhost
              Server Port:            8080
              
              Document Path:          /
              Document Length:        606474 bytes
              
              Concurrency Level:      10
              Time taken for tests:   14.784 seconds
              Complete requests:      1000
              Failed requests:        0
              Total transferred:      606594000 bytes
              HTML transferred:       606474000 bytes
              Requests per second:    67.64 [#/sec] (mean)
              Time per request:       147.842 [ms] (mean)
              Time per request:       14.784 [ms] (mean, across all concurrent requests)
              Transfer rate:          40068.29 [Kbytes/sec] received
              
              Connection Times (ms)
                            min  mean[+/-sd] median   max
              Connect:        0    0   0.2      0       6
              Processing:    43  147  36.4    139     279
              Waiting:       42  135  26.9    130     233
              Total:         43  147  36.4    140     279
              
              Percentage of the requests served within a certain time (ms)
                50%    140
                66%    164
                75%    171
                80%    176
                90%    196
                95%    220
                98%    232
                99%    239
               100%    279 (longest request)
              

              srv:

              Server Software:
              Server Hostname:        127.0.0.1
              Server Port:            8000
              
              Document Path:          /
              Document Length:        197365 bytes
              
              Concurrency Level:      10
              Time taken for tests:   10.032 seconds
              Complete requests:      1000
              Failed requests:        0
              Total transferred:      197486000 bytes
              HTML transferred:       197365000 bytes
              Requests per second:    99.68 [#/sec] (mean)
              Time per request:       100.323 [ms] (mean)
              Time per request:       10.032 [ms] (mean, across all concurrent requests)
              Transfer rate:          19223.65 [Kbytes/sec] received
              
              Connection Times (ms)
                            min  mean[+/-sd] median   max
              Connect:        0    0   0.3      0       6
              Processing:    31  100  48.4     92     566
              Waiting:       26   86  42.7     79     545
              Total:         31  100  48.4     92     566
              
              Percentage of the requests served within a certain time (ms)
                50%     92
                66%    107
                75%    118
                80%    126
                90%    146
                95%    170
                98%    241
                99%    291
               100%    566 (longest request)
              

              srv when I turned off console logging:

              Server Software:
              Server Hostname:        127.0.0.1
              Server Port:            8000
              
              Document Path:          /
              Document Length:        197365 bytes
              
              Concurrency Level:      10
              Time taken for tests:   8.793 seconds
              Complete requests:      1000
              Failed requests:        0
              Total transferred:      197486000 bytes
              HTML transferred:       197365000 bytes
              Requests per second:    113.73 [#/sec] (mean)
              Time per request:       87.931 [ms] (mean)
              Time per request:       8.793 [ms] (mean, across all concurrent requests)
              Transfer rate:          21932.85 [Kbytes/sec] received
              
              Connection Times (ms)
                            min  mean[+/-sd] median   max
              Connect:        0    0   0.2      0       4
              Processing:    21   87  32.3     82     257
              Waiting:       15   75  29.3     71     223
              Total:         21   87  32.3     82     257
              
              Percentage of the requests served within a certain time (ms)
                50%     82
                66%     96
                75%    105
                80%    111
                90%    132
                95%    150
                98%    170
                99%    187
               100%    257 (longest request)
              

              And then I tested with a ~130 MB file ab -n 100 -c 10 http://127.0.0.1:${PORT}/Youre_Not_Alone.zip:

              | metric                                      | miniserv   | srv        |
              |---------------------------------------------|------------|------------|
              | Requests per second [#/sec] (mean)          | 11.37      | 17.16      |
              | Time per request [ms] (mean)                | 879.423    | 582.788    |
              | Time per request [ms] (mean, all con. req.) | 87.942     | 58.279     |
              | Transfer rate [Kbytes/sec] received         | 1441324.06 | 2174943.64 |
              
              

              Cool!

              1. 1

                srv’s higher RPS is definitely due to the fact that I’m generating a very small amount of HTML. I bet if someone rewrote it in Rust it might get a tiny bit faster.

                1. 2

                  Certainly true. Look at the transfer rate, though, especially for the single file test.

                  1. 1

                    Especially? It doesn’t matter for the rest because the size per response is much higher for miniserve.

                    Anyways, that’s pretty surprising. You have a pretty fast disk, I’m jealous. Maybe srv is using better syscalls? Or maybe you ran miniserv first, and that warmed up all the I/O cache.

                2. 1

                  Does miniserve not log to console? [Comparing them with default options shows miniserve as having better tail latency unless you disable console logging for srv.]

                  1. 1

                    It doesn’t produce logs at all, IIRC.

                    1. 1

                      It doesn’t produce logs at all, IIRC.

              2. 1

                Edit: Sorry for the wall of text haha, I’m rereading sniffSignatures and it seems pretty good actually. TIL mkv shares the same signature as webm, and opus audio shares ogg. If I find anything that’s missing then I’ll just contribute it to golang.


                I forgot to add: does anyone have opinions on or a need for serving with the correct Content-Type mimetype? Python’s http server just sends everything as plain octet-stream, which means the browser won’t like, show an audio player or display the image.

                I do get some for free from golang’s DetectContentType, which is a few calls up the http stack, but if you look at sniffSignatures, it’s fairly incomplete. For example, there is no video/x-matroska.

                There’s 2 paths here: I could either resolve the file extension to the corresponding mimetype, or I could either swap out DetectContentType with something else that sniffs magic bytes… the most up-to-date is probably coreutils file, where the functionality i’d be looking for is in libmagic. There seems to be prior art here: https://github.com/rakyll/magicmime.

                I prefer the latter, since I don’t really trust file extensions; plenty of files I’ve downloaded in the past are suffixed, for example, with png when in reality they’re JPEG images.

                So definitely leaning towards exploring magicmime here, but I’m wondering if it’s really worth the added complexity. I honestly could even disable DetectContentType and hardcode an extension to mimetype mapping for some of the most popular media mimetypes.

                … I could also just contribute to golang’s DetectContentType itself.

                1. 3

                  Purely anecdotal, but as a semi-frequent user of -m http.server I apparently have never run into a situation where I realized nor cared that it wasn’t sending a mimetype. So, if that’s your target, maybe it’s not really needed.

                  1. 1

                    Oh! I just tried http.server for some things, looks like it’s sending it now…

                    I wrote the first minor version of srv quite a while ago and I could have sworn it didn’t send mimetypes. Or maybe I was using an older python version, I dunno.

                    1. 2

                      I was curious so checked out the implementation:

                      At least as of 3.9 http.server overrides and explicitly specifies types for a small number of compression formats (not sure why, wasn’t commented, and I didn’t dig into the git blame there) – but otherwise tries to lookup a type via mimetypes.guess_type() which loads mappings from filename extension to type from a hodgepodge of different config files on posixy systems, or the registry on windows. If a file doesn’t have an extension, or the extension doesn’t have a mapping in either of those, it defaults to application/octet-stream. Notably it doesn’t attempt to look for magic numbers or anything like that.

                      1. 1

                        Notably it doesn’t attempt to look for magic numbers or anything like that.

                        Sure, since it’s mapping from extension to mimetypes using those databases. Quoting myself:

                        There’s 2 paths here: I could either resolve the file extension to the corresponding mimetype, or I could either swap out DetectContentType with something else that sniffs magic bytes…

                        I prefer the latter, since I don’t really trust file extensions

                        Those are two separate approaches to guessing the Content-Type, the latter being more correct.

                    2. 1

                      One case I ran in to it is when developing an app which uses WASM locally, which needs to be served with application/wasm or it won’t work.

                      It’s easily solved with Python’s http.serve, but it took me a bit to figure out since the browser message wasn’t too helpful.

                      #!/usr/bin/env python3
                      
                      import http.server
                      h = http.server.SimpleHTTPRequestHandler
                      h.extensions_map = {'': 'text/html', '.wasm': 'application/wasm', '.js': 'application/javascript'}
                      http.server.HTTPServer(('127.0.0.1', 2000), h).serve_forever()
                      
                      1. 1

                        FYI, mimtypes.types_map in the standard library is a dict mapping common file extensions to their MIME type values. On my system it already knows .js and .wasm and has the right mappings for them, so setting extensions_map to that would probably do what you want:

                        Python 3.8.0 (default, Nov 23 2019, 00:30:22)
                        Type 'copyright', 'credits' or 'license' for more information
                        IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help.
                        
                        In [1]: import mimetypes
                        
                        In [2]: mimetypes.types_map['.wasm']
                        Out[2]: 'application/wasm'
                        
                        In [3]: mimetypes.types_map['.js']
                        Out[3]: 'application/javascript'
                        

                        (or if yours is missing types you want, I guess .copy() the base one and update it for one-off use, or register the new types through mimetypes.add_type())

                        1. 1

                          Surprise! Go has that: https://golang.org/src/net/http/sniff.go#L197

                          Therefore srv does too!

                    3. 1

                      I really like the idea of a simple server focused on one-off interactive runs. I typically use caddy, instead of Python’s built-in server.

                      Where do you feel srv sits relative to caddy (especially since they’re both written in Go)?

                      My only complaint of caddy is that it focuses a bit much on configurability for daemon mode rather than ease of use for interactive mode: caddy2 needs to be run as caddy file-server --listen :2015 which is a bit more annoying than caddy1’s caddy browse. Not entirely sure if this is something that is PR-worthy for caddy, or if it’s better to focus on a dedicated tool like srv.

                      [Edit: Articulating this finally prompted me to open an issue] [Edit2: Aaaand closed as WONTFIX, nevermind.]

                      1. 2

                        srv is way, way simpler than caddy, but they’re trying to solve different problems. As you mentioned: a simple server focused on one-off interactive runs. Caddy’s like, a full-blown nginx competitor.

                        You can quite literally just type srv:

                        $ srv
                        2020-06-06 11:52:59     Serving . over HTTP on 127.0.0.1:8000
                        
                      2. 1

                        I am more of a “php -S localhost:3000” person myself. Any idea now srv compares to that?

                        1. 1

                          Well, for starters, I did php -S localhost:8000, then requested http://localhost:8000/ and got a 404:

                          PHP 7.3.11 Development Server started at Sat Jun  6 01:56:28 2020
                          Listening on http://localhost:8000
                          Document root is /Users/joshua.li/bin/src/srv
                          Press Ctrl-C to quit.
                          [Sat Jun  6 01:56:31 2020] [::1]:61468 [404]: / - No such file or directory
                          

                          So there’s that!

                          1. 1

                            Never had that happen to me… Well, glad you have srv then!

                            1. 2

                              I’m calling it right now, you’re now cursed to encounter it sometime in the future. :)

                          2. 1

                            Even PHP’s built-in server has some whack design choices: https://bugs.php.net/bug.php?id=69655

                            1. 1

                              That’s why everyone says “don’t use it in prod”! I’ll be honest with you, most of my dev happens in php so it’s just the easiest thing to grab. Might switch to OP’s solution once I get the chance to try it

                              1. 2

                                That’s why everyone says “don’t use it in prod”!

                                Not entirely sure whether you understand the issue I linked. This is downright stupid in development too IMO. If anything the singular reason it works that way (performance) makes less sense in development.

                                1. 1

                                  I agree that I must not be understanding the implications of this, it sounded like a simple bug. But thanks for clarifying, I will switch servers for development, OP’s solution looks great!

                                  1. 1

                                    Basically the php server freezes, drops connections, or returns 415 when it encounters unknown or malformed http methods. “unknown methods” includes some exotic but valid HTTP methods. I am not sure if this is still the case, but it did cost me a lot of time in debugging back then. Surely this matters less with static hosting where everything is GET

                          3. 1

                            Does it comes free with the standard library of a language that does many other things or do I have to install it explicitly? :-)

                            1. 1

                              How well does it handle forking? One semi-common use I have is having multiple people try to test a website. Right now I use code like the below. Does this roughly do the same?

                              #!/bin/env python
                              
                              from socketserver import ForkingMixIn
                              from http.server import SimpleHTTPRequestHandler, HTTPServer
                              
                              class ForkingSimpleServer(ForkingMixIn, HTTPServer):
                                  pass
                              
                              import sys
                              import os
                              
                              if sys.argv[1:]:
                                  port = int(sys.argv[1])
                              else:
                                  port = 8000
                              
                              if sys.argv[2:]:
                                  os.chdir(sys.argv[2])
                              
                              server = ForkingSimpleServer(('', port), SimpleHTTPRequestHandler)
                              try:
                                  while 1:
                                      sys.stdout.flush()
                                      server.handle_request()
                              except KeyboardInterrupt:
                                  print("Finished")
                              
                              1. 1

                                It is written in Go. There is no forking, concurrent requests are handled via concurrent goroutines.