srv: minimalist alternative to python -m http.server

17

srv: minimalist alternative to python -m http.server ☶ go github.com
authored by JoshuaRLi 3 years ago | archive
Archive.org Archive.today Ghostarchive
| 34 comments

34

Hi everyone, this is srv, a minimalist http server and file browser. I felt like lobsters readers would be interested in this kind of software.

I made it as a much lighter and faster alternative to python -m http.server. Benchmarks can be found here.

If you look at the source code, I went to some pretty great lengths to really make things as fast and simple as possible. Golang made it really easy to write something like this, I like the language a lot.

Please open issues if you’d like to request features or report bugs. Though I probably won’t accept most features (depends on how much complexity they’d introduce), I’d be interested in hearing people’s use cases.

2

qznc 3 years ago | link

There is also https://unix4lyfe.org/darkhttpd/
1. 2
  
  JoshuaRLi 3 years ago | link
  
  Oh this is real nice, didn’t know about this one. It feels very similar to me in spirit. Reminder to self to match darkhttpd in logging info.
2

cadey 3 years ago | link

I have something like this called quickserv that I recently ported to Rust from Go. I use this to host a production-facing website at work and for my various eBook projects.

colindean 3 years ago | link

How does this compare to https://github.com/svenstaro/miniserve?

2

svenstaro 3 years ago | link

Thanks for mentioning miniserve :D I’m currently rewriting it to work on stable rust and to upgrade its version of actix-web.

JoshuaRLi 3 years ago | link

I looked at that one while researching prior art, it seems good. More featureful than my personal liking though, for example, it lets you download a tarball which is nice in principle, but I’d just use the CLI for the rare occasion I’d personally need that, rather than embed all that extra code + complexity into srv.

colindean 3 years ago | link

For science, I used ab -n 1000 -c 10 http://localhost:${PORT}/ against the latest miniserve from Homebrew and srv from GH Releases. I used default options for each serving up the content of my ~/Downloads directory.

miniserve:

Server Software:
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        606474 bytes

Concurrency Level:      10
Time taken for tests:   14.784 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      606594000 bytes
HTML transferred:       606474000 bytes
Requests per second:    67.64 [#/sec] (mean)
Time per request:       147.842 [ms] (mean)
Time per request:       14.784 [ms] (mean, across all concurrent requests)
Transfer rate:          40068.29 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       6
Processing:    43  147  36.4    139     279
Waiting:       42  135  26.9    130     233
Total:         43  147  36.4    140     279

Percentage of the requests served within a certain time (ms)
  50%    140
  66%    164
  75%    171
  80%    176
  90%    196
  95%    220
  98%    232
  99%    239
 100%    279 (longest request)

srv:

Server Software:
Server Hostname:        127.0.0.1
Server Port:            8000

Document Path:          /
Document Length:        197365 bytes

Concurrency Level:      10
Time taken for tests:   10.032 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      197486000 bytes
HTML transferred:       197365000 bytes
Requests per second:    99.68 [#/sec] (mean)
Time per request:       100.323 [ms] (mean)
Time per request:       10.032 [ms] (mean, across all concurrent requests)
Transfer rate:          19223.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0       6
Processing:    31  100  48.4     92     566
Waiting:       26   86  42.7     79     545
Total:         31  100  48.4     92     566

Percentage of the requests served within a certain time (ms)
  50%     92
  66%    107
  75%    118
  80%    126
  90%    146
  95%    170
  98%    241
  99%    291
 100%    566 (longest request)

srv when I turned off console logging:

Server Software:
Server Hostname:        127.0.0.1
Server Port:            8000

Document Path:          /
Document Length:        197365 bytes

Concurrency Level:      10
Time taken for tests:   8.793 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      197486000 bytes
HTML transferred:       197365000 bytes
Requests per second:    113.73 [#/sec] (mean)
Time per request:       87.931 [ms] (mean)
Time per request:       8.793 [ms] (mean, across all concurrent requests)
Transfer rate:          21932.85 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       4
Processing:    21   87  32.3     82     257
Waiting:       15   75  29.3     71     223
Total:         21   87  32.3     82     257

Percentage of the requests served within a certain time (ms)
  50%     82
  66%     96
  75%    105
  80%    111
  90%    132
  95%    150
  98%    170
  99%    187
 100%    257 (longest request)

And then I tested with a ~130 MB file ab -n 100 -c 10 http://127.0.0.1:${PORT}/Youre_Not_Alone.zip:

| metric                                      | miniserv   | srv        |
|---------------------------------------------|------------|------------|
| Requests per second [#/sec] (mean)          | 11.37      | 17.16      |
| Time per request [ms] (mean)                | 879.423    | 582.788    |
| Time per request [ms] (mean, all con. req.) | 87.942     | 58.279     |
| Transfer rate [Kbytes/sec] received         | 1441324.06 | 2174943.64 |

Cool!

1

adsouza 3 years ago | link

Does miniserve not log to console? [Comparing them with default options shows miniserve as having better tail latency unless you disable console logging for srv.]
1. 1
  
  colindean 3 years ago | link
  
  It doesn’t produce logs at all, IIRC.
2. 1
  
  colindean 3 years ago | link
  
  It doesn’t produce logs at all, IIRC.
1

JoshuaRLi 3 years ago | link

srv’s higher RPS is definitely due to the fact that I’m generating a very small amount of HTML. I bet if someone rewrote it in Rust it might get a tiny bit faster.
1. 2
  
  colindean 3 years ago | link
  
  Certainly true. Look at the transfer rate, though, especially for the single file test.
  1. 1
    
    JoshuaRLi 3 years ago | link
    
    Especially? It doesn’t matter for the rest because the size per response is much higher for miniserve.
    
    Anyways, that’s pretty surprising. You have a pretty fast disk, I’m jealous. Maybe srv is using better syscalls? Or maybe you ran miniserv first, and that warmed up all the I/O cache.

1

JoshuaRLi edited 3 years ago | link

Edit: Sorry for the wall of text haha, I’m rereading sniffSignatures and it seems pretty good actually. TIL mkv shares the same signature as webm, and opus audio shares ogg. If I find anything that’s missing then I’ll just contribute it to golang.

I forgot to add: does anyone have opinions on or a need for serving with the correct Content-Type mimetype? Python’s http server just sends everything as plain octet-stream, which means the browser won’t like, show an audio player or display the image.

I do get some for free from golang’s DetectContentType, which is a few calls up the http stack, but if you look at sniffSignatures, it’s fairly incomplete. For example, there is no video/x-matroska.

There’s 2 paths here: I could either resolve the file extension to the corresponding mimetype, or I could either swap out DetectContentType with something else that sniffs magic bytes… the most up-to-date is probably coreutils file, where the functionality i’d be looking for is in libmagic. There seems to be prior art here: https://github.com/rakyll/magicmime.

I prefer the latter, since I don’t really trust file extensions; plenty of files I’ve downloaded in the past are suffixed, for example, with png when in reality they’re JPEG images.

So definitely leaning towards exploring magicmime here, but I’m wondering if it’s really worth the added complexity. I honestly could even disable DetectContentType and hardcode an extension to mimetype mapping for some of the most popular media mimetypes.

… I could also just contribute to golang’s DetectContentType itself.
1. 3
  
  abeyer 3 years ago | link
  
  Purely anecdotal, but as a semi-frequent user of -m http.server I apparently have never run into a situation where I realized nor cared that it wasn’t sending a mimetype. So, if that’s your target, maybe it’s not really needed.
  1. 1
    
    JoshuaRLi 3 years ago | link
    
    Oh! I just tried http.server for some things, looks like it’s sending it now…
    
    I wrote the first minor version of srv quite a while ago and I could have sworn it didn’t send mimetypes. Or maybe I was using an older python version, I dunno.
    1. 2
      
      abeyer edited 3 years ago | link
      
      I was curious so checked out the implementation:
      
      At least as of 3.9 http.server overrides and explicitly specifies types for a small number of compression formats (not sure why, wasn’t commented, and I didn’t dig into the git blame there) – but otherwise tries to lookup a type via mimetypes.guess_type() which loads mappings from filename extension to type from a hodgepodge of different config files on posixy systems, or the registry on windows. If a file doesn’t have an extension, or the extension doesn’t have a mapping in either of those, it defaults to application/octet-stream. Notably it doesn’t attempt to look for magic numbers or anything like that.
      1. 1
        
        JoshuaRLi 3 years ago | link
        
        Notably it doesn’t attempt to look for magic numbers or anything like that.
        
        Sure, since it’s mapping from extension to mimetypes using those databases. Quoting myself:
        
        There’s 2 paths here: I could either resolve the file extension to the corresponding mimetype, or I could either swap out DetectContentType with something else that sniffs magic bytes…
        
        I prefer the latter, since I don’t really trust file extensions
        
        Those are two separate approaches to guessing the Content-Type, the latter being more correct.
  2. 1
    inactive-user edited 3 years ago | link
    
    One case I ran in to it is when developing an app which uses WASM locally, which needs to be served with application/wasm or it won’t work.
    
    It’s easily solved with Python’s http.serve, but it took me a bit to figure out since the browser message wasn’t too helpful.
    
    #!/usr/bin/env python3 import http.server h = http.server.SimpleHTTPRequestHandler h.extensions_map = {'': 'text/html', '.wasm': 'application/wasm', '.js': 'application/javascript'} http.server.HTTPServer(('127.0.0.1', 2000), h).serve_forever()
    1. 1
      
      JoshuaRLi 3 years ago | link
      
      Surprise! Go has that: https://golang.org/src/net/http/sniff.go#L197
      
      Therefore srv does too!
    2. 1
      
      ubernostrum edited 3 years ago | link
      
      FYI, mimtypes.types_map in the standard library is a dict mapping common file extensions to their MIME type values. On my system it already knows .js and .wasm and has the right mappings for them, so setting extensions_map to that would probably do what you want:
      
      Python 3.8.0 (default, Nov 23 2019, 00:30:22) Type 'copyright', 'credits' or 'license' for more information IPython 7.13.0 -- An enhanced Interactive Python. Type '?' for help. In [1]: import mimetypes In [2]: mimetypes.types_map['.wasm'] Out[2]: 'application/wasm' In [3]: mimetypes.types_map['.js'] Out[3]: 'application/javascript'
      
      (or if yours is missing types you want, I guess .copy() the base one and update it for one-off use, or register the new types through mimetypes.add_type())
1

yarmo 3 years ago | link

I am more of a “php -S localhost:3000” person myself. Any idea now srv compares to that?
1. 1
  JoshuaRLi 3 years ago | link
  Well, for starters, I did php -S localhost:8000, then requested http://localhost:8000/ and got a 404:
  
  PHP 7.3.11 Development Server started at Sat Jun 6 01:56:28 2020 Listening on http://localhost:8000 Document root is /Users/joshua.li/bin/src/srv Press Ctrl-C to quit. [Sat Jun 6 01:56:31 2020] [::1]:61468 [404]: / - No such file or directory
  
  So there’s that!
  1. 1
    
    yarmo 3 years ago | link
    
    Never had that happen to me… Well, glad you have srv then!
    1. 2
      
      JoshuaRLi 3 years ago | link
      
      I’m calling it right now, you’re now cursed to encounter it sometime in the future. :)
2. 1
  
  untitaker 3 years ago | link
  
  Even PHP’s built-in server has some whack design choices: https://bugs.php.net/bug.php?id=69655
  1. 1
    
    yarmo 3 years ago | link
    
    That’s why everyone says “don’t use it in prod”! I’ll be honest with you, most of my dev happens in php so it’s just the easiest thing to grab. Might switch to OP’s solution once I get the chance to try it
    1. 2
      
      untitaker 3 years ago | link
      
      That’s why everyone says “don’t use it in prod”!
      
      Not entirely sure whether you understand the issue I linked. This is downright stupid in development too IMO. If anything the singular reason it works that way (performance) makes less sense in development.
      1. 1
        
        yarmo 3 years ago | link
        
        I agree that I must not be understanding the implications of this, it sounded like a simple bug. But thanks for clarifying, I will switch servers for development, OP’s solution looks great!
        
        1
        
        untitaker 3 years ago | link
        
        Basically the php server freezes, drops connections, or returns 415 when it encounters unknown or malformed http methods. “unknown methods” includes some exotic but valid HTTP methods. I am not sure if this is still the case, but it did cost me a lot of time in debugging back then. Surely this matters less with static hosting where everything is GET

zxtx 3 years ago | link

How well does it handle forking? One semi-common use I have is having multiple people try to test a website. Right now I use code like the below. Does this roughly do the same?

#!/bin/env python

from socketserver import ForkingMixIn
from http.server import SimpleHTTPRequestHandler, HTTPServer

class ForkingSimpleServer(ForkingMixIn, HTTPServer):
    pass

import sys
import os

if sys.argv[1:]:
    port = int(sys.argv[1])
else:
    port = 8000

if sys.argv[2:]:
    os.chdir(sys.argv[2])

server = ForkingSimpleServer(('', port), SimpleHTTPRequestHandler)
try:
    while 1:
        sys.stdout.flush()
        server.handle_request()
except KeyboardInterrupt:
    print("Finished")

1

jrwren 3 years ago | link

It is written in Go. There is no forking, concurrent requests are handled via concurrent goroutines.

1

shazow edited 3 years ago | link

I really like the idea of a simple server focused on one-off interactive runs. I typically use caddy, instead of Python’s built-in server.

Where do you feel srv sits relative to caddy (especially since they’re both written in Go)?

My only complaint of caddy is that it focuses a bit much on configurability for daemon mode rather than ease of use for interactive mode: caddy2 needs to be run as caddy file-server --listen :2015 which is a bit more annoying than caddy1’s caddy browse. Not entirely sure if this is something that is PR-worthy for caddy, or if it’s better to focus on a dedicated tool like srv.

[Edit: Articulating this finally prompted me to open an issue] [Edit2: Aaaand closed as WONTFIX, nevermind.]
1. 2
  JoshuaRLi 3 years ago | link
  srv is way, way simpler than caddy, but they’re trying to solve different problems. As you mentioned: a simple server focused on one-off interactive runs. Caddy’s like, a full-blown nginx competitor.
  
  You can quite literally just type srv:
  
  $ srv 2020-06-06 11:52:59 Serving . over HTTP on 127.0.0.1:8000
1

azazel 3 years ago | link

Does it comes free with the standard library of a language that does many other things or do I have to install it explicitly? :-)