1. 29

So here’s the gist.

I’ve been digging for weeks to try and find a good solution to remote data connection access that’s compact, affordable and relatively easy to set up and start using in a global context and it’s unbelievably hard…

To give some context, I work in the humanitarian space, building data collection and analysis systems. We cover large swathes of Africa and the Pacific, but we have the consistent problem of connectivity issues, it’s oft-times our biggest problem. Our international field staff can get away with using data roaming but it gets crazy expensive very quickly and only helps our embedded staff and not local doctors, nurses and other field staff in-country.

What we need is a way to provide some type of data access (even if it’s just periodic) to these people for a mobile app or a desktop app. Email is also a bonus but not necessarily needed. All we really need at the base of it is the ability to remotely sync records that the user has submitted on their phone or in our desktop app on their laptop. We’ve dabbled with SMS, but some of these records are just too large and complex for SMS.

The overarching context of this is disease surveillance. We need to know when for instance a case or cases of Ebola have been reported in a remote village as quickly as possible. Right now, it can be weeks before we get that information because the user has to travel to get a good enough connection, or travel to a higher population density location where internet access is available or a larger refugee settlement or village, sometimes it’s as simple as having to just walk to the highest local point to get the hill, others it’s a matter of having to get in a truck and drive for hours to try and get a signal. Finding out a village has a case of ebola three weeks ago is not OK, we need that information as quickly as possible in order to make informed decisions about next steps and outbreak suppression in order to react quickly, our reaction times are fast, but our reporting delays for these remote locations aren’t.

That all being said, we’re budget constrained, we’re talking about countries which don’t have a ton of funds, we already provide mobile phones, a basic installable server as a relay node and a couple of cheap laptops in a giant black pelican case along with solar chargers. We have room for some kind of equipment to enable access in the remotest of locations. We have access to data sims from local telecoms but the cost is high for minimal amounts of data and provisioning 300+ sims quickly is a lot harder than provisioning a handful and our handsets aren’t the most powerful out there.

So basically I’m trying to draw on the lobste.rs communal knowledge and experience to see if you guys have knowledge of, or guidance on a workable solution to what is a herculean roadblock in the humanitarian space. A lot of the time the field staff aren’t technologically savvy, they’re smart people, but their exposure to complex tech is limited, we don’t have the resources to send out tech people to every village and location, it needs to be something that has few steps to set up and the guidance can be clear on it (i.e. put in sim card, plug in antennae, plug in box, hit switch kind of instructions).

tl;dr In a global context, what are some ideas for what I can stick in a pelican case that will guarantee some type of connectivity to remote locations so that we can gather data to prevent disease outbreaks?

  1.  

  2. 11

    Use a wifi gun to connect base-stations to each other. You can use similar point-to-point antennas to build cheap ISPs.

    1. 5

      That is amazing. I’ll have a look at point to point but looking at the specs on some of the point to point they can reach 12+ kms, but not sure that would be enough. Although in remote clusters it might work where we have lots of villages settlements clustered in a small region where at least one of the locations has a data connection.

    2. 10

      Never underestimate the effectiveness of pressing “sync” on a cheap Android device, then handing that phone to someone in a truck bound for the closest metropolitan.

      We encountered similar issues in the RapidFTR project. tl;dr we used CouchDB over wifi, bluetooth, and 3G (mobile data).

      Each device runs a CouchDB instance. Workers in refugee zones use their local devices to collect information. The devices talk to a local station, aggregating the information. The local station either has a direct uplink to the Internet, or can be brought into sync with a device that will eventually reach a place with decent Internet. Regardless, the underlying data and communication model was mesh-like. All devices held as much of a full history as possible— for a particular crisis event.

      (My colleagues also worked on Ebola projects… PM me, I could put you in contact!)

      1. 4

        Oh I definitely don’t underestimate it. That’s where we’re basically at now. But in comparison to PoC situations like refugee camps we end up in less centralised locations which is when it gets trickier. Instead of handling large dense clusters of people were covering entire coubtries or subsections of entire countries. So we have wide rural areas where people going in and out can be few and far between.

        Ebola is just a single disease we tack but were active across most of central Africa, the Pacific and parts of Asia.

        Africa is the place where we hit the most issues with connectivity though.

        Somebody needs to start like a humanitarian software dev working group for all the disparate projects out there so we can pool resources and workable solutions…

        1. 5

          Somebody needs to start like a humanitarian software dev working group for all the disparate projects out there so we can pool resources and workable solutions…

          IEEE Humanitarian Technology. “Tech4Dev” and the heaps of associated mailing lists.

          1. 5

            Considering i’ve been building systems in the humanitarian context for like eight years now it’s pretty shocking I didn’t know about these. I’m sort of under the radar most of the time.

            More or less was meaning something like IRC, lobste.rs or similar focused on humanitarian dev projects and people working on them. There are a whole lot of them when sometimes each and every department has autonomy to come up with their own solutions to similar problems.

            Will see if I can dig up some mailing lists…

      2. 7

        You have two different problems in your hands and I think they requires different solutions. Fast updates in case of events and faster (bulk) data upload. SMS and HAM radio are the cheapest possible form of fast update I can think of. It’s possible to do IP routing on radio equipment and I think you can establish a mesh service to provide internet to your user if they can bring some more equipment. Once they are online they can send you data but this leads to delays in update from a user. If your app can do peer to peer replicable data you can ameliorate this problem by sharing your data to others that will maybe come back online before you. Both problems can be solved using these kind of mesh services (hardware and software). A totally different approach is to become an ISP and negotiate peering with other providers, this can cut your bills and requires not that much infrastructure.

        1. 4

          SMS is hard to get up and running quickly, primarily because of trying get sims in bulk from whoever the local telecom is. What I was thinking was if we can get away with getting 4-5 sims to provide sort of connection points where we could set up a basic tower or something that broadcasts wifi over an extended area running off a solar battery. We used to have these wires for sat phones that we’d toss up a tree to get a signal for instance. Even something like amplifying the tether signal from an android phone then sending the message over 3g or SMS to the countries application servers. At least we’re then down to procuring a smaller batch of sims and we can spend less time haggling with the local telecom.

          Another thing we thought about sort of pie in the sky was drones, basically getting drones to do “rounds” where during the week they travel between locations, sending a notification to local mobile apps that they’ll have access to internet for the next 2 hours, or to plug in the drone and let it charge up so it can continue on it’s rounds, but drones open up a whole other can of worms in terms of regulations and also potentially getting shot down or just downright freaking people out. Even potentially setting up relay drones, where we drop drone “stations” on the way out to remote localities, then use drones to leapfrog to the location to provide syncing, lab sample pick up and leapfrog back to a central location. But they’re cost prohibitive for the complexity of the drone as well as in terms of dealing with local regulations and would only really be viable in a longer-term health surveillance context, would be too hard to get up and running in an emergency ramp-up.

          The HAM radio looks really interesting actually. I’m just digging into it, they don’t really need wider internet access, we just need a way to get data from them and in some cases send back some minimal data (for instance, the wait time for a lab result for these guys can be weeks, we can’t do much about the time it takes for a sample to get to a lab, but we can speed up the return time by pushing the lab result down to them).

          We’re also working on the peer to peer syncing of data on mobile and the desktop application. We’ve made some prototypes using the zyre c libs that are built on 0mq but it’s finicky so we can’t deploy it yet. The idea being that we eventually get data as users running our apps come into contact with each other and then eventually the wider internet or the ministry of healths or an NGOS installation of our systems.

          We’re sort of trying to build as many avenues as possible for them to get data in/out of the locality so there’s almost always an option because the context is so critical.

          The HAM radio packet data looks interesting, i’m definitely going to dig into that more and see if I can figure out prototype test for that for shifting data. Thanks for pointing me to that!

          1. 2

            The HAM radio looks really interesting actually. I’m just digging into it, they don’t really need wider internet access, we just need a way to get data from them and in some cases send back some minimal data (for instance, the wait time for a lab result for these guys can be weeks, we can’t do much about the time it takes for a sample to get to a lab, but we can speed up the return time by pushing the lab result down to them).

            AMPRNet will definitely be your friend here. Much of the foundation exists for you to build on. I’d wager it’s mostly an “amateur vs. non-amateur” question then.

        2. 7

          There is satellite based SMS, one such product: https://www.findmespot.com/en/index.php?cid=666. Important events you can almost certainly shrink into an SMS. something like: location 45 ebola +1

          Another option is to use a gossip protocol, something like https://www.scuttlebutt.nz/. It’s not really there for Mobile yet, but there are some Android implementations. Basically it lets any scuttlebutt user bring the data back, not just the one who entered it into their device, but securely knowing which user input the data. So the local staff will input the data into their device, and sync with the people that travel across the region(s). As travelers wander into better connectivity, they could then sync to your pub server, getting the data to you, while the local staff are still local doing their thing, and never had to leave. I’d recommend trying to get transport people that travel regularly through the region to be your traveling scuttlebutt nodes (bus drivers, water carriers, etc).

          The protocol is described here: https://ssbc.github.io/scuttlebutt-protocol-guide/

          The protocol is sufficiently general enough that you can run the git VCS across it.

          I run a public scuttlebutt pub here: https://www.zie.one/ if you want to try it out.

          1. 2

            In a lot of cases we need a bit more information than what a single SMS can provide.

            We’re already working on a sort of gossip protocol using the C zyre libraries on top of 0mq as well as bluetooth, it’s in development still but about 90% done, it’s a good idea, but it’s predicated on coming into contact with someone who is running our apps/systems in a timely manner, which may not be the case in a lot of instances. By the time they come into contact with another person they would probably be in a place that has internet or a connection of some sort.

            1. 3

              I don’t know your specific use-case, but at the very least you could easily get, “send someone out to collect more information” out of an SMS. Also, if you encode data, you can pack a fair bit of data into an SMS, given the constraints. But iridium and other satellite communications providers are not limited to SMS, the spot I showed is just the consumer level version of this, but if you are large enough you can reach out to the providers themselves and possibly work out some deal with them.

              Cool on the gossip protocol. The upside to doing something like SSB(scuttlebutt) is you get use-cases outside of just your domain, so the chances of people, such as transportation drivers, wanting to use it much higher. The more people you can get running a gossip protocol the faster you can sync your data across large swaths of unconnected/sporadically connected populations.

              Anyways, good luck!

          2. 5

            I would approach someone like Iridium and see if they want the PR for this.

            In particular, their https://www.iridium.com/products/iridium-go/ comes to mind.

            1. 3

              Given that you mostly are below 50 KB and that you specifically mentioned Ebola, I assume that a large portion of the work is done in Africa.

              In this case it seems like you are exactly the kind of customer the satellite company Thuraya is targeting. See http://www.thuraya.com/pricing-plans for tariff plans. They bundle satellite and roaming charges and there is also a coverage map of the network on that site.

              Their coverage is not suitable for maritime use or use in the America’s, but it works in most of the rest of the world.

              1. 2

                Called them, waiting for more information, they work through resellers I guess, and the resellers provide and SDK. Basically the units cost around £600 (~$800USD) not including the data plan and you have to use their iridium go app on the phone which has apps in it that are set up to work on the connection, so you can’t use a native app. As well the data is metered in something called go! minutes. Waiting for more information from them.

                1. 1

                  Maybe try talking to Viasat?

                  They might appreciate the PR/tax write off.

                  1. 2

                    Holy these guys look hardcore, military grade systems. Doing some more digging into this one to see what kind of offerings they have. Thank you.

                    1. 0

                      Strictly speaking, they’re just a small satellite-based ISP. ;)

              2. 5

                Brainstorming/spewing…

                • Maybe interesting article on similar problems in India, http://www.lowtechmagazine.com/2015/10/how-to-build-a-low-tech-internet.html.
                • If there’s line of sight, hacking a laser pointer for communication can be an option. The wikipedia Free-space optical communication page and Li-Fi have other and more recent links.
                • I like the idea of a gossip protocol and I’d also consider a general store and forward scheme like UUCP or FidoNet though that also assumes a relatively persistent route.
                • I’d attack the 50kb or larger payloads by trying a few things:
                  • start with straight compression, gzip, etc.
                  • encoding regularly used data into numeric columns with a composite structure, cf. IPUMS data sets and distributing the codebook separately and infrequently.
                  • reducing the character set for free fields to Base36 (A-Z0-9) or similar and then packing as was done for Z-Machine text.
                  • similarly, packed integers can reduce size.
                1. 2

                  These are all really good starting points, thank you. We dabbled with SMS sending data for a bit and I think we’re going to have to go down that route no matter what to have it as a fallback when data isn’t available. What we may have to do is subvert the telecoms a bit and set up an android app to act as a temporary SMS gateway with a sim in it so we can hit the ground running instead of trying to get a gateway provisioned in country.

                2. 4

                  Have a look at Radio e-mail in West Africa? In 2002 Wayne Marshall, working for the International Rescue Committee (now there’s a name straight out of (but much older than) Thunderbirds), used 9002 HF Data Modems by Codan to connect remote computers over 600 km (375 miles) away. The connection was apparently too slow for pleasant interactive sessions, but good enough for store-and-forward protocols, specifically e-mail. The article has a very thorough overview of how they configured qmail.

                  I don’t know how much these modems cost – vendors seem to operate on an ‘email us for a quote’ basis, which is an unpleasant practice and a bad sign. Found a similar-looking second-hand one in Australia for 350 AUD. Hmm.

                  Good luck, and go(o)d speed!

                  1. 2

                    This is an awesome article, thank you so much, I was still sort of casting around with articles that were getting far too into the radio technicalities rather than implementation details and this article has spelled it out pretty awesomely. Thank you!

                  2. 4

                    I don’t personally have any suggestions that haven’t already been mentioned, but you might want to contact the Rainforest Connection people and see if their stuff works for you. They’re streaming audio out of rainforests via cheap 2nd-hand cell phones and I think there was something with either bigger antennas for the phones or repeaters for the cell signal or something.

                    1. 3

                      Awesome! I’ve dropped them an email through their web form to see how they’re working it. Looks like they’re using much older gen phones as well as basic kits so might be a worthwhile avenue to look into how they’ve accomplished what they have (which is really cool from what i’ve read)

                    2. 3

                      I once came accross the company https://wefarm.org/ which uses a peer-to-peer SMS network between farmers in Africa. It may point you down a route of some kind.

                      1. 1

                        Wow this is another really cool project. Will email them and ask about how they’re accomplishing it. Thanks!

                        1. 2
                          1. 2

                            This looks really promising. I just had a quick chat with a rep at Arica Talks about the USSD codes they sell access to, right now it’s only 4 countries but trying to find out if I can buy sims in one of those countries and use the USSD codes while roaming in other countries!

                            Thank you this is an awesome find!

                      2. 3

                        I’m unclear if you need to build an infrastructure to support existing software/hardware tools or if you need software (and maybe devices) that is tolerant to the conditions of the infrastructure you have available.

                        This was recently posted to lobste.rs

                        http://www.lowtechmagazine.com/2015/10/how-to-build-a-low-tech-internet.html

                        I like Secure Scuttlebutt. There is also maybe Briar:

                        https://briarproject.org/

                        SSB and, I think Briar, support building applications on top of the protocol.

                        HAM, has rules (laws) that may or may not be an issue in places like Africa. For example, in the US, transmissions cannot be encrypted. They can be encoded in a publicly available protocol, but anyone must be able to decode it. If you have those constraints in-country, the data might be of a nature that you wouldn’t want to be transmitting it to the world.

                        1. 1

                          Its existing tools and software (I’m the developer). Essentially what i’m looking to get out of this post is a slew of ideas for what we can do when the best case scenarios for getting data out of a remote place fall down.

                          Right now the core of what we’re doing is selective syncing of data where the user has control over when they sync their data from the phone or desktop application. We’re moving to something similar to scuttlebutt but implemented on top of zyre on top of 0mq for the desktop application and trying to sort out a similar implementation on android where we can piggyback data through multiple nodes to the end central points.

                          I actually jumped into ##hamradio on freenode and asked some questions (and got what felt like a little bit a ribbing for not knowing enough about HAM radios) but the gist of what I got out of the conversation was that it’s probably only worth doing if you licence a frequency for use and even then it’s very complicated. I thin kit’s still worth digging into, just not something that could be implemented on a short turnaround time.

                          What I’m sort of looking at is all these amazing answers and pointers I’m getting from you guys and trying to sort of group and organize them and of the things that we can implement now, implement them so we have multiple modes per platform (ios, android, osx, windows, linux) for getting data to where it needs to go, the criticality in terms of timeframes for what we’re involved in means we need that data as soon as we can for decision making.

                          1. 1

                            I actually jumped into ##hamradio on freenode and asked some questions (and got what felt like a little bit a ribbing for not knowing enough about HAM radios) but the gist of what I got out of the conversation was that it’s probably only worth doing if you licence a frequency for use and even then it’s very complicated.

                            I’m not one but the ham’s I know are a kind of stodgy but knowledgeable bunch who expect you to have done your reading. Anyway, radio might not be out of the question and will depend on local regulations. In the USA (not true everywhere, for example Canada) there is MURS which requires no license, operates over the 151–154 MHz range (formerly business radio, so inexpensive transceivers), can be used for digital transmission, and can achieve several miles. At least in the USA packet-forwarding/repeating is prohibited which limits some interesting uses. You’ll need to ask someone who has local expertise as to what is permitted in each country.

                        2. 3

                          Have you tried BGAN? Satellite internet from a panel the size of a laptop. I field tested some of them for a project a few years ago, they worked pretty well and were easy to set up.

                          We didn’t end up going with them for that project, IIRC because it was for deployment in less remote locations in the US, and they didn’t support the US at the time, and we had good alternatives - cellular data in most places, and we had the transport capability to bring full-size satellite dishes.

                          1. 1

                            We actually have BGANs in some places, From what I understand they’re terribly expensive and we only have a limited number of them, so in a wider scale deployment it gets very expensive very fast to be distributing multiple BGANs and data plans all over the place.

                            To give you an idea in a non-emergency context, we may enter a country and start with 20, 30, 49 sentinel reporting sites. But if we go into outbreak mode, we’re talking potentially 100s of locations needing to sync data…

                            Coverage is actually pretty good with them in Africa as well, but maybe what we could do is use BGANs as a data access point and create local mesh networks or something for denser clusters of reporting sites…

                            1. 1

                              Ah, I see what you mean. I don’t suppose you’ve turned any procurement resources your org has towards seeing if you can get any kind of deal on them for being a humanitarian org? I didn’t really see the prices when I worked with them, but I thought it might not be too bad, since it sounds like your data requirements are pretty low.

                              Might be possible to do something with using them or some other satellite system as the backhaul for a local mesh network. Though I’d be worried that getting the point-to-point range above a few km and not using massive power would get you into an area where everything is either very technically complex or expensive, or both.

                          2. 2

                            Have you looked at secure scuttlebutt or dat? They are currently being used successfully for helping periodically-connected communities exchange information asynchronously, like remote villages and sailors etc…

                            1. 1

                              We’re working on an implementation similar to scuttlebutt but on top of the C zyre libs for our desktop app, looking at scuttlebutt for possible usage on our mobile app.

                              I haven’t seen dat before, doing a bit of reading on that to see if we can use it for some things.

                            2. 2

                              Hmm, you could probably look into HAM radio.

                              A LF or MF can go up to 2000km and 500km respectively (LowFER and MedFER, in reality probably a bit shorter).

                              Considering you need to transfer about 50kb, accounting for some overhead and buffer, a baud rate of about 8000-9000 baud should be enough to ensure up to a thousand stations can get their data send within a day guaranteed. Considering the constraints you’ll probably want to ensure data really gets there so you’d probably want around 200kHz at minimum and maybe transmit in something easy like morse. You’d have to build most of that from ground up and it could be a lot of work if existing tools don’t work (or adapt existing tools)

                              MF should handle your use case if the stations are around 100-200miles apart, if you rely on skywave overnight it can be more.

                              You’ll probably need a license in that band and find a frequency with the properties you want too.

                              Otherwise, there might be satellite internet but that might get expensive. Or Wifi Guns could be a solution.

                              1. 2

                                Possibly relevant is this (old, but accomplishing what you describe; not sure about how costly though): https://lobste.rs/s/kun536/radio_e_mail_west_africa_2002

                                1. 1

                                  Just read through this from another post above, awesome implementation and really good detail, taking notes down and see if I can get my hand on one of these modems (as well as seeing if using it where I am violates any laws)…

                                2. 2

                                  We’ve dabbled with SMS, but some of these records are just too large and complex for SMS.

                                  What would be the typical size for these records? Ordinary sms has 140 bytes, but you could use some binary serialization and concatenated SMS (pdu mode) to get more.

                                  1. 1

                                    We’re generally under 50kb, but we have contact trace forms that have up to 200 fields in them, some variable text length. I did a bit of looking into SMS but part of the problem is getting sims and an SMS gateway in place in country. At one point we were in talks with a local telecom for almost 3-4 months to get a gateway set up before they just stopped responding to our emails and they wouldn’t give us any specs or docs on how their gateways work or even what gateway they were using. This is very typical.

                                    1. 1

                                      This may be a silly question but are you sending compressed data?

                                      If your software uses the gzip, deflate, or zlib formats, zopfli is a compatible replacement that can generate much smaller files.

                                  2. 1

                                    I liked this article: How we built Watsi Coverage without stable electricity, WiFi, or email which discusses a number of problems that sounds like yours.

                                    1. 2

                                      I’d actually heard of these guys but hadn’t dug into them before. Man what I would give to have a team of people to do development… They’re running a pretty close parallel to ours but we’re specifically geared towards disease surveillance and outbreak management versus long-term health management (though we do stray into that when countries don’t have any viable alternative in place).

                                      Thanks for sharing this, I might actually contact these guys at some point to see if we can integrate in some meaningful ways.

                                    2. 1

                                      I think a BGAN is your best bet, even if it’s expensive. Alternatively, something from Iridium, like the RockBlock. Keep in mind that you won’t be pushing a whole lot of bandwidth out of it.