Socket-served Internet mapping, anyone?
I may or may not be on to something.
For awhile I’ve been growing more and more tired of the typical Internet mapping user experience. It’s basically this: open a map and zoom/pan somewhere, wait a moment while (hopefully) some “working” graphic gives you a clue that the map hasn’t forgot about you; meanwhile, some basemap tiles load, then after a moment or two, “poof!” all the interactive data (i.e. the sole reason you are using that particular map) appears all at once.
Fortunately, in most cases, the basemap tiles “sprinkle in” one by one as they become available, which does give the user an impression that something is happening. But more often than not, basemap tiles dramatically out-perform the vector data component, and the overall feel of the map becomes this kind of “clicky kludgy .. ..CLUNK”, as the tiles pepper in, followed by the hammer that is all the vector data landing at once.
So I’ve always looked for ways to streamline the vector IO as much as possible–1) Maybe it’s possible to simplify the data a little, removing redundant vertices? 2) Maybe it’s possible to return that data with less coordinate precision (i.e. the number of decimal places in the coordinate values), because a precision above 8 probably isn’t even legit. 3) Is the database using it’s geometry index, or is it possible to better-tune the DB so that it handles the Data Access step more athletically? 4) ..what about pre-cacheing the entire data table into RAM so the DB never has to hit disk to serve a query on that table? ….that’s pretty much the request/access step, but what about 5) returning the data in the leanest possible format–for example, GML is much heavier than GeoJSON. Or, 7) what about returning a home-rolled JSON response using Well Known Text (WKT) and just a few important fields, rather than providing the entire record for every feature?
I feel like I’ve tried everything I know to try, and I just cant serve a deep table of polygons (specifically, parcel polygons) to go through all the motions as fast as I’d like. At the end of the day, the choke-point is the HTTP protocol, itself, and the very client <–> server paradigm that represents almost 100% of the static web. (By static web, I mean basic websites where you go to fetch assets, read/learn things, watch things, etc. The Dynamic web would be things like video games, chat rooms, Skype, and stuff like that.)
That got me to thinking, maybe an answer lives somewhere in the Dynamic web. A major service-level component of the dynamic web is the “Socket”, and they’ve been around for awhile. The original AOL chat rooms used sockets, as did instant messaging apps. Socket’s allow servers to “push” data into listening clients. This is why you don’t have to click a refresh button to see if your friend ever responded. The response just appears because it’s pushed in.
For this reason, I wanted to explore the idea of a socket-served web map. I thought of it as being a chat-like interaction. The map says “hey I’m over here!” Then the server gets any relevant data and just pushes it right into the map. Most of the get/access-level optimizations I described above apply equally here. However a win in user experience *may* come from the fact that each individual feature—much like the basemap tiles—can be drawn as it becomes available to the client. This removes a huge choke point imposed by the HTTP protocol, specifically waiting for the browser to receive the entire payload of data, parse it, and render it—all before you see the first feature. Instead, features are drawn one-by-one.
My theory feels good, and the approach seems to work. At the very least it’s doing what I hoped. But I have yet to test in a deployed scenario. Right now it’s all on my development machine. But it looks pretty cool in action! check it out below.
If anyone is interested, I’ll share the code. Otherwise, I’ll make it available after I get it a little further along and prove to myself it works well once deployed.
..to be continued
– elrobis
[…] Internet mapping, anyone? http://cartometric.com/blog/2015/03/17/socket-served-internet-mapping-anyone/ Share this:ShareEmailShare on Tumblr Posted on March 19, 2015Author […]
Socket-served Internet mapping, anyone? | Geo-How-To News
19 Mar 15 at 7:41 am
Man this looks great! But I can’t help but notice when you pan the map everything reloads from scratch even stuff that was already loaded and have stayed on the screen through the panning. Would it be a good idea to cache or not reload those parts that stay on the screen?
Seth
20 Jun 15 at 6:36 pm
Hey Seth! You’re absolutely right. It would actually save a ton of redundant loading and make the user experience much more streamlined. The little demo I put together is pretty simple and immature. Properly evolved, the client would need to keep a list of all the unique geometry ids currently in the map, then it would need to send those to the server gateway when the layer is refreshed. Then, when the map is panned or zoomed, the server should request all the geometries for the map’s current bounding area and return only those geometries not in the client’s list. Very likely, the client will also need a routine to clear away any geometry objects sufficiently out of sight, because otherwise, the client performance will start to really suffer.
elrobis
19 Oct 15 at 1:39 pm
I was wondering if you ever got the equidistant azimuthal projection map issue figured out? i was trying to setup an ISS tracker on that projection but I haven’t found a map that uses this or allows it to be easily setup. If you have an idea let me know.
Terrence
26 Sep 15 at 1:59 pm
Terrence, hi. Sorry for the slow reply. I’m assuming you mean the azimuthal projection stuff I was tinkering with in this question at GIS.SE? I did get it working for my use case, but I think it was pretty different than what you are doing.
If I understand, it sounds like you need an auto-scrolling basemap, where the ISS position is constantly at the map’s center. Am I correct? If so, I think that will be tricky to develop (though not impossible), largely because the space station follows some kind of sinuous, irregular orbital path. In other words I think every map refresh would need a unique tile/tileset generated for it.
One way to do this could be to use Mapnik to render your basemap tiles, on demand. Another way could be to render your basemap features using PostGIS geometry reprojection (relative to your azimuthal every time the map needed to refresh) coupled with Leaflet client-side rendering.
I did a very brief Google search, and I didn’t find a simple answer regarding on-demand azimuthal renderings for something like Geoserver, which ideally, could serve your azimuthal tiles as a WMS endpoint; however, if you look harder than I did, you might find a way to achieve that. The main issue is, defining a custom projection in Geoserver apparently requires restarting it, yet obviously you can’t be restarting it for every map load, so that approach becomes unrealistic.
As for realistic, Mapnik requires an XML style sheet defining your map and its layer traits—including the projection, in your case a centered azimuthal projection. So you could arguably create a Python or PHP endpoint that accepts a parameter defining your map center, and for every map refresh, dynamically submit the map style XML with the updated center to Mapnik to render your tiles on demand. The obvious downsides I foresee is the basemap tiles may load a little slowly compared to other services you’re familiar with, and you will not want to store the tiles very long—in fact, you should probably just overwrite the tiles on every subsequent refresh.
The PostGIS/Leaflet approach I described might be simpler to stand up, however, I think your basemaps will be somewhat limited by the variety of data you could show, because you’ll have to push all the feature geometries over the wire, and that can start to add up really quickly with all the vertices comprising squiggly country borders, etc.
Honestly, you have a pretty complex map idea.
Then again you could always do what everyone else does and just show the ISS’s lat/long position relative to a Web Mercator (EPSG:3857) basemap. But perhaps you have other design goals for your map that setup would not accommodate.
If it were me, and assuming you want relatively rich basemap tiles ..I would experiment with a Python web server and tie in Mapnik, and dynamically modify the map center in your style XML with REST url parameters, and serve basemap tiles on demand. However, if you do not need a rich basemap, I’d experiment with leaving tiles out of it, and try the PostGIS/Leaflet approach I described, and have the server/client do the work of reprojecting and rendering the geometries.
elrobis
19 Oct 15 at 4:00 pm
I think there’s a lot of scope to use web sockets to serve vector map data, especially if the data is real-time. I’ve just completed a project standing up a websocket server that serves real-time lightning strikes to clients viewing a traditional web map (like, OL2 traditional). It serves potentially hundreds of GeoJSON features per second, and the client side code to add the real-time strikes is so minimal. Even after 6 hours of accumulated lightning strikes on the map, we do not notice a performance problem in the client. It even has websockets at the other end: the server itself listens to several websockets (serving lightning in ad hoc formats, yuck) and accumulates them before serving them out to its own clients.
I’m now going to set up even more websocket servers serving GeoJSON for different pieces of information. I’m going to make a QGIS plugin for such servers, too.
Richard Law
8 Feb 16 at 5:15 pm