Elijah Robison | GIS Blog

I may or may not be on to something.

For awhile I’ve been growing more and more tired of the typical Internet mapping user experience. It’s basically this: open a map and zoom/pan somewhere, wait a moment while (hopefully) some “working” graphic gives you a clue that the map hasn’t forgot about you; meanwhile, some basemap tiles load, then after a moment or two, “poof!” all the interactive data (i.e. the sole reason you are using that particular map) appears all at once.

Fortunately, in most cases, the basemap tiles “sprinkle in” one by one as they become available, which does give the user an impression that something is happening. But more often than not, basemap tiles dramatically out-perform the vector data component, and the overall feel of the map becomes this kind of “clicky kludgy .. ..CLUNK”, as the tiles pepper in, followed by the hammer that is all the vector data landing at once.

So I’ve always looked for ways to streamline the vector IO as much as possible–1) Maybe it’s possible to simplify the data a little, removing redundant vertices? 2) Maybe it’s possible to return that data with less coordinate precision (i.e. the number of decimal places in the coordinate values), because a precision above 8 probably isn’t even legit. 3) Is the database using it’s geometry index, or is it possible to better-tune the DB so that it handles the Data Access step more athletically? 4) ..what about pre-cacheing the entire data table into RAM so the DB never has to hit disk to serve a query on that table? ….that’s pretty much the request/access step, but what about 5) returning the data in the leanest possible format–for example, GML is much heavier than GeoJSON. Or, 7) what about returning a home-rolled JSON response using Well Known Text (WKT) and just a few important fields, rather than providing the entire record for every feature?

I feel like I’ve tried everything I know to try, and I just cant serve a deep table of polygons (specifically, parcel polygons) to go through all the motions as fast as I’d like. At the end of the day, the choke-point is the HTTP protocol, itself, and the very client <–> server paradigm that represents almost 100% of the static web. (By static web, I mean basic websites where you go to fetch assets, read/learn things, watch things, etc. The Dynamic web would be things like video games, chat rooms, Skype, and stuff like that.)

That got me to thinking, maybe an answer lives somewhere in the Dynamic web. A major service-level component of the dynamic web is the “Socket”, and they’ve been around for awhile. The original AOL chat rooms used sockets, as did instant messaging apps. Socket’s allow servers to “push” data into listening clients. This is why you don’t have to click a refresh button to see if your friend ever responded. The response just appears because it’s pushed in.

For this reason, I wanted to explore the idea of a socket-served web map. I thought of it as being a chat-like interaction. The  map says “hey I’m over here!” Then the server gets any relevant data and just pushes it right into the map. Most of the get/access-level optimizations I described above apply equally here. However a win in user experience *may* come from the fact that each individual feature—much like the basemap tiles—can be drawn as it becomes available to the client. This removes a huge choke point imposed by the HTTP protocol, specifically waiting for the browser to receive the entire payload of data, parse it, and render it—all before you see the first feature. Instead, features are drawn one-by-one.

My theory feels good, and the approach seems to work. At the very least it’s doing what I hoped. But I have yet to test in a deployed scenario. Right now it’s all on my development machine. But it looks pretty cool in action! check it out below.

If anyone is interested, I’ll share the code. Otherwise, I’ll make it available after I get it a little further along and prove to myself it works well once deployed.

..to be continued

– elrobis

Today my friend (shout out to Nic Zamora) wanted to know if I still had a recording we made of a song he co-wrote way back in ancient history ..like 2007 or something. Gah..

Well it got me to thinking—my other buddy (shout out to Corey Nolan) had a similar request awhile back for a recording we did with his brother, Dustin, around the same time. The problem was, the only recording I had of the latter was on CD. Since then, that one simple deterrent—being on CD—was solely responsible for preventing me from ever “getting around to it”, like I said I would.

The reason I hate ripping CDs is because it’s such a ridiculous process ..first you have to open some asinine software (and God help you if it’s Windows Media Player), click around like a fool, be confused, drag files and/or locate folders, agonize over a variety of settings, many of which you, A) may not know anything about, 2) remember from the last time you did this resulting in inconsistently ripped music, and D) ..worst of all, potentially but unintentionally introduce some weak-sauce copy protection.

There had to be a better way.

Caveat—since writing this post, I’ve discovered that CDs containing extra material, like music videos, etc., unfortunately don’t play well with this program. I may return and solve that problem in the future, but with only one CD affected so far, it’s not a high priority.

As a GIS software developer, I use ALOT of open source tools, especially when I need to bend data out of one format and into another—usually from flat files like shapefile into some database instance, like PostgreSQL or MySQL. The best tool for this, of course, is the ogr2ogr command line utility in the GDAL/OGR suite. It’s the best for many reasons. But in this context, it’s the best because everything you want to do is done over the command line. No clicking, no dragging. No BS. Just type the “instruction” you want to execute into a command line window, run it, and you’re done. Simple.

So this morning I had enough and finally asked the obvious question—why can’t I convert music over the command line as easily as I can bend and twist GIS data formats??

It turns out, I can! Using the VLC Media Player, which provides a command line interface..

First, the Google took me to where someone already asked a similar question on Stack Overflow. And after toying around with a batch file provided by a helpful SO user, I tweaked it a little so that I could reuse the file more easily—specifically by passing in the path to any folder where I wanted new mp3 tracks to be saved into.

And its wicked simple. In the time I’ve spent so far writing this post, I’ve already used this solution to rip seven different CDs—Chopper One, ACDC live, Social Distortion, DGC Rarities, Robert Bradley’s Blackwater Surprise, Nerf Herder, and ..yep, the song my friend asked for. 🙂

Oh, and I’ll probably have 8 or 10 CD’s ripped by the time I publish. I love open source software.. LOVE it.

Now that I’m editing the post a little while later—make that 14 albums exported ..going on 15. 8)

So without additional storytelling, here’s what you have to do to rip your CDs to MP3 like a GIS jedi..

1. Install VLC.

When VLC installs, make a note of the directory VLC installs to. It should be something like

C:\Program Files\VideoLAN\.. or maybe

C:\Programs\VideoLAN\..

You’ll need this later in Step 5.

 

2. Create some directory near your drive root that DOES NOT HAVE SPACES in it. Like C:\MP3s

*Note: From here on, let’s assume you won’t put any spaces in any file paths.*

 

3. Open Notepad. Start > Run > type “notepad” without quotes > ::hit Enter::

 

4. Copy and paste the code between these lines into your notepad window..

 

---

@ECHO OFF

setlocal ENABLEDELAYEDEXPANSION

set out_path=%1

:: IMPORTANT!!!!
:: Set the path to the VLC installation on your system.
:: Make sure you see the file "vlc.exe"
::
set vlc_path=C:\Program Files\VideoLAN\VLC\vlc

:: Add a trailing backslash if necessary..
if defined out_path if not "%out_path:~-1%"=="\" set out_path=%out_path%\

:: Create the output directory, if necessary..
if not exist %out_path% (
echo Creating output directory..
mkdir %out_path%
) else (
echo Output directory already exists, reusing it..
)

echo Exporting to "%out_path%" ..

SET /a x=0

:: IMPORTANT!!!!
:: E: represents your CD player's letter drive. You may need to change this to D:
FOR /R E:\ %%G IN (*.cda) DO (CALL :SUB_VLC "%%G")
GOTO :eof

:SUB_VLC
call SET /a x=x+1
ECHO.
ECHO Transcoding %1
SET out_track=%out_path%Track!x!.mp3
ECHO Output to "%out_track%"
ECHO exporting .. ..

:: Here's where the actual transcoding/conversion happens. The next line
:: fires off a command to VLC.exe with the relevant arguments:
:: IMPORTANT!!!!
:: E: represents your CD player's letter drive. You may need to change this to D:
CALL "%vlc_path%" -I http cdda:///E:/ --cdda-track=!x! :sout=#transcode{vcodec=none,acodec=mp3,ab=128,channels=2,samplerate=44100}:std{access="file",mux=raw,dst="%out_track%"} vlc://quit

:eof

---

 

5. Depending on your version of Windows, you may need to edit the path to your VLC installation.

Look for “IMPORTANT!!!!” in the code you just pasted. This is where you set the path to VLC.

Back in Step 1 I asked you to make note of the VLC installation path. My home system runs Windows XP 32 bit (yes, still), which means VLC installed to..

C:\Program Files\VideoLAN\VLC\vlc

—see where that path is declared in the file? YOU, however, are probably running Windows 7 ..or maybe 8, and if that’s the case, I think you need to change the VLC path to..

C:\Programs\VideoLAN\VLC\vlc

..which you should confirm before you continue. Try opening or browsing into that directory and make sure you see the file, vlc.exe. If you see that file, you’re good to advance.

 

6. Similarly, you may need to edit the code to use the correct letter drive for your CD player. On my system, it’s the E: drive, but on your system, it’s probably the D: drive.. So if changing this value is necessary, find the two remaining “IMPORTANT!!!!” comments in the code and change E: to the letter drive for your CD player.

 

7. Save the file as a so-called Batch file (.bat).

In Notepad, select File > Save as..

Make sure you change “Save as type” to “All Files”. Then browse into the C:\MP3s folder you created in Step 2 and save the file as “_rip-cd.bat”, which starts with an underscore, “_”.

Frankly, you can name the file whatever you want, but the underscore should cause the file to always appear above or below anything else in this folder, which I find helpful.

Make sure that the file REALLY ends in the .bat extension, and not .txt. If this is correct, your file icon should look like a little gear, like in the image above. If you’re having trouble with this, look here for some help.

 

8. Now it’s time to put your hard work to use, so pop in a CD and open a fresh command window (a.k.a. the terminal).

Start > Run > type “cmd” without quotes > ::hit Enter::

In the terminal, type the full path to your batch file. If you stuck to the example, that would be..

C:\MP3s\_rip-cd.bat

..follow it with a space, and then a path on your system to where you want to save the exported MP3s. So your full command might look like below. (FYI: “TREoS” is a band, and “Between the Heart and the Synapse” is an album.)

C:\MP3s\_rip-cd.bat C:\MP3s\TREoS\BetweenTheHeartAndTheSynapse

You do NOT have to create the export folder first, the script will create it for you automatically.

..in other words, if you have multiple albums by the same artist, you’ll want to save them in different folders. This is important, because the program can’t name the songs for you, so they’ll be created with names like Track1.mp3, Track2.mp3, etc., and if you try to export another album into the same directory, it will probably fail and complain when it tries to save another Track1.mp3 into a folder already containing a Track1.mp3. Hopefully this makes sense.

Here’s what it looks like when you fire it up and run it. Just so you know what you’re seeing, the part I actually typed was

D:\x_Rip\_rip-cd.bat D:\x_Rip\STP\Purple

(yeah ..I’m using different drive letters and a different folder path, but hopefully you get the gist.)

That’s all there is to it. Using this approach, it’s ultra-painless to export a CD to MP3s. As a bonus, you’ll export the same way every time, never having to worry about required or optional configurations, or some pesky copyright protection that some software try to impose.

Cheers!

elrobis

Here’s a solid article on tuning your PostgreSQL instance and optimizing your indexes.

Also, here’s a nice primer on creating and managing indexes in PostgreSQL.

I wanted to share ’em now and save ’em for later.

/E

Just updated my LinkedIn profile.. This is just TOO FUNNY right now!

I totally had to save this for posterity. Unfortunately it will probably disappear before you can say “Jack Robinson”. 8)

While this post is not in-theme with the stated goals of the blog, I wanted to save my notes somewhere as well as make a URL to show-off for my friends. On the other hand, I’m opimistic my experiment will enable some “eco-minded research”, so perhaps there’s still a loose tether linking this back to the GIS community..

Basically, I hacked my old, 3 megapixel Nikon Coolpix 3100 today so that I could take NIR (near infrared) pictures. The following images serve as a storyboard for the process.

0:  The tools and materials involved include (clockwise from bottom-left)
—-> small round-nose pliers (my wife uses these for beading)
—-> small phillips head screwdiver
—-> butt-end of unprocessed camera film
—-> small curve-nosed pliers
—-> scissors
—-> electrical tape
—-> the Nikon Coolpix 3100 (the pic is post-hack)
—-> one depleted roll of hockey tape*

* The expired roll of hockey tape became the “housing” for a DIY visible bands filter, which you see fitted to the front of the camera. By some amazing coincidence the cardboard tubing holding the tape fit the camera perfectly. The “donut hole” was just the right diameter to cling to the face of the camera; plus, it was just deep-enough to stay clear of the lens when in use.

 

 

1:  Looking at the guts –I had to pull 22 screws to reach the NIR filter and CCD mount. At this point, I was relatively convinced the sucker was dead meat. The IR filter was just a piece of pinky-nail-sized-glass clinging to a rubber footing. I was able to gently tease the filter from the footing using my index finger.

Caveat: If you try a similar project, I recommend going to the trouble to illustrate or take notes documenting the locations of screws as you remove them. While my camera still worked without any screws being replaced (no kidding), once I was finished experimenting, I wasn’t able to replace 7 of 22 screws into the camera assembly. There wasn’t a functional consequence in my case, but I hate knowing that I didn’t get it 100% back together.

 

2: After removing the NIR filter, I snapped the camera back together (using ZERO of the 22 screws, expecting to crack it open again shortly) and was fully amazed when it powered-on following surgery. Next, I fired-up a burner on the stove and photographed the hot burner across from its cool neighbor. At this point, without the NIR filter the rear-burner coils appear bright-pink, which I expected, but the distinction between the active burner and the inactive burner was not as pronounced as I hoped. So I decided to incorporate another trick revealed by my research.. 

Note: The coils were not visibly red; that is, I could not distinguish the hot burner from the cool burner by sight alone at the time of the exposure.

 

3: One of the related articles I researched provided the idea of making a DIY Visible Bands Filter out of a slice of unprocessed camera film. So I tried it. I found a nearly-depleted roll of hockey tape to serve as the filter housing. Then I “sharpied” the insides of the tape’s cardboard tubing and used elecrical tape to attach a choice cut of unprocessed camera film. As you can see, the end result fit the camera very nicely —I can’t exaggerate how perfectly the depleted roll of hockey tape fit the camera exterior. Would it work? 

Regarding the cut of film, you don’t want to use the see-through, sepia-toned portion with off-color images, as those are processed negative frames. Instead, you want the black, unprocessed portion. I found a good piece  with plenty of slack at the butt-end of some old negatives.

 

4: With my visible bands filter attached and an enthused state-of-being, I returned to the stove for another experiment. Once again I fired up the rear burner, waited for it to radiate a reasonable amount of heat, then took another pic. As you can see, the hot burner is about the only thing visible in the photo.

Our stove is white, so clearly my visible bands filter leaks some light into the exposure. However, note how dark/black the forward burner is. Once again, there was no difference between these burners at the time of exposure (at least, as a human can see). This was the sort of result I was hoping to get.

 

Blurry Images: In my Nikon Coolpix 3100, it seems the stock NIR Filter pulls double-duty, behaving not only as a spectral sieve, but also providing some optical focus. James Wooten’s blog post provides some insight into this issue noting that without the NIR Filter/Glass..

..the camera will be “near-sighted” without it. I used a optical glass that has about 85% transmission in the IR and visible range. It helps if you have a friend that cuts glass for a living.

 I happen to have a specific goal in mind for my NIR camera, so I’m not immediately concerned about this shortcoming; however, if I want to take photos with a reasonable amount of visual acuity, I’ll have to resolve this issue at some point. When and if I get around to that, I’ll return to this post and update my notes.

Related Pages: Each of the following sites povided some form of inspiration or insight, so I wanted to log them along with the article–

http://4photos.de/camera-diy/index-en.html: Many, many links devoted to camera repair, modification, and hackery..

http://www.lifepixel.com/tutorials/infrared-diy-tutorials/nikon-coolpix-5400: These people will do the camera hack for you if you’re willing to part with some money (my Nikon would’ve cost about $200). However, I’m wondering if LifePixel may sell the clear optical lens I need to fix the blurred images problem? Theirs is the first site I’ll check when I decide to cross that bridge. This URL goes to their tutorial for hacking a similar Coolpix model, but they had several such tutorials, so if you’re considering your own project, I recommend perusing their site for a tut describing the hack on a camera model near to the one in your crosshairs.

http://www.geektechnique.org/projectlab/254/how-to-turn-a-digital-camera-into-an-ir-camera.html: Mark Hoekstra does a nice job of presenting his camera hack using several step-by-step images. Read his site in full before you jump head-first into your own poject. In the end, I felt my project required going to a similar “component depth”.

http://www.abe.msstate.edu/~jwooten/camera/lense.html: The original James Wooten post, cited in the body of my article.

http://www.echeng.com/photo/infrared/faq/
http://www.echeng.com/photo/infrared/wooten/
http://www.echeng.com/photo/infrared/hot_mirror_replacement.gif
http://www.echeng.com/photo/infrared/wooten/IR_Block.GIF: Eric Cheng maintains an interesting site; it was actually through his page that I ran across the Wooten article. Eric did the research and posted some details related to the much-needed replacement lens, which I appreciated.

http://www.diyphotography.net/take_infrared_pictures_with_digital_camera_ir_filter: This post inspired my implementation of the DIY visible bands filter. They do a swell job of documenting the materials and steps involved in making the filter (this is if you don’t happen to have a used roll of hockey tape langushing somewhere in your gear bag).

http://www.jollinger.com/photo/articles/cheap_ir.html: One of the first posts I ran across; however, James’ camera was quite different from mine, so most of the post wasn’t applicable. But, it was open in my tabs when I embarked on this venture, so I’ll cite it anyway.

Home Energy Audit: So, why would I make an NIR camera if my chief interest isn’t taking unique and stunning photographs? Well I hope can photograph my house from all sides to learn where the structure leaks the most heat. On the other hand, NIR isn’t exactly heat, so this may not correlate in practice, but it’s worth a shot if only for the fun of trying. My wife and I have been thinking about replacing some of our windows, so if we can’t afford to replace all our windows, I thought perhaps we could zero-in on those windows that are costing us the most money. I don’t know if this will work but, once I think it’s cold-enough to try it, I’ll take some pics of the house in all its raging un-eco glory and post them here in the article.

🙂

/Elijah

“When life gives you lemons..”

Let’s just review the signs:

• First, the Google Maps Flash/Flex API had not been updated for quite some time.
• Meanwhile, some developers were sharing a hack to bypass a nasty initialization bottleneck in the Maps Flash/Flex API that stalled map loading in AIR apps.
• At the other end of the yard, Google lifted its registration key requirement for the Maps JavaScript API V3; yet, the Flash/Flex counterpart remained unchanged in this respect (curiously, it was key-hashing that caused AIR apps to stall).
• Finally, and though unrelated, Pamela Fox, one of the higher-profile support engineers attached to Maps developer relations, left Google for other pursuits.

In spite of the above indicators, I started an eBook titled: Google Mapping the Shapefile with Flex, PHP, and PostGIS. After four or five chapters, I threw in the towel to enjoy what was left of the summer ..but, maybe that was a good thing, because lo and behold, the Maps Flash/Flex API was a dead man walking. In their defense, Google saw comparatively less consumption of the Flash/Flex API vs the JavaScript flagship, so they pulled the plug on the Flash stuff. It’s an understandable reaction ..but I do wonder if that inertia will eventually affect StreetView, which (I believe) remains in Flash.

I had planned to return to my book over the winter, and I was optimistic about re-approaching O’Reilly with the finished book and a suggestion that it be released as a print title. So clearly I was disappointed by the deprecation announcement. Also, I really like Flex! If I want to make an interactive web map, I’d rather do it with Flex! Where does this leave me?

“..make lemonade.”

That was lemons. But it turns out, there’s a chance to make lemonade. It’s called OpenScales, and it’s a Flex mapping API with some real potential. While investigating OpenScales, I had some difficulty finding examples of raw implementations, where the Flex client UI has a direct line (PHP) to vector GIS data stored on a PostGIS-enabled PostGREsql database. Fortunately it wasn’t difficult to setup. So, rising from the ashes of my failed eBook, and primed by the excitement of finding OpenScales, I decided to start this blog as a place to post OpenScales examples. And that’s exactly what I’ll do.

In the next few months, I’ll post several OpenScales examples here, with the goal of touching upon the following topics:

• Publish Shapefile or Personal Geodatabase data to an OpenScales map by converting it into usable flatfiles or migrating it into a spatially-aware database and pulling it from there.
• Load geodata into OpenScales from the following formats: XML, JSON, GML, and KML.
• Load data faster by enabling AMF support with AMFPHP, et cetera.
• Pull WMS and WFS data from 3rd-party sources (I’m imagining a weather map).
• Prepare custom basemap tiles for an OpenScales map and either load raster tiles direct from the filesystem or pull them from a PostGIS database.
• Publish OpenScales-based applications to Android devices.

If I can do all that in a handful of months, I’ll be a happy camper, and with a little luck, a few people will stumble across this site and they will be too.

Wild! Stallions!