About a 2 years ago, a client for the company I work for, needed map tiles for an application we were building for them. One requirement was that we cut the tiles from ArcGIS mapdocs that they created, so that they could control the order of their overlapping aerial images. We just needed to cut the tiles and move them to an S3 bucket so that they’d be available for our application. For this we decided to use ArcMap with Arc2Earth which can be configured to push tiles to S3 as they are cut.
This seemed to work great, however, we noticed well after the tiles were cut that a small number of tiles (less that 0.001%) were missing, while other tiles were solid white when they should have shown aerial imagery. We couldn’t be certain of the cause; data issues, software problems, maybe network connectivity, and there was no clear pattern in the invalid or missing tiles.
So, I starting writing some Python to check for rogue tiles. Because of the very large number of tiles, we defined a set of sample points and used those coordinates to retrieve tiles from S3. If a tile was not missing, we checked whether could be opened by Python’s PIL library, and collected descriptive stats about the distribution of pixel values. The results were written to SQL files, loaded into Postgres/PostGIS and viewed in QGIS. This allowed use to identify particular regions where there were tile issues and then re-cut tiles just in those areas. The process ended up saving quite a bit of time and provided a lot more confidence in our tile cache.
Last weekend I decided to write something similar – but this time using NodeJS rather than Python. I’ve worked on a couple of Express projects, but I wanted to get some experience writing Node CLI tools. So, I wrote a very simple version of maptile-qc and pushed it up to github. It’s a very early version, but I hope to have something useful completed soon.