Search This Blog

Wednesday, January 29, 2020

Importing .shp (shape) GIS files into mySQL db

Steps to import SHP files exported from GIS systems, into mySQL db:

To import GIS data into mySQL, we need to translate the geometry GIS descriptions into sql statements.
This can be achieved by using the opensource shp2mysql utility, along with cygwin1.dll (both in the same folter)

After generating the .sql file, all is left to do is to import it into mysql. I did it using sqlyog: right click in the target database → Import → Execute SQL Script → select the .sql file generated above.

But before doing that, you might do the following:

The table name where the data will be imported is derived from the sql output file name. Make sure you rename it in the Create Table statement, accordingly.

I had to manually fix the .sql file, to overcome the following issues:

  1. shp2mysql will create two columns named ID. I renamed the 2nd one to _ID
  2. my shp2mysql execution generates a 3D representation of each point, and I did not find a 3D syntax for MULTILINESTRING. I ended up replacing all 0.000000000000000 occurrences in the sql file to blank spaces, to fix it.
  3. the generated sql file had a -1 SRS id for each generated gis representation in WKT. mySql stated -1 is invalid, we checked which SRS id was used in our other GIS dbs, figured out it was 0, then I replaced all occurrences of ,-1) with ,0) in my sql file.
  4. shp2mysql will generate the sql file using a deprecated mysql function, GeometryFromText. According to mysql documentation, it was deprecated in mySql 5.7.6. Since my machine has mySql 8, I renamed all occurrences of it to ST_GeomFromText:

GeomFromText() and GeometryFromText() are deprecated as of MySQL 5.7.6 and will be removed in a future MySQL release. Use ST_GeomFromText() and ST_GeometryFromText() instead.

Now the file is valid and free of errors, and you can proceed and run the Execute SQL Script described above.

Sunday, December 15, 2019

Create circle with google map api with mousemove event to enlarge / shrink area

function addCircle_new(lat, lng, radius, alreadyDrawn) {
    function createCircle(lat, lng, radius) {

        var coordinates = new google.maps.LatLng(lat, lng);

        var options = {
            strokeColor: '#910000',
            strokeOpacity: 0.5,
            strokeWeight: 1,
            fillColor: '#FC7341',
            fillOpacity: 0.35,
            map: map,
            center: coordinates,
            radius: radius
        };
        // Add the circle for this city to the map.
        circle = new google.maps.Circle(options);
        return circle;
    }

    var circle;

    if (alreadyDrawn) {
        circle = createCircle(lat, lng, radius * 1000);
        drawObject = circle;
        pois.push(circle);
    }
    else {
        //google.maps.event.removeListener(clickListener);
        google.maps.event.addListenerOnce(map, 'click', function (x) {

            //circle = createCircle(x.latLng.lat(), x.latLng.lng(), radius * 1000);
            circle = createCircle(lat, lng, radius * 1000);

            var clickEventParameters = x;
            var mouseMoveEventParameters;

            var mouseMove = google.maps.event.addListener(map, 'mousemove', function (y) {
                mouseMoveEventParameters = y;

                var xOffset = mouseMoveEventParameters.qa.x - clickEventParameters.qa.x;
                var yOffset = mouseMoveEventParameters.qa.y - clickEventParameters.qa.y;
                var radiusOffset = Math.sqrt(Math.pow(xOffset, 2) + Math.pow(yOffset, 2)) * 100000;

                circle.setRadius(radiusOffset);

            });
            var dblClick = google.maps.event.addListener(map, 'dblclick', function (w) {
                drawObject = circle;
                google.maps.event.removeListener(mouseMove);
                pois.push(circle);

                //clickListener = google.maps.event.addListener(map, 'click', map_MouseClick);
            });
        });

    }
}

Monday, December 9, 2019

Elasticsearch aggregation - show aggregations only + overcome default limitation of aggregated buckets

Elasticsearch aggregation - output showing only aggregated field + overcoming default limitation of aggregated values (in this case, set to 1000)

GET customer_success_reports/_search
{
  "size":0,
    "aggs" : {
        "Country": {
            "composite" : {
              "size": 1000,
                "sources" : [
                    { "country": { "terms" : { "field": "Meta.GeoInfo.Country.keyword","missing_bucket": true } } }
                ]
            }
        }
     }
}


size:0 -> makes the output show the aggregated buckets only instead of the whole documents

Sunday, November 3, 2019

Fixing date field formats to upgrade elastic to version 7

Here is what did, that eventually worked:


  1. The only way I found to EDIT the field format was by creating a new (empty) index, by specifying its mapping. The date fields I replaced (had my json in a text editor and did a "replace all" on each different format to a elastic-7-complying one. In my case, most of my fields were "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd||epoch_millis", and I changed them to "8yyyy/MM/dd HH:mm:ss||8yyyy/MM/dd".
  2. Then I issued a _reindex, meaning I copied the contents of each index I had to fix to its corresponding new empty index I created, in the step above, to hold its data, now in a compatible format
  3. At the end of the reindex (do not worry about the "backend connection closed" message, the copy keeps running, check the "Reload Indices" in Index Management), I added an alias to the new index so that new documents being added to the original one are now inserted in the new one (using the _alias api)
  4. If you want the indexes to remain with their original names at the end, you can then copy once again the indexes to a new one, using the original name this time. Be aware that before doing that you must remove the alias you added in step (3), other wise the engine thinks the index already exists (because references to the alias translate into references to the new index).

CAUTION: My dashboards stopped working. I have an open ticket with Elastic's support at the moment waiting for instructions.

At this stage, after doing the above to all indexes that had warnings on date formats, you should reach the stage where you have no warnings left, and are ready for the upgrade.