Communauté

My GIS performance impact of Meltdown and Spectre patches

Geomatips - mar, 01/09/2018 - 19:34
When I saw performance regression few days ago of Meltdown and Spectre patch from cpu conception problem, I would like to perform some basic test to understand if it could slow down my process.

For those who don't know, here : a postgres article about performance regression : https://www.postgresql.org/message-id/20180102222354.qikjmf7dvnjgbkxe@alap3.anarazel.de 
Since some users finished to make benchmark with Meltdown and Spectre patch, I will not test it on my computer.

How to test  ?
First, I tested it on Archlinux based distribution named Manjaro Linux.


To know if patch is applied to my kernel, I just run this command :

$ cat /proc/cpuinfo | grep cpu_insecureWhich reports this :

bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecure
bugs        : cpu_insecureWhen showing cpu_insecure, you know that KPTI patch is enable.

Then there is a specific C code to test your cpu : 

    #include
    #include
    #include
    #include
    // gcc -Wall -O3 -o getpid getpid.c
    int main(void)
    {
    long value;
    clock_t begin = clock();  
    for (int i = 0; i < 100000000; i++)
     value = syscall(SYS_getpid);
    clock_t end = clock();
    double time_spent = (double)(end - begin) / CLOCKS_PER_SEC;
    printf("%d\n", (int) value);
    printf("%f\n", time_spent);
    return 0;
    }


Write these lines to pti.c file, build it with :

gcc -Wall -O3 -o pti pti.c
and finally run it with
./ptiFor my cpu aka Intel i7 4790k, switching to patched kernel, my computer slow down a lot from this generic test. When upgrading to pacthed kernel, you can still disable patch with kernel grub option pti=off. Next tests will be shown with pti=on and pti=off

With pti=off and generic test, my results could be shown below :


1211
3.571401 seconds With pti=on and generic, we could watch a several regression :
1208
13.998673 seconds Now to know when rebooting my system, I know if patch is apply or not and before making tests, I will build and run this C code.
Simple GIS test : 
From years now and unfortunately, I fight with ECW files. I have some big ecw which I found far away better in performance in comparison to multiple files like 36GB.
But for this test I will use a moderate ECW raster which is an orthophotography of Mauritius Island with 15cm resolution pixel and size of this file is 15 GB and cover 56 x 67 kilometers. Sample overview :

For those are curious here is in detail :

$ gdalinfo ALL.ecw
Driver: ECW/ERDAS Compressed Wavelets (SDK 3.x)
Files: ALL.ecw
       ALL.ecw.aux.xml
Size is 372000, 450000
Coordinate System is:
PROJCS["SUTM40",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.257223563,
                AUTHORITY["EPSG","7030"]],
            TOWGS84[0,0,0,0,0,0,0],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9108"]],
        AXIS["Lat",NORTH],
        AXIS["Long",EAST],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",57],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",10000000],
    UNIT["Meter",1]]
Origin = (531000.000000000000000,7796700.000000000000000)
Pixel Size = (0.149999999999993,-0.149999999999993)
Metadata:
  COLORSPACE=RGB
  COMPRESSION_RATE_TARGET=4
  VERSION=2
Corner Coordinates:
Upper Left  (  531000.000, 7796700.000) ( 57d17'46.37"E, 19d55'32.96"S)
Lower Left  (  531000.000, 7729200.000) ( 57d17'50.53"E, 20d32' 8.78"S)
Upper Right (  586800.000, 7796700.000) ( 57d49'45.72"E, 19d55'26.88"S)
Lower Right (  586800.000, 7729200.000) ( 57d49'57.39"E, 20d32' 2.50"S)
Center      (  558900.000, 7762950.000) ( 57d33'49.99"E, 20d13'48.53"S)
Band 1 Block=256x256 Type=Byte, ColorInterp=Red
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=59.587, StdDev=56.411
  Overviews: 186000x225000, 93000x112500, 46500x56250, 23250x28125, 11625x14062, 5812x7031, 2906x3515, 1453x1757, 726x878, 363x439, 181x219
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=59.586901452342
    STATISTICS_MINIMUM=0
    STATISTICS_STDDEV=56.410809243179
Band 2 Block=256x256 Type=Byte, ColorInterp=Green
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=57.965, StdDev=53.573
  Overviews: 186000x225000, 93000x112500, 46500x56250, 23250x28125, 11625x14062, 5812x7031, 2906x3515, 1453x1757, 726x878, 363x439, 181x219
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=57.964855468501
    STATISTICS_MINIMUM=0
    STATISTICS_STDDEV=53.572679192972
Band 3 Block=256x256 Type=Byte, ColorInterp=Blue
  Min=0.000 Max=255.000
  Minimum=0.000, Maximum=255.000, Mean=47.609, StdDev=46.083
  Overviews: 186000x225000, 93000x112500, 46500x56250, 23250x28125, 11625x14062, 5812x7031, 2906x3515, 1453x1757, 726x878, 363x439, 181x219
  Metadata:
    STATISTICS_MAXIMUM=255
    STATISTICS_MEAN=47.608960074074
    STATISTICS_MINIMUM=0
    STATISTICS_STDDEV=46.082615455709


Idea is to create a pretty well compress geotiff (and a good way to switch from ECW to geotiff since Paul Ramsey make a great tutorial about it and it's now embeded in Qgis raster menu ! ) to understand if these kernel patches have influences over a standard GIS tranformation.
I use gdal_translate with time command in a bash terminal to benchmark it. Command line is like that :

$ time gdal_translate -co BIGTIFF=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -co TILED=YES -co NUM_THREADS=8 in.ecw out.tif With pti=off I've got this result : 


Input file size is 372000, 450000
0...10...20...30...40...50...60...70...80...90...100 - done.

real 113m36,193s
user 106m22,559s
sys 0m34,030s And with pti=on this :
Input file size is 372000, 450000
0...10...20...30...40...50...60...70...80...90...100 - done.

real 117m4,998s
user 107m28,128s
sys 0m44,773s
Results shiow +/- no difference which demonstrate there is no impact regression on a Gdal GIS usage. Time for this process is big due to usage of JPEG compression which is only single core in gdal. Test is perform on external usb 3.0 2.5" hard drive.I made some others tests with gdalwrap and multithreading option and also there is no significant differences ...CPU / GPU test with blender and generic blenchmark : 
Finally, I use my computer to render 3D files sometimes and I would like to test CPU and GPU in this test. May be not related but cpu manage (with a small amount) gpu during gpu computation.

With pti on or off I had +/- same results : 45/47s with dual GPU and 2min10 / 2min11




So applying and using patch is recommanded since they publish how to use this leaks but we clearly show that they want to sell new platforms with a great new performances (in fact same as before without KPTI patches).
Happy proofing and selling thing but in my case it doesn't matter except with my postgres / postgis database. Does postgres tests made by others with pgbench really perform impact of GIS usage ?!

Catégories: Technique

WFS 3.0 et autres petites nouvelles de l’OGC

NeoGeo-Online - mar, 12/19/2017 - 10:29
WFS 3.0

En parcourant rapidement un appel à participation au prochain testbed de l’OpenGeospatialConsortium, en vue de mieux savoir ce sur quoi cet organisme travaille actuellement, je me suis rendu compte qu’une version 3.0 de WFS était en cours de gestation.

Très peu d’informations semblent circuler sur ce sujet sur internet. J’ai essentiellement trouvé 2 choses intéressantes (en plus des informations présentes dans l’appel à participation) :

Ce que je retiens de ces sources d’information :

  • WFS 3.0 sera vraiment REST et, en cela, présente une vraie rupture ou sacrée évolution de WFS : utilisation d’OpenAPI, JSON et codes d’erreur HTTP standards au programme.
  • jusqu’ici, les standards de l’OGC mettaient en œuvre des schémas XML auxquels les requêtes et les réponses HTTP devaient se conformer. L’utilisation de tels schémas, ni même d’XML, ne semble pas requise pour mettre en œuvre WFS 3.0.
  • la question de l’authentification et de la sécurité des accès aux données est prise en compte par l’OGC : l’utilisation d’OAuth2 est citée à la fois au niveau du projet Github et du testbed.

 

Maps for HTML

Un projet « Maps for HTML » visant à disposer d’éléments permettant d’intégrer facilement une carte dans le contenu de pages HTML comme on peut déjà le faire avec d’autres contenus multimédia (vidéos, sons…). Je suis plutôt content de cette évolution car pour tout vous dire, lors d’une réunion plénière de l’OGC à Bruxelles en 2011 portant sur HTML5 et la géolocalisation (cf. ici), j’avais demandé à un représentant du W3C et d’Opera si le W3C avait l’intention d’intégrer ce genre de chose au standard HTML. La réaction de l’assistance m’a fait comprendre que ma demande devait être complètement déplacée. Je vois que je ne suis plus seul à espérer un élément <(geo)map> dans HTML et cela ne devrait plus tarder à voir le jour. Quelques liens sur ce sujet : The HTML <map> Element proposal, Customized Built-In <map> Element et Maps for HTML Community Group.

 

Orléans en mars

Les prochaines réunions du comité technique de l’OGC se tiendront à Orléans au cours de la semaine du 19 mars 2017 (cf. ici).

Catégories: Société

Une belle offre commerciale sur ReCap Pro

Geospatial France - mar, 11/14/2017 - 10:27
Je parle assez peu des offres commerciales Autodesk. Mais pour celle-ci, je ne suis pas certain que tout le monde en ait connaissance. A l’occasion de la sortie de ReCap Photo, une promotion...

{Cliquez sur le titre pour lire la suite...}

Batimat 2017 : le programme des présentations

Geospatial France - mar, 10/31/2017 - 17:37

Autodesk et ses partenaires seront présents sur Batimat 2017, au stand P94 du hall 5A.

Nous aurons un espace de démonstration avec un agenda très complet :

 

Pour ma part, je serai sur le salon lundi après-midi, jeudi et vendredi. J’espère vous y voir !

Détails pratiques :

Dates : du 06 au 10 Novembre 2017

Lieu : Parc Des Expositions Paris Nord Villepinte

Stand : Hall 5A -P94

Heures d’ouverture : Visiteurs de 9h à 19h

Plus de détails sur le site du salon.

Convergence SIG et BIM : une journée pour en savoir plus

Geospatial France - jeu, 09/28/2017 - 16:06

Le forum Français de l’OGC et la Commission de Normalisation de l’Information Géographique et Spatiale de l’AFNOR co-organisent  la 8e édition de la Journée Interopérabilité et Innovation.

Cette journée se tiendra à Marne-la-Vallée le 7 Novembre 2017, et le thème sera “Convergence GIS-BIM”.

J’y serai avec Vincent Fredon pour présenter un atelier nommé “Intégration GIS/BIM avec les solutions Autodesk”

L’agenda est disponible en ligne :

 

J’espère vous y croiser !

Pour vous inscrire, remplissez le formulaire sur le site de l’évènement  :

Additions to the MapFish Protocol

Eric Lemoine - sam, 04/18/2009 - 23:55

We recently added new stuff to the MapFish Protocol.

As a refresher, let’s first take a look at what the MapFish Protocol had before the new additions.

(Note that you’d need the JSONovich FireFox extension to see the output of the examples given below in your web browser.)

Geographic query params

  • box={x1},{y1},{x2},{y2}: the features within the specified bounding box
  • geometry={geojson_string}: the features within the specified geometry
  • lon={lon}&lat={lat}&tolerance={tol}: the features within the specified tolerance of the specified lon/lat

Examples:

Limiting and Sorting

  • limit={num}: the maximum number of features returned
  • offset={num}: the number of features to skip
  • order_by={field_name}: the name of the field to use to order the features
  • dir=ASC|DESC: the ordering direction

Examples:

The new params

  • no_geom=true|false: so that the returned feature has no geometry (“geometry”: null)
  • attrs={field1}[,{field2},...]: to restrict the list of properties returned in the features
  • queryable={field1}[,{field2},...]: the names of the feature fields that can be queried
  • {field}__{query_op}={value}: filter expression, field must be in the list of fields specified by queryable, query_op is one of “eq”, “ne”, “lt, “le”, “gt”, “ge”, “like”, “ilike”

And now an example combining all the new parameters:

The above query returns a GeoJSON representation of the summits whose names include “col” and whose elevations are greater than or equal to 3500. The returned features have no geometry and their attributes include “name” and “elevation” only.

Not including the geometry in the features makes the parsing in the browser much faster, so for cases where the geometries aren’t needed this is a big win.

Credits for the “queryable={field}&{field}__{query_op}={value}” syntax goes to FeatureServer!


Catégories: Technique

Secure TileCache With Pylons and Repoze

Eric Lemoine - dim, 02/15/2009 - 19:14

This post shows how one can secure TileCache with Pylons and Repoze.

In a Pylons application one can run a WSGI application from within a controller action. Here is a simple example:

class MainController(BaseController) def action(self, environ, start_response): return wsgiApp(environ, start_response)

TileCache is commonly run from within mod_python. TileCache can also be run as a WSGI application, therefore it can be run from within the controller action of a Pylons application. Here’s how:

from TileCache.Service import wsgiApp class MainController(BaseController) def tilecache(self, environ, start_response): return wsgiApp(environ, start_response)

Pretty cool… But it gets really interesting when repoze.what is added to the picture. For those who don’t know repoze.what, repoze.what is an authorization framework for WSGI applications. repoze.what now provides a Pylons plugin, making it easy to protect controllers and controller actions in a Pylons application. Here’s how our tilecache action can be protected:

from TileCache.Service import wsgiApp from repoze.what.predicates import has_permission from repoze.what.plugins.pylonshq import ActionProtector class MainController(BaseController) @ActionProtector(has_permission('tilecache')) def tilecache(self, environ, start_response): return wsgiApp(environ, start_response)

With the above, anyone who tries to access /tilecache will have to be granted the tilecache permission. Otherwise, authorization will be denied.

TileCache is secured!

People often want finer-grained authorization, like give certain users access to certain layers. With Pylons’ routing system this can be easily and elegantly achieved using repoze.what, I will show that in a later post.


Catégories: Technique
S'abonner à OSGeo-fr agrégateur