OS Translator II is a graphical loader / translator for Ordnance Survey GML-based products. It reads .gml and .gml.gz files and loads them into PostGIS (an open source spatial database). Its development has focussed on usability and import speed.

For benchmark test results comparing OS Translator II and other tools, see this blog post.

It can also fetch and apply styles:

This tool is currently in active development. If you find a problem, please tell us so we can address it.

Features

• Automatically apply official Ordnance Survey styles (see below)
• Speed up imports by taking advantage of multiple CPU cores
• Batch-import whole folders of .gz or .gml files
• Specify which fields to import
• Automatically remove duplicate features (artefacts of Ordnance Survey's data delivery process)

Official OS MasterMap QGIS Styles

In July 2015 Ordnance Survey released a complete set of QGIS styles for OS MasterMap Topography Layer. To use these sexy new styles (like the one pictured above), simply import your data with the following options checked:

OS Translator II takes care of everything else: adding any additional columns required, obtaining the latest styles and saving them as default style(s) in the PostGIS database.

To fully appreciate OS' new styles we recommend installing the accompanying font and SVGs (scalable vector graphics). This can be done by:

1. Downloading the latest release of the OS styles
2. Unzipping and locating the folder containing the QGIS .qml files
3. Installing the font, OSMasterMap.ttf
1. Windows users can simply double-click this file (Administrator rights required)
4. Next, copying the osmmsymbols folder (the folder, not just its content) somewhere sensible (it contains the SVG symbols)
5. Then in QGIS:
1. Settings > Options
2. System tab
3. Adding the path to the folder in which you placed the osmmsymbols folder to the SVG paths
4. OK
5. Restarting QGIS (pedantic)

You should now see the new styles as intended including pattern fills.

If you're wondering how to make best use of this in your organisation then just ask. We're always happy to talk about QGIS deployment.

Known issues

Some users have experienced QGIS locking-up or crashing when attempting to use the new styles without updating their SVG path as described above. This is a QGIS issue currently under investigation. If you experience similar problems then ensure you have correctly set up your SVG path.

PostGIS Performance Tips

In this section you'll find tips on how to speed-up the import process.

Processor cores

Many modern CPUs take advantage of something called simultaneous multithreading (SMT) to make better use of available CPU resources in most common situations. This causes the operating system to 'see' twice as many CPUs cores as there are physically present. Consider this screenshot of Task Manager on a dual core i3 processor:

However CPU-Z shows the CPU to have 2 cores as expected:

For tasks where CPU performance is a bottleneck (as in the case of OS Translator II) using more CPU cores than physically available may actually degrade performance. For this reason it is recommended to set the Processor Cores option equal to the number of physical processor cores present (2 in this example).

You can determine whether SMT is enabled (referred to as Hyper-threading on Intel-based CPUs) in your machine's BIOS or by using a tool like CPU-Z.

Only remove duplicates when needed

The Remove duplicates option creates a temporary hash index on the fid column to speed-up the removal of duplicate features. Creating the index requires OS Translator II to read the fid column for the entire table which can be time-consuming. Therefore it's recommended to only use the Remove duplicates option when loading 'chunked' data although the option is safe to use in all situations.

Only import the fields you need

Importing more data takes more time and more disk space. Therefore it is recommended to only import the fields / columns you require.

PostgreSQL configuration

A number of changes are recommended to the default PostgreSQL configuration which will lead to a performance boost:

• shared_buffers 512MB
• work_mem 16MB
• maintenance_work_mem 1024MB
• checkpoint_segments 6
• random_page_cost 2.0

In addition to the above changes the following temporary changes can be made to significantly speed up the import process but may not be suitable in your specific environment (see warning below):

• fsync off

Turning fsync off is dangerous and can lead to data loss in the event of an unexpected power outage. Always switch fsync back on after loading and never use this option on a database containing critical data.

Disk hardware

Especially for the indexing and de-duplication phases, having your destination tablespace on a solid state disk (SSD) can significantly speed-up load times.

Limitations

This early release has some limitations. General limitations include:

• Supports OS MM Topo (v7), OS MM ITN (v7) and OS MM ITN Urban Paths Theme (v8)
• Only supports create or replace (not append)
• Source folder tree must contain only one type of OS product mentioned above
• Field selection preference not currently stored for later use
• Change-only update not currently supported

ITN-specific limitations include:

• Bounding box for Road layer not imported
• However this geometry can easily be generated with a post-import SQL query
• Fields currently not imported:
• dateTimeQualifier
• environmentQualifier
• vehicleQualifier
• subsectionPoint
• Loading ITN and ITN Urban Paths into the same schema causes the following tables to be overwritten:
• ferrylink
• ferryterminal
• ferrynode

What about Shapefiles?

OS Translator I supports translation to Shapefile and is still available and supported. We strongly recommend users consider migrating to PostGIS for storing OS MasterMap products due to the many advantages it has over Shapefiles. Please contact us for advise on how to go about doing this. PostGIS is a powerful free and open source spatial database.

---