Obstacle data - symbology as on aeronautical charts

DOF_Manager QGIS plugin (2) - inserting parsed data into PostGIS database

This post won't be revealing if you familiar with previous posts related to insert data into database.

Method dof_dat_file2postgis simply reads each line of the data file, parse it and after parsing 10 000 lines (or all file if there is less lines) into database.  Notice, that before executing inert satatement parsed data is validated by method check_dof_parsed_data(), which at this satge does nothing and will be completed in the next post.:

Dealing with obstacles data stored in CSV files (2) - validating parsed data

Data should be checked - e. g. coordinates have 'sense' before inserting it into database.
In this post I am  going to write method which checks data in terms of constraint imposed by our model of obstacle database.

Quick 3D visualizaion of airspaces in QGIS

We can use Qgis2threejs to visualize DEM and vector data in 3D.

DOF_Manager QGIS plugin (1) - introduction

Let's back to the problem of dealing with obstacles coming from Digital Obstacle File (DOF) with GIS. I have covered this issue in following posts so far:
PostGIS database for storing FAA Digital Obstacle File - SQL statements to create database
Challenge: importing FAA digital obstacle file content into PostGIS database - Python script to import data from dat file with obstacle data directly into PostGIS database
QGIS Plugin to convert DOF (Digital Obstacle File) data - QGIS plugin that reads dat file and convert it into CSV, KML or shapefile format.
As you can see, we are already able to import obstacles into spatial database, display it using QGIS and convert data into other formats. But we want to go further - a QGIS plugins that reads data and import it directly to database, updates imported data with data from daily change files, easily search and export data with specified criteria (e.g type, location within circle of specified point) to other formats, manually adding data etc.

QGIS customized attribute dialog - Value Relation

In the post you saw how to use Value Map 'method' to bind values between two tables that are linked together - there is a relation between the tables in other words.
But what if the foreign keys or attributes assigned to those keys are changed in one table? This method ('hard coding' values using Value Map) does not track such changes - unless values are updated every time such changes takes place.
Good news is that QGIS has other methods, that are better for such cases - you can use
In simply words any changes in database tables will be reflect in customized attribute dialog.

insert into country (ctry_iso2, ctry_name) values 
('AF', 'Afghanistan'),
('AX','Aland Islands'),
('AL','Albania'),
('DZ','Algeria'),
('AS','American Samoa');

insert into country (ctry_iso2, ctry_name) values 
('GA', 'Gabon'),
('GM', 'Gambia'),
('GE', 'Georgia'),
('DE', 'Germany'),
('GH', 'Ghana'),
('GI', 'Gibraltar'), 
('GR', 'Greece'),
('GL', 'Greenland');

select *
from airspace
where ctry_iso2 = 'GE';

Airspace database - adding country table and further customizing attribute dialog

Let's modify airspace database a bit. We have ctry_iso2 column in airspace table, because we want to to know in which country given airspace is. But what if we want to store more information about country? For example, link to authority that publishes aeronautical data or online version of aeronautical publication?

Customized dialog for airspace database - part 1

In this post we want to achieve customized attribute dialog that looks as follows:

Airspaces and PostGIS - introduction

The shapefile has been the only format for polygon 'part' of aviation data (airspaces). This format is handy, when we want  to store, manage or share data among user but has many limitations - and I don't mean only those specified here:

Another airspace shape

There is another airspace shape that we have not covered yet in our plugins 'family' to create airspaces: rectangle shape. Airspaces sometimes are defined as 'buffer' zone around line connecting two points or more points. Let's deal with the case that only two waypoints so far because this special case is quite often used as part of other more complex airspaces.

Dealing with obstacles data stored in CSV files (1)

Authorities sometimes publishes obstacle data sets called eTOD, which stands from Electronic Terrain and Obstacle Data. Those data sets comes in various formats: csv files, dat files, xml files (AIXM format) or even shapefiles.
This post will start a mini-series which result will be a QGIS plugin to import eTOD data from CSV files into PostGIS database.

QGIS Plugin: csv2polygon

Another plugin that takes data stored in CSV file. This time you can create polygon.  It is especially useful, when you have tabular data that describes polygon (e. g. airspace) as pair of latitude an longitude values. The biggest advantage is that you can create many polygons in one run of a plugin.

AviationLatLonCalculator enhancement: user can choose which CSV field is what

Fields from CSV file have been assigned to its values from CSV header file (e .g.  BRNG, DIST) inside the code so far. User has to prepare input file with 'correct' header - otherwise won't be able to perform calculations. And this is a big disadvantage.
Let's assume that we have following data set:

Parsing NASR data: NAV file

Let's start with defining the fields that values we want to extract from data file. Notice that this for the time being, only some basic data we want to extract so, dictionary contains only one 'subdictionary' NAV1. For more information refer to file description.

 navaid_fields = {'NAV1': {'NAVAID_FAC_TYPE': (20, 9),
                          'NAVAID_NAME': (30, 43),
                          'NAVAID_LAT': (14, 372),
                          'NAVAID_LON': (14, 397),
                          'MAG_VAR': (5, 480),
                          'MAG_VAR_EPOCH_YEAR': (4, 485),
                          'CHANNEL': (4, 530),
                          'FREQUENCY': (6, 534)}
                 }

Parsing NASR data: APT file (landing facility data)

Dictionary that keeps information about 'fields boundary' of attributes we want to extract looks as follows:

apt_fields = {'APT': {'FAC_TYPE': (13, 15),
                      'LOC_IDENTIFIER': (4, 28),
                      'EFF_DATE': (10, 32),
                      'CITY_NAME': (40, 94),
                      'OFFICIAL_NAME': (50, 134),
                      'ARP_LAT': (15, 524),
                      'ARP_LON': (15, 551),
                      'ARP_ELEV': (7, 579),
                      'MAG_VAR': (3, 587),
                      'MAG_VAR_EPOCH_YEAR': (4, 590),
                      'ARFF_TYPE_DATE': (15, 843),
                      'FUEL_TYPE': (40, 901),
                      'REPAIR_SERVICES': (5, 941),
                      'BOTTLED_OXYGEN': (8, 951),
                      'BULK_OXYGEN': (8, 959)
                      },
              'ATT': {'ATTENDENCE_SCHEDULE': (108, 19)}
              }

QGIS Plugin AviationPolygonCreator

As the name suggests with this plugin will be helpful to create polygons that are common in various aeronautical publications and other legal documents. The result of this plugin will be, of course, polygon, from which you will be able to derived WKT string and use it to build airspaces with all attributes, that are relevant for your project. A brief introduction to the problem of 'airspace shapes' you can find in this post: Airspaces - grouping by shape.

How to 'convert' table from pdf to csv (example of extracting points)

Let's imagine that you have  pdf files with some data. You need to extract some of this data for further processing: e. g. file contains table with points with some attributes such as name and your goal is to get points layer from this file. If tables are nicely formatted in the way that if you copy and paste them into text file you will get something that looks like a 'table', I have a good news for you: there is no need to use extra libraries of tools to extract desired data. You can extract data manually by using spreadsheet for example (which is something we are not going to do), websites which converts pdf into other format and etc. You can also take advantage of the fact, that copied table is 'nicely' formatted  and write parser that will extract desired data. It is worth to do that, if we are going to process data in cycles, for example parts of AIP.

QGIS Plugin to extract nodes from polygon (and not only...)

This plugins allows you to easily get following result for selected polygon:

CoordinateExtractor Plugin: improvements

However CoordinateExtractorPlugin work as we expect - there are still some features that are missing that significantly increase capability of getting pairs of coordinates from plain text. The first is quite obvious - user should be able to define her or his own latitude-longitude separator. (I might have implemented only this option from the beginning actually). The second one comes form the fact, that there are possible some deviation from the pattern of coordinates, such as variable number of blank characters as latitude-longitude separator, or in the string that is latitude-longitude separator.

First, we need to change a bit GUI for dialog:
for comboBoxCoordSeparator add new item 'User defined'
new QLineEdit: lineEditUserSeparator
new QCheckBox: checkBoxReomveBlanks
 
Method that sets separator between latitude and longitude has to be changed, so it will takes user defined separator: 

def set_latlon_separator(self):
    if self.dlg.comboBoxCoordSeparator.currentIndex() == 0:
        self.dlg.lineEditUserSeparator.setEnabled(False)
        self.latlon_separator = None
    elif self.dlg.comboBoxCoordSeparator.currentIndex() == 1:  # Null separator
        self.dlg.lineEditUserSeparator.setEnabled(False)
        self.latlon_separator = cet.SEP_NULL
    elif self.dlg.comboBoxCoordSeparator.currentIndex() == 2:  # Space
        self.dlg.lineEditUserSeparator.setEnabled(False)
        self.latlon_separator = cet.SEP_SPACE
    elif self.dlg.comboBoxCoordSeparator.currentIndex() == 3:  # Hyphen
        self.dlg.lineEditUserSeparator.setEnabled(False)
        self.latlon_separator = cet.SEP_HYPHEN
    elif self.dlg.comboBoxCoordSeparator.currentIndex() == 4:
        self.dlg.lineEditUserSeparator.setEnabled(False)
        self.latlon_separator = cet.SEP_SLASH
    elif self.dlg.comboBoxCoordSeparator.currentIndex() == 5:
        self.dlg.lineEditUserSeparator.setEnabled(False)
        self.latlon_separator = '\\'
    elif self.dlg.comboBoxCoordSeparator.currentIndex() == 6:
        self.dlg.lineEditUserSeparator.setEnabled(True)
        self.latlon_separator = self.dlg.lineEditUserSeparator.text()

Module: core_predetermined_angle

In most cases when we will deal with aviation content, especially for large dataset, such eTOD, format of coordinate is known in advance, or 'predetermined' in other words. Some parts of AIP also have one, chosen specified coordinate format also. So, there is no need to 'guess' what coordinate format is in order to convert it into DD. This issue was typical when we deal with frequently changing source, such NOTAM or single instances of airports, runway thresholds when.

Loading point data into PostGIS: significant points example

In the previous post I created table and populated it with data from CSV file, but with no spatial 'part'. This time I going to load from CSV source points into database. CSV files are very common and there are plenty of various kind of data stored in this format.
In AIP, ENR part you can find list of  'Name-code designators for significant points'.
If we have this data into CSV forma, we can do following steps to create spatial data:

Create temporary waypoint table:

What STWL stands for? And FAS or DER? Abbreviation table.

Aviation domain has countless number of abbreviations and acronyms that are used in publications such as NOTAMs or charts.
To make life a bit (or rather much) easier it is a good idea to create a table with abbreviations, which will contain as many as possible abbreviations, acronyms and their meanings for quick reference if we work with aviation content.

Extracting coordinates - user friendly tool (4) (QGIS Plugin) - creating points, line, polygon

We have reach point when we are able to get WKT representation of plain text - for points, line and  polygon. Now it's time to visualize this 'horrible' string. I will show how to create spatial features from plain text in this post. If you familiar with one of the post dealing with creating 'asirpsace-like' polygons, you probably got idea how to do it. It is easy - actually we already got it and we only need to add very little code.

First we want to clear WKT string and table with extracted point, than extract coordinates - create table with points, to make sure that feature are created from current plain text:

self.dlg.textEditRawTextWKRTString.clear()
self.dlg.tableWidgetCoordinates.setRowCount(0)
self.extract_coordinates()

aviation_gis_tools2: 'core' part

It's been a while since I announced major changes to my 'aviation-python-gis' related project. I have even started using the redesigned classes in latest plugins, so it's time to share this new scripts. I am not going to be too much verbose, because basically it is more 'object oriented' version of previous version aviation_gis_tools.
In this post I am discussing 'core; part, I mean Python modules that names starts with 'core'. Those modules contains all basic functionality  such as: constants that are used throughout other modules and plugins and classes  and function that deal with distances, angles, bearings, magnetic variations, calculation on ellipsoid an other.

Source code can be found here:https://github.com/strpaw/aviation_gis_tools2

QGIS Plugin: AviationLatLonCalculator

Based on previous plugins related to calculating geographical coordinates given in various ways, it's time to combined all them into one handy plugin. In this case I will also use enhanced version of aviation gis tools scripts.

Extracting coordinates - user friendly tool (3) (QGIS Plugin) - getting WKT string

If we have extracted coordinates, it would be nice to automatically to do something useful with them - creates points, line or polygon. So, next step is to add possibility to get WKT string  or create features such points, lines and polygonx. WKT  can be very handy - in case we want to copy and paste 'coordinates' into SQL statements in PostGIS for example.

Populate aviation_postgis_db with data from NASR files (1)

28 Day NASR Subscription data is a very good  source to start a series of post related to populating aviation_postgis_db database.  
In the post Parsing FAA NASR data: NATFIX file  I show the way of extracting data from NATFIX file into database. Our sample aviation database has limited scope, so we are not going to  retrieve all kind of information that can be found in NASR files, but only part of them. If you get familiar with files descriptions (refer to https://www.faa.gov/air_traffic/flight_info/aeronav/aero_data/NASR_Subscription_2019-02-28/, section 'The layout data for the Legacy (TXT) formats...) you will notice that basically data is arranged into fixed-length records for each airport, navigation aid and etc. The numbers of records can be variable, in case of Airport and Other Landing Facilities (APT file), in order to contain information about all runways related to given airport or aditional information. A 'record' is a line of text file, which  has 'fields' described by length, start location and justyfi - left or right. If there are various of record types in given file, each record starts with record type identifier. So, if we know at what  position the attribute we are interested in is, we are able to get it and assign to appropriate airport, navigation aid and so on.

Extracting coordinates - user friendly tool (2) (QGIS Plugin)

We have managed to define what format of coordinates we are going to extract so far, now it's time to extract coordinates - 'real work' begins.
In this post following work is to do: user can paste raw text (NOTAM, part of eAIP or any other text with coordinates) into plugin, press button to extract coordinates, and get as result nice, easy to read table with coordinates.

Extracting coordinates - user friendly tool (1) (QGIS Plugin)

In the real life (work) we rather not expect following workflow:
parse text and extract coordinates in the way described in the previous post.

Let's hide all that 'What is this all about' stuff behind QGIS Plugin. User can set all required parameters to extract coordinates and press 'extract' button to do so:

Extracting data tools - coordinates

I will use regular expressions to extract data from raw text and named tuples for convenient way to access tuple items, so first I import  required modules:

import re
from collections import namedtuple

I am going to build desired regular expression to extract coordinates from text based on coordinates order (lat-lon or lon-lat), coordinates formats and separator character that separates latitude and longitude within coordinate pair not one coordinate pair from another one.

aviation_gis_tools: new approach

If you have followed previous posts and dig a bit into code you notice that there is a room for improvement. Because of nature of main issue - calculation of  coordinates based on various input data (reference point in different coordinate formats, bearings, magnetic variation, local Cartesian coordinate system etc.) it is obvious that well designed classes are a good choice to deal with it. We also want to track other information which I have already mention previously.
We have also started dealing with something else: parsing and extracting data from aeronautical sources. So far we are able to

Points in local Cartesian coordinate system - multiple points

It should be unsurprisingly that this  post will be an 'extension' of the solution described in previous one: QGIS plugin to deal with many points in local Cartesian coordinates system. It is likely that data you will deal with will be in a tabular form, or in a form which tabular form can be obtain more or less easily.

Challenge: QGIS Plugin to insert points in 'Local Cartesian' coordinates

In the previous post I create class, that allows easily to calculate latitude and longitude of point that is given in local Cartesian coordinates. Now it's time to make life even more easier - custom QGIS plugin. The  result is as below:

aviation_gis_tools: Point given in 'Local Cartesian' coordinate system

Sometimes points are given in Cartesian coordinates (x, y) with origin of coordinate and orientation of X and Y axes. There are a few approaches that we can transform from local Cartesian coordinate system (x, y) into global geographic coordinate system (latitude, longitude).
Here I am going to use approach similar to those described in post. In other words I will treat x, y coordinates and information about axes orientation as point with offset against reference point - origin of the local coordinates in this case.

LocalCartesianPoint is a class that will calculate latitude and longitude point
Constructor of LocalCartesianPoint class:

class LocalCartesianPoint:
    def __init__(self, origin_point: BasePoint, x_axis_brng: Bearing, y_axis_brng: Bearing, x: Distance, y: Distance):
        self.origin_point = origin_point
        self.x_axis_brng = x_axis_brng
        self.y_axis_brng = y_axis_brng
        self.x = x
        self.y = y
        self._calc_dd_lat = None
        self._calc_dd_lon = None

aviation_db: QGIS plugin to insert obstacles

So, if we have got aviation_db (created in post)  it's time to populate it with some real data. We have learnt so far, that obstacle data are published in variety of forms in aeronautical publications, especially the 'spatial' part of them: 'direct' coordinates, distances and bearings against point (threshold, aerodrome reference point), offset against distance and bearings against specified points, 

Parsing FAA NASR data: NATFIX file

NASR stands for National Airspace System Resource.
For more infromation about NASR, please refer to:https://www.faa.gov/air_traffic/flight_info/aeronav/aero_data/NASR_Subscription/

aviation_postgis_db1: tables

NOTE: The aviation_postgis_db1 is spatial database for training and educational purposes only: that means that is rather used to learn and practice how to use Python, GIS software and PostGIS to parse, create, maintain and visualize aviation data that to perform real work, at least at this stage of develpment cycle.
The aviation_postgis_db1 is spatial database to keep aviation data that can be obtained from various online sources such as: eAIP portals, FAA website with data (DOF, NASR). Content of databse is in limited scope: that means that some selected attributes are used to model aviation world. Selection is based on what information can be found in aeronautical publications (eAIP, FAA Digitial Obstacle File ) and etc.

Country table:

CREATE TABLE country (
 rec_id serial,
 ctry_icao_code char(2) PRIMARY KEY,
 ctry_name varchar(100) NOT NULL
);

PostGIS aviation_database and introduction to real world examples

I have covered some basic but indispensable functionality (distance, coordinate, bearing, local coordinates etc.) so far. I have also developed some tools (QGIS plugins) that allows us to easily create features that are common in aviation world:

• polygons based on circles and circle parts (useful to create airspaces)
• creating points based on bearing, distance, offset against reference point - which is common way of defining location in aviation

Now, it's time to move on and show how those tools can be used in order to create and manage data with aeronautical content. It will require to create a spatial database and new tools.

QGIS plugin: PointByPolarCoordCSV (multipoint mode)

This is version of PointByPolarCoord plugin if we have many points for which bearings, distances and reference point is given.