Miscellaneous documentation

Karmana

Karmana.jl is a library which implements utilities to munge CMIE CPHS data and visualize it. It exposes multiple, orthogonal capabilities.

Karmana's visualization utilities are built on the Makie.jl ecosystem, including the GeoMakie.jl package.

The package is built to automate some processes to:

• Retrieve and process data from the CMIE CPHS and Capex databases
• Plot this data on maps of India.
• Create coherent and good-looking posters of plots quickly and easily.

Installing

Karmana.jl is meant to work with CMIE CPHS data, and is not meant to be released to the General registry.

using Pkg
Pkg.add(url = "https://github.com/xKDR/Karmana.jl")

To add a specific branch or tag, provide the rev = "branch_name" keyword argument.

Usage

Karmana.jl implements several orthogonal functions. For more information, please see the documentation API page, or by running ?funcname in the REPL to access Julia's help mode.

Plotting and visualization

• create_page(page_size::Symbol, args...; kwargs...): Creates a Makie.jl figure which is optimized for a figure of the appropriate size, along with a "header" row (GridLayout) which has a logo and poster title, and a "footer" row (GridLayout) which has a description Label, space for a legend or colorbar, and a QR code with a customizable link. See the documentation for more!
• indiaoutline!(admin_level::Symbol, ids, vals): A Makie.jl recipe which is able to plot at one of three admin levels (:State, :HR, and :District) - and display the other admin levels' borders.
• TernaryColormap(xgrad, ygrad, zgrad), which creates a "ternary colormap" that can be called on x, y, z values for which x + y + z = 1, and returns a ternary interpolated version of the color at the specified coordinates on the plane.

Global variables

All of these variables are populated by Karmana.__init__(), and can their values can be accessed by, for example, Karmana.state_df[] (note the empty square brackets, which indicate that you're accessing the value of the Ref).

• state_df::Ref{DataFrame}: A DataFrame which holds geometry data and identification keys for India's states.
• hr_df::Ref{DataFrame}: A DataFrame which holds geometry data and identification keys for India's homogeneous regions, as defined by CMIE.
• district_df::Ref{DataFrame}: A DataFrame which holds geometry data and identification keys for India's districts.
• india_rivers::Ref{ArchGDAL.IGeometry}: An ArchGDAL.IGeometry which holds the intersection of the rivers of the world with the border of India.

CPHS helper functions

Karmana has several CPHS helper functions to parse data.

Capex helper functions

Karmana has some parsers for CMIE Capex data lat/long strings.

Spatial utilities

Karmana has some geodetic/spatial utilities, like annular_ring and line_to_geodetic_width_poly. See the docs and examples for more information!

Environment variables

Karmana can be configured by the following environment variables:

• KARMANA_DISTRICT_SHAPEFILE which points to a shapefile which Karmana should use to populate the district, HR and state dataframes. Note that there are a lot of assumptions made on the structure of the shapefile - look at the code of Karmana.__init__() to see what these are.
• KARMANA_APPLY_SHAPEFILE_PATCHES indicates whether to apply certain patches to the shapefile ("true") or not ("false"). Defaults to true.
• KARMANA_RIVER_SHAPEFILE indicates the path to some shapefile which contains a selection of rivers (usually as linestrings). If not found, Karmana will download this data from UNESCO.

Contains the District dataframe

Contains the HR dataframe

Contains an ArchGDAL.IGeometry which contains a multilinestring of the intersection of the world's rivers with India.

Contains the State dataframe

struct TernaryColormap
TernaryColormap(; xmap, ymap, zmap)

Represents a ternary colormap.

Construct by passing three PlotUtils.ColorGradients or objects which can be converted to them (symbols, strings).

Call by using the ternary colormap object (tmap) as a callable - methods include tmap(x, y, z) or tmap(Point3f(...)) or tmap((x, y, z)). Returns an RGBAf object when called.

Visualize by calling Makie.plot(tmap).

Karmana.__init__()

Initializes the package by loading geometries. This is only really relevant to the indiaoutline recipe.

First, load the state, hr and district geometries of India from data.mayin.org, or the provided cache path. Then, compute the intersection between the world's rivers (provided by env variable) and India, or load from cache stored in scratchspace.

_missing_in(x, set)

Handles the case where x is missing, which in does not. If x is missing and there is a missing value in set, then returns true. If there is no missing value in set, returns false. Otherwise, the behaviour is the same as Base.in.

prepare_merged_geom_dataframe(df::DataFrame, hr_column_id::Symbol; capture_cols::Tuple{Symbol})

Prepares a dataframe of merged geometries by grouping df by hr_column_id. The values of each of the capture_cols in the first row of each group are also included in the new dataframe, along with the value of hr_column_id. Each group in the input corresponds to a row in the output dataframe.

This method assumes that there is a geometry column in the DataFrame which contains objects which have a MultiPolygonTrait in GeoInterface.

Returns a DataFrame.

_set_plot_z(plot, zlevel::Real)

Sets the plot's z-level to the provided argument, leaving the rest of the translation attributes the same.

annular_ring(f, source::Raster, lon, lat, outer_radius, inner_radius; pass_mask_size = false)

Returns the result of applying f to the subset of source which is masked by the annular ring defined by lon, lat, outer_radius, inner_radius. The annular ring is constructed in geodetic space, i.e., distance is physically preserved.

source may be a 2D raster (in which case this function returns a value), or a 3D RasterStack or a RasterSeries of 2D rasters, in which case this function returns a Vector of values.

Arguments

• f is a function which takes a Raster and returns a value. This is the function which will be applied to the subset of source which is masked by the constructed annular ring. The result of f is returned.

• source is a Raster which will be masked by the annular ring. This can be 2D, in which case annular_ring will return a single value, or 3D, in which case annular_ring will return a Vector of values.

• lon, lat are the coordinates of the centre of the annular ring, in degrees.

• outer_radius and inner_radius are the outer and inner radii of the annular ring, in metres.

• pass_mask_size determines whether to pass a second argument to f containing the number of points in the mask. This is useful if you are taking statistical measures.

How it works

• First, an annular ring polygon is created (in geodetic space, using get_geodetic_circle).
• Next, the extent of the annular ring polygon is computed, and the source raster is subsetted to that extent. This is for efficiency, and so that the minimum possible number of multiplications are performed.
• The annular ring polygon is rasterized, using Rasters.boolmask.
• The subsetted source raster is multiplied by the rasterized annular ring polygon.
• Finally, f is applied to the result of the multiplication.

create_page(paper_size, qr_code_link; landscape = automatic, naxes = 1, supertitle = "Title", description = "...", kwargs...)

Creates a figure of the specified size with the specified arguments, themed for XKDR. Applies the appropriate paper size theme (theme_a4, theme_a3, etc.)

Arguments

• paper_size: A symbol representing the desired paper size; can be :a[0/1/2/3/4]. More planned. In the future, you may also be able to pass a tuple.
• qr_code_link: The contents of the QR code shown at the bottom left of the page. Must be a string.

Keyword arguments

• landscape = automatic: Decides whether the figure should be in landscape or portrait mode. If automatic, decides automatically. To set this manually, set landscape = true or landscape = false.
• naxes = 1: The number of axes to create in the central grid. Automatically laid out.
• axistitles = Makie.automatic: The titles for each axis. If set to automatic, they will be the positions of the axes in the layout.
• hideaxisdecorations = true: Whether to hide the decorations (tick labels, tick marks, etc.) for each axis.
• hideaxisspines = true: Whether to hide the spines (frame lines) for each axis.
• supertitle = "Title": The title of the figure, in the "header" gridlayout.
• description = "Placeholder": The description of the figure, in the "footer" gridlayout (page.description_layout[1, 1]).
• padding: The padding around the figure. If a number, sets all sides to the same value. If a tuple, sets the padding to (left, right, top, bottom).
• axisaspect = DataAspect(): Sets the aspect ratio of the axis. You can set this to nothing if you want the default.

Returns

Returns a NamedTuple containing the following items:

• figure: The Figure in which everything is plotted.
• supertitle: The Label which serves as the figure's title.
• axis_layout: The GridLayout in which the axes are placed.
• axes: A Matrix{Union{Axis, Nothing}} which contains the axes which can be placed. If nrows * ncols > naxes, then the remaining positions will be nothing.
• description_layout: The GridLayout in which the description is placed. Has 3 columns and 1 row. The description label is located in description_layout[1, 1], and [1, 3] is reserved for a box representing the QR code. You can plot a legend or colorbar in description_layout[1, 2].
• description_label: The Label which holds the figure's description.

The items can be extracted from the named tuple using standard getproperty syntax, as in the following example:

page = create_page(:a4, "https://xkdr.org")
page.figure
page.axes[i::Int, j::Int]
page.description_layout
page.description_label
...

do_geoquery(connection, layer; geometrycols = ["SHAPE"])::DataFrame

Performs a SELECT * FROM $layer operation on the database which connection points to, but all geometrycols are additionally wrapped in ST_AsBinary, which converts geometries from SQL format (which has an extra CRS indicator) to well known binary (WKB) format, which is parseable by e.g. ArchGDAL (or WellKnownGeometry.jl, which is substantially slower).

WKB columns are given the suffix _wkb to differentiate them from the original columns.

Results are returned as a DataFrame.

get_HR_number(hr::Union{String, Missing})::Union{Int, Missing}

Extracts the number from a string of a form "HR ???" and returns it. If the input is missing, then missing is returned.

get_geodetic_circle(lon, lat, radius; npoints = 100, geodesic = Proj.geod_geodesic(6378137, 1/298.257223563))

Returns a Vector of Makie.Point2f which represent a circle of radius radius centered at lon, lat, computed in geodetic coordinates.

lon and lat are in degrees, radius is in metres.

!!! note Performance Because this calls out to C, it's a bit slower than I'd like it to be, but not by much. 100 points takes about 28ms on my machine, a souped up Macbook Pro M1.

Making annular rings

To create an annular ring, it's sufficient to say:

lon, lat = 72, 19
inner_radius = 1000
outer_radius = 10000
annular_polygon = GeometryBasics.Polygon(
        get_geodetic_circle(lon, lat, outer_radius), 
        [reverse(get_geodetic_circle(lon, lat, inner_radius))] # note the `reverse` here - this is for the intersection fill rule.
)

get_sentiment_props(df, sentiment_key; good = "Good times", bad = "Bad times", uncertain = "Uncertain times")

Takes in a DataFrame from the CPHS aspirational wave database, and returns a tuple of (bad_prop, good_prop, uncertain_prop).

The keys can be changed by keyword arguments; fundamentally, this is a helper function to extract proportions from a three-value system.

indiaoutline!(admin_level::Symbol, ids::Vector, vals::Vector{<: Real}; kw_args...)
indiaoutline!(admin_level::Symbol, dataframe::DataFrame, [id_column::Symbol], value_column::Symbol; kw_args...)

Plots an outline of India, merged with the data passed in. This data must fundamentally have two things: a column of IDs, and a column of values. The IDs must match the IDs in the CPHS database, and the values may be either numbers, or explicit colors.

Arguments

admin_level must be one of :State, :HR, or :District.

ids must be a Vector{Union{Int, Missing}} or a Vector{Int}. It and vals must have the same length.

Attributes

One can set the attributes of the various plot elements by setting the values of the corresponding nested Attributes. These are plot.State, plot.HR, plot.District, and plot.River.

For example, to set the stroke width of districts to 0.25, one would do:

plot.District.strokewidth[] = 0.25

The attributes available for State, HR, and District are those of poly; the attributes available for River are those of lines.

Cropping the map to provided data

If the attribute crop_to_data is true, then this crops the map to the bounding box of the provided IDs only, and does not draw any other states/HRs/districts. Otherwise, all available geometries are drawn, but only the provided IDs are colored by their values; the rest of the geometries remain transparent.

Controlling how the data is merged

You can control the column on which data is merged by setting the merge_column and external_merge_column keyword arguments.

• merge_column specifies the key with which to merge of the provided ids to the CPHS database for that admin level.
• external_merge_column specifies the key with which to merge the provided ids with the lower admin level geometries.

For example, if the provided admin_level is :State, then merge_column will control the key for state_df, and external_merge_column will control the key for hr_df and district_df.

To see all available attributes and their defaults, have a look at the extended help section by running ??indiaoutline! in the REPL.

Extended help

Available attributes, and their values

Available attributes and their defaults for Combined{Karmana.indiaoutline!} are:

indiaoutline!(admin_level::Symbol, ids::Vector, vals::Vector{<: Real}; kw_args...)
indiaoutline!(admin_level::Symbol, dataframe::DataFrame, [id_column::Symbol], value_column::Symbol; kw_args...)

Plots an outline of India, merged with the data passed in. This data must fundamentally have two things: a column of IDs, and a column of values. The IDs must match the IDs in the CPHS database, and the values may be either numbers, or explicit colors.

Arguments

admin_level must be one of :State, :HR, or :District.

ids must be a Vector{Union{Int, Missing}} or a Vector{Int}. It and vals must have the same length.

Attributes

One can set the attributes of the various plot elements by setting the values of the corresponding nested Attributes. These are plot.State, plot.HR, plot.District, and plot.River.

For example, to set the stroke width of districts to 0.25, one would do:

plot.District.strokewidth[] = 0.25

The attributes available for State, HR, and District are those of poly; the attributes available for River are those of lines.

Cropping the map to provided data

If the attribute crop_to_data is true, then this crops the map to the bounding box of the provided IDs only, and does not draw any other states/HRs/districts. Otherwise, all available geometries are drawn, but only the provided IDs are colored by their values; the rest of the geometries remain transparent.

Controlling how the data is merged

You can control the column on which data is merged by setting the merge_column and external_merge_column keyword arguments.

• merge_column specifies the key with which to merge of the provided ids to the CPHS database for that admin level.
• external_merge_column specifies the key with which to merge the provided ids with the lower admin level geometries.

For example, if the provided admin_level is :State, then merge_column will control the key for state_df, and external_merge_column will control the key for hr_df and district_df.

To see all available attributes and their defaults, have a look at the extended help section by running ??indiaoutline! in the REPL.

Extended help

Available attributes, and their values

Available attributes and their defaults for Combined{Karmana.indiaoutline} are:

  District               Attributes with 7 entries:
  label => Districts
  names => false
  nan_color => RGBA{Float32}(0.0,0.0,0.0,0.0)
  strokecolor => RGB{N0f8}(0.549,0.549,0.549)
  strokewidth => 0.2
  visible => true
  zlevel => 98
  HR                     Attributes with 7 entries:
  label => HR regions
  names => false
  nan_color => RGBA{Float32}(0.0,0.0,0.0,0.0)
  strokecolor => RGBA{Float32}(0.901961,0.623529,0.0,1.0)
  strokewidth => 0.2
  visible => true
  zlevel => 99
  Legend                 Attributes with 2 entries:
  draw => true
  polypoints => 1
  River                  Attributes with 5 entries:
  color => RGB{N0f8}(0.678,0.847,0.902)
  label => Rivers
  linewidth => 0.2
  visible => true
  zlevel => 97
  State                  Attributes with 7 entries:
  label => States
  names => false
  nan_color => RGBA{Float32}(0.0,0.0,0.0,0.0)
  strokecolor => RGB{N0f8}(0.0,0.0,0.0)
  strokewidth => 0.2
  visible => true
  zlevel => 101
  colormap               :viridis
  colorrange             MakieCore.Automatic()
  crop_to_data           false
  external_merge_column  MakieCore.Automatic()
  highclip               "nothing"
  lowclip                "nothing"
  merge_column           MakieCore.Automatic()

latlong_string_to_points(latlong_string)

Parses a string of the form lat1,long1 : lat2,long2 : lat3,long3 : ... and returns a Vector of Point2e which define points as (long, lat).

Is robust to cutoff errors and other potential issues.

line_to_geodetic_width_poly(line::Vector{<: Point2}, width; geodesic = Proj.geod_geodesic(6378137, 1/298.257223563))

Returns a Vector{Point2{Float64}} which represents a polygon which is width metres wide, and follows the path of line.

This is mostly useful for tracing wide lines on Raster maps.

Fundamentally, you can think of this function as creating a polygon from a line, with a specified width. There's no interpolation, though - if you want interpolation, pass an interpolated vector of points in.

maps_db_connection(user = ENV["MAPS_USER"], password = ENV["MAPS_PASSWORD"])::DBInterface.Connection

Returns a connection to the maps database on data.mayin.org, which must be closed by DBInterface.close!.

merge_polys(polys::AbstractVector{<: Union{Polygon, MultiPolygon}})

Merges a vector of polygons into a single MultiPolygon using ArchGDAL.union.

Returns an ArchGDAL geometry.

nearest_paper_size(width::Real, height::Real)::Symbol

Returns the closest paper size to the provided size, which must be a 2-tuple.

points_weights(latlong_strings::Vector{:< AbstractString}, costs::Vector{<: Real})

Parses strings of the form lat1,long1 : lat2,long2 : lat3,long3 : ... and returns a Vector of Point2e which define points as (long, lat), as well as a vector of weights per point. If the string has more than one point defined, the weight is spread across all n points such that each point has a weight of cost[i]/n.

Returns (::Vector{Point2e}, ::Vector{<: Real}).

prepare_merged_river_geom(shapefile_path, mask_poly)

Uses ArchGDAL to prepare a multilinestring which shows river paths within India.

prepare_page(
    paper_size::Union{Symbol, NTuple{2, <: Real}},
    qr_code_contents::String;
    landscape = false,
    padding = 3,
    logo = rotr90(FileIO.load(assetpath("logos", "XKDR_Logomark_RGB_White.jpg"))),
    logo_height = 40,
    logo_padding = 5,
    qr_code_height = 60,
    qr_code_padding = 10,
)

rgb_to_cmyk_pdf(source_file::AbstractString, dest_file::AbstractString)

Runs Ghostscript on source_file to convert its color schema from RGB to CMYK, and stores the result in dest_file.

This works well when preparing a PDF for printing, since many printer drivers don't perform the conversion as well as Ghostscript does. You might see a green tint on black when printing in color; this ameliorates that to a large degree.

searchsortednearest(a, x)

Returns the index of the nearest element to x in a. a must be a sorted array, and its elements must be mathematically interoperable with x.

shape_wkb_to_module_geom!(mod::Module, table::DataFrame; new_colname = :geometry, wkb_colname = :SHAPE_wkb)

Converts the geometries in old_colname (in WKB format as eltype(old_colanme) = Vector{UInt8}) into geometries of the provided module mod. This goes through ArchGDAL instead of being pure-Julia with WellKnownGeometry, since that's faster.

state_hr_district_dfs()

A wrapper function which materializes the state, HR, and district dataframes in Julia, by connecting to the maps database of data.mayin.org.

Returns 3 DataFrames (state_table, hr_table, district_table) which all have columns :geometry populated by GeometryBasics geometry, which is suitable for plotting.

target_point(lon, lat, azimuth, arclength; geodesic = Proj.geod_geodesic(6378137, 1/298.257223563))

Returns a Makie.Point2f which represents the point at arclength metres from lon, lat in the direction of azimuth. Basically a thin wrapper around Proj's GeographicLib geod_directline.

jldoctest julia> target_point(0, 0, 0, 0) 2-element Point2{Float64} with indices SOneTo(2): 0.0 0.0`