For accurate economic inference using nighttime lights data, it's essential to perform thorough data cleaning. The rationale behind this process is elaborated in the work "But clouds got in my way: Bias and bias correction of VIIRS nighttime lights data in the presence of clouds" by Patnaik, Ayush et al. (2021)
Additionally, a comprehensive description of the API can be found in the paper "Foundations for nighttime lights data analysis" by Patnaik, Ayush, Ajay Shah, and Susan Thomas (2022)
NighttimeLights.na_recode
— MethodThe na_recode function also works on datacubes.
na_recode(radiance_datacube, ncfobs_datacube)
Wherever the number of cloud-free observations is 0, radiance will be marked as missing.
NighttimeLights.na_interp_linear
— MethodUses linear interpolation to fill for missing values. This function work for any kind of array, and not just nighttime lights radiance time series data.
x = rand(1:10.0, 10)
x = Vector{Union{Missing, Float64}}(x)
x[5] = missing
na_interp_linear(x)
NighttimeLights.outlier_variance
— MethodThere are extremely high values in the nighttime lights data due to fires, gas flare etc. You may find some values even greater than the aggregate radiance of large cities. Such pixels also have high standard deviation. These pixels may not be of importantance from the point of view of measureming prosperity. The outlier_variance
function generates a mask of pixels with standard deviation less that a user defined cutoff value, that defaults to the 99.9th percentile. Essentially, this function can be used to remove the pixels with the high standard deviation. A mask can be provided to the function, so that it calculates the percentile based on the lit pixel of the mask. For example, if the radiance_datacube
is a box around Mumbai and the mask is the polygon mask of Mumbai, the outlier_variance
function will calculate the 99th percentile (default cutoff value) of the standard deviation of the pixels inside Mumbai's boundary.
outlier_variance(radiance_datacube, mask; cutoff = 0.999)
NighttimeLights.outlier_hampel
— FunctionThe time series of a pixel may show a few outliers, but as a whole the pixel may be of importantance in measuring economic activity. The outlier_hampel function uses replaces the outlier observations with interpolated values. This is done using the tsclean function the forecast package of R.
outlier_hampel(radiance_pixel_ts)
NighttimeLights.bgnoise_PSTT2021
— FunctionPixels with no economic activity may show some light due to background noise. These pixels could be in forests, oceans, deserts etc. The bgnoise_PSTT2021
function generates a background moise mask such that those pixels which are considered dark are marked as 0 and those considered lit are marked as 1. The function uses the datacubes of radiance and ncfobs to generate annual image of the last year the data. The function considers all the pixels below a provided threshold as dark and remaining to be lit.
bgnoise_PSTT2021(radiance_datacube, ncfobs_datacube)
NighttimeLights.bias_PSTT2021
— FunctionThe bias correction function can use the datacubes of radiance and the number of cloud-free observations to correct for attenuation in radiance due to low number of cloud-free observations.
bias_PSTT2021(radiance_datacube, ncfobs_datacube)
NighttimeLights.bias_PSTT2021
— MethodThe bias correction function can use the datacubes of radiance and the number of cloud-free observations to correct for attenuation in radiance due to low number of cloud-free observations.
bias_PSTT2021(radiance_datacube, ncfobs_datacube)
NighttimeLights.PSTT2021_conventional
— MethodAll steps of data cleaning that most researchers do can be performed using the conventional cleaning funciton.
PSTT2021_conventional(radiance_datacube, ncfobs_datacube)
NighttimeLights.PSTT2021
— MethodThe PSTT2021 function performs all the steps of the new cleaning procedure described in But clouds got in my way: Bias and bias correction of VIIRS nighttime lights data in the presence of clouds, Ayush Patnaik, Ajay Shah, Anshul Tayal, Susan Thomas as conventional cleaning.
PSTT2021(radiance_datacube, ncfobs_datacube)
NighttimeLights.clean_complete
— MethodThe function clean_complete()
represents our views on an optimal set of steps for pre- processing in the future (for the period for which this package is actively maintained). As of today, it is identical to PSTT2021()
`