GitHub

ALCF

Automatic Lidar and Ceilometer
Framework

Installation

Below you can find installation instructions for supported operating systems:

You can also build the ALCF from the source code if you want to modify any of its parts.

Linux

The instructions below assume that you enter the commands in the terminal.

  1. Install required system packages.

    On Debian-based Linux distributions (Ubuntu, Debian, Devuan, …), install dependencies with:

     sudo apt install gcc make gfortran libhdf5-dev libnetcdf-dev \
     libnetcdff-dev python3 python3-setuptools python3-pip cython3 pipx \
     libeccodes-tools ncl-ncarg

    Note: ncl-ncarg is not available on Ubuntu 24 and later. You can omit this package, but alcf convert for JRA55 will not work unless you install NCL (the command ncl_convert2nc is required for the conversion).

    On Fedora, install dependencies with:

     sudo yum install make gcc gfortran hdf5-devel netcdf-devel \
     netcdf-fortran-devel python3-setuptools python3-pip python3-Cython \
     pipx eccodes ncl

    On AlmaLinux 9 (or later), install dependencies with:

     sudo dnf install epel-release
 sudo dnf config-manager --set-enabled crb
 sudo dnf install make gcc gfortran hdf5-devel netcdf-devel \
     netcdf-fortran-devel python3-devel python3-pip python3-setuptools \
     python3-numpy python3-Cython pipx eccodes ncl

  2. Install the ALCF with:

     pipx install alcf
 mkdir -p ~/.local/share/man/man1
 ln -s ~/.local/pipx/venvs/alcf/share/man/man1/alcf*.1 ~/.local/share/man/man1/

    Make sure that $HOME/.local/bin is in the PATH environment variable.

Windows

It is recommended to run ALCF on Linux.

Installation on Windows is possible under the “Windows Subsystem for Linux”.

  1. Install “Ubuntu” from the Microsoft Store. You might have to enable “Windows Subsystem for Linux” under “Windows Features” first.

  2. Open “Ubuntu” from the Start Menu. Update packages with:

     apt update
 apt upgrade

    Follow the instructions above for installation on Linux (Debian-based distributions).

    Note: You can use cd /mnt/c/Users/<user>, where <user> is your Windows user name to change the current directory to your home directory, and ls to list the directory contents.

macOS

The installation has been tested on macOS Ventura on Intel. Installation on Apple ARM (M1 CPUs and later) may be possible but is untested.

  1. Install MacPorts.

  2. Install required MacPorts packages:

     sudo port install gcc12 hdf5 netcdf netcdf-fortran ecCodes ncarg
 sudo port select --set gcc mp-gcc12

  3. Install required Python packages:

     python3 -m pip install numpy

  4. Install the ALCF:

     python3 -m pip install alcf

Make sure that /Users/<user>/Library/Python/<version>/bin is included in the PATH environment variable if not already, where <user> is your system user name and <version> is the Python version. This path should be printed by the above command. This can be done by adding this line to the file .zprofile in your home directory and restart the Terminal:

PATH="$PATH:/Users/<user>/Library/Python/<version>/bin"

On macOS, the default shell zsh does not work with the command-line syntax of the ALCF. It is highly recommeded to run any alcf commands in the bash shell:

% bash
bash-3.2$ alcf
...

Post-installation

After completing the installation, you should be able to run the ALCF in the terminal:

alcf

should output:

alcf -- Tool for processing of automatic lidar and ceilometer (ALC) data and intercomparison with atmospheric models.
====

Synopsis
--------

    alcf <cmd> [<options>] [<arguments>]
    alcf [<cmd>] --help
    alcf --version

Arguments
---------

- `cmd`: See Commands below.
- `arguments`: Command arguments. Use `alcf <cmd> --help` for more information.
- `options`: Command options (see Options below).

Commands
--------

- `auto`: Peform automatic processing of model or lidar data.
- `calibrate`: Calibrate lidar backscatter.
- `convert`: Convert input instrument or model data to the ALCF standard NetCDF.
- `download`: Download model data.
- `lidar`: Process lidar data.
- `model`: Extract model data at a point or along a track.
- `plot`: Plot lidar data.
- `simulate`: Simulate lidar measurements from model data using COSP.
- `stats`: Calculate cloud occurrence statistics.

Options
-------

- `--debug`: Enable debugging information.
- `--help`: Print general help or help for a command and exit.
- `--version`: Print version and exit.

How to uninstall ALCF

To uninstall ALCF on Linux:

pipx uninstall alcf
rm ~/.local/share/man/man1/alcf*.1

To uninstall ALCF on macOS with Anaconda:

pip uninstall alcf

Building from the source code

If you want to build the ALCF from the source code, download the source code from GitHub or one of the releases below, and run the following commands in the source code directory:

./download_cosp
pipx install . # Replace pipx with pip for installation in Anaconda.
mkdir -p ~/.local/share/man/man1
ln -s ~/.local/pipx/venvs/alcf/share/man/man1/alcf*.1 ~/.local/share/man/man1/

This will download and unpack ALCF-COSP (a version of COSP with support for ground-based lidars), and compile and install the ALCF. Use this option if you want to customise any parts of the ALCF.

Preparing a source distribution

To prepare a source distribution:

./download_cosp
python3 setup.py sdist

You might have to replace python3 with python depending on the Python distribution.

The resulting package can be found in dist/alcf-<version>.tar.gz. This is the type of package available on PyPI.

Releases

Below is a list of releases of the ALCF. The version numbers follow the Semantic Versioning. Installation instructions have been changing with versions. Please follow the installation instructions in the documentation of the particular version.

2.1.2 (2024-10-18)

Release notes
  • Fix reading certain Vaisala CL61 files, in which the time dimension is "time" instead of "profile".

2.1.1 (2024-09-09)

Release notes
  • alcf lidar: Fixed handling of required variables.
  • alcf lidar: Added track option for specifying a voyage track.
  • alcf auto lidar: The default blim is now based on the wavelenght.
  • alcf download: Close sessions to prevent memory leaks.
  • alcf downlaod: Fixed end time when the end is at midnight.
  • alcf download: Fixed ERA5 download requests.
  • alcf download: Fixed ERA5 pressure levels in request.
  • alcf download: Fixed longitude wrapping.
  • alcf stats: Fixed undefined backscatter_sd_z variable.
  • alcf stats: Added lon_lim and lat_lim options.
  • alcf stats: Added support for multiple inputs [experimental].
  • alcf stats: Added keep_vars option for calculating averages of arbitrary variables [experimental].
  • alcf simulate: Fixed setting of output metadata.
  • Support for keep_vars for keeping variables.
  • Improved error handling.

2.0.1 (2024-05-08)

Release notes
  • A choice of vertical interpolation functions: area-block, area-linear and linear.
  • New default vertical interpolation function is area-linear (previously area-block). This can increase model cloud occurrence profile by up to a few percent. Total cloud fraction is usually unchanged.
  • Cloud mask is now stored as a floating-point variable. Missing values are stored as NaN. Previously they were stored as 0, which could be confusing.
  • alcf model: Default njobs is now limited to 16.
  • alcf stats: Now fails with an error when no input files are provided.
  • alcf stats: Fixed division by zero.

1.9.0 (2024-05-03)

Release notes
  • Improvements in logging and error reporting.
  • Added --version option to print version and exit.
  • Added generic output file attributes (version and software).
  • alcf model: Empty files are no longer output.
  • alcf model: Check for filename instead of time variable in the UM module to ignore the orography file.
  • alcf stats: Avoid division by zero.
  • Improvements in the documentation.

1.8.1 (2024-04-26)

Release notes
  • alcf download: New command for downloading ERA5 and MERRA-2 data for a point or track.
  • alcf stats: Added cloud base height distribution calculation.
  • alcf auto: All-sky and clear fine backscatter histograms are now plotted.
  • alcf auto: Cloud base height distribution is now plotted.
  • alcf model: ERA5 cl variable is now trimmed to [0, 1] (-0.0 sometimes occurs in the data).
  • alcf model: Fixed handling of negative longitude.
  • alcf model: The option --track_lon_180 is now deprecated. The conversion is done automatically.
  • Division by zero warnings in LR calculation are now prevented.
  • Improvements in the documentation.

1.7.0 (2024-04-08)

Release notes
  • alcf model: New track_gap option to specify how gaps in the track should be detected. This changes the default behaviour to detect gaps for intervals longer than 6 hours. Previously gaps were never detected and the track was interpolated between every adjacent pair of points.
  • alcf stats: Fixed parsing of filter_include and filter_exclude options when only one filter file is supplied.
  • alcf stats: Fixed parsing of track option when multiple tracks are supplied.
  • Improvements in the documentation.

1.6.0 (2024-02-28)

Release notes
  • alcf model: Support for overriding year for situations when model time is different from observation time.
  • alcf model: Support for AMPS GRIB format converted to NetCDF with ncl_convert2nc.
  • alcf model: Support for ICON via Intake-ESM on HEALPix grid (icon_intake_healpix).
  • alcf model: Add support for caching of vgrid files in the icon module.
  • alcf convert: Support for converting AMPS GRIB to NetCDF with ncl_convert2nc.
  • alcf convert: Support for more file extensions, such as .asc for CL31.
  • alcf lidar: New cloud_threshold_exp option for exponentially varying cloud detection threshold.
  • alcf lidar: New options bsd and bsd_z for specifying explicit standard deviation of noise for cloud detection.
  • alcf lidar: New align_output option.
  • alcf lidar: New keep_vars option for keeping specified lidar variables.
  • alcf lidar: Fixed backscatter coupling.
  • alcf lidar: Fixed reading of lon and lat in the default lidar driver.
  • alcf stats: Added time_total output variable.
  • alcf stats: Specifying multiple filters is now supported.
  • Improved documentation.
  • eccodes is now a required package (required by alcf convert jra55).
  • mpl2nc is now a required package (required by alcf convert mpl).

1.5.2 (2023-09-09)

Release notes
  • Fixed installation of manual pages.
  • Fixed output aggregation with short time intervals.
  • Fixed installation on macOS Ventura.
  • aclf lidar: Fixed an array type error with certain values of zres.

1.5.1 (2023-08-19)

Release notes
  • alcf plot: New render option.
  • alcf plot: Fixed closing of temporary plot objects.
  • alcf plot: Fixed time axis tick locations.
  • alcf stats: Support for filtering by file.
  • alcf lidar: Support for adding near-range noise.
  • Fixes related to short output time intervals.
  • Support for installation with pipx.
  • Fixed output metadata.
  • Imporved documentation.

1.5.0 (2023-04-22)

Release notes
  • Fixed calculation of time and time bounds in model drivers.
  • Require ds-format 3.6.1, which fixes an issue with readdir.
  • Support for index reading in the AMPS model driver.
  • Less verbose output from the COSP simulator.

1.4.1 (2023-04-21)

Release notes
  • Fixed a missing function argument in model drivers.

1.4.0 (2023-04-21)

Release notes
  • IMPORTANT: Fixed temperature reading in the AMPS driver. Temperature was read as 32 K colder than it should, resulting in a relatively small but potentially significant error in lidar cloud occurrence calculated from AMPS data.
  • Fixed skip option handling.
  • Added support for ARM format of CL51 data.
  • Parallel processing of model input.
  • Partial time limiting support in lidar drivers using the time option.
  • Faster interpolation implemented in Cython.
  • Speed improvements in the ICON driver.
  • Improvements in documentation.

1.3.1 (2023-03-16)

Release notes
  • Fixed issues with a new version of NumPy.
  • Minor improvements in the documentation.

1.3.0 (2023-03-15)

Release notes
  • Support for the ICON model.
  • Minor improvements in the documentation.

1.2.1 (2023-01-30)

Release notes
  • Support for reading Vaisala CL61 data files with zero-dimensional elevation variables.

1.2.0 (2022-11-22)

Release notes
  • Recursive directory processing options in alcf convert, alcf lidar and alcf model.
  • Support for double-dash command line delimiter.
  • cl51: Reading of files with arbitrary time units.
  • alcf simulate NetCDF matadata.
  • Improved documentation.

1.1.4 (2021-12-12)

Release notes
  • Use proleptic Gregorian calendar for time variables.
  • Include required fonts.

1.1.2 (2021-11-30)

Release notes
  • Simplified installation by removing a dependency on CMOR.

1.1.0 (2021-06-29) [documentation] [DOI: 10.5281/zenodo.5153867]

Release notes
  • Support for Vaisala CL61 and NetCDF files produced by BL-VIEW.
  • Improved documentation.

1.0.1 (2021-02-24) [documentation] [DOI: 10.5281/zenodo.5036683]

Release notes
  • Fixed download links to dependencies (udunits archive was removed upstream).

1.0.0 (2021-01-02) [documentation] [DOI: 10.5281/zenodo.4411633]

Release notes
  • First stable release. No change from 1.0.0-beta.3.

1.0.0-beta.3 (2020-10-15) [documentation] [DOI: 10.5281/zenodo.4088217]

Release notes
  • alcf lidar option for coupling of observed data with simulated molecular backscatter.
  • Removal of molecular backscatter in plots (if available).
  • alcf stats filter option now supports "night" and "day" and passing of multiple arguments.
  • New lidar type "default" for re-processing of already processed lidar data.
  • Support for plotting of model cloud liquid water, ice content and cloud fraction.
  • Calculation of lidar ratio changed to effective lidar ratio.
  • Backscatter plots now show effective lidar ratio and cloud mask by default.
  • Changed default vlim for backscatter plots to { 0.1 200 } and default sigma to 5.
  • Output files names are now without colons, which are not compatible with Windows.
  • More accurate plot labels.
  • Improved time sampling: exact profile time bounds are used from weighting.
  • Improved handling of errors and stopping with Ctrl+C.
  • Improved NetCDF metadata.
  • Improved compatibility with newer versions of matplotlib.
  • Fixed clearing of figures in alcf plot backscatter.

1.0.0-beta.2 (2020-05-01) [documentation] [DOI: 10.5281/zenodo.3779518]

Release notes
  • Initial beta release.