GitHub

ALCF

Automatic Lidar and Ceilometer
Framework

Installation

Below you can find installation instructions for supported operating systems:

You can also build the ALCF from the source code if you want to modify any of its parts.

Linux

The instructions below assume that you enter the commands in the terminal.

  1. Install required system packages.

    On Debian-based Linux distributions (Ubuntu, Debian, Devuan, …), install dependencies with:

     sudo apt install gcc make gfortran libhdf5-dev libnetcdf-dev \
         libnetcdff-dev python3 python3-setuptools python3-pip cython3 pipx libeccodes-tools ncl-ncarg
    

    On Fedora, install dependencies with:

     sudo yum install make gcc gfortran hdf5-devel netcdf-devel \
         netcdf-fortran-devel python3-setuptools python3-pip python3-Cython pipx eccodes ncl
    
  2. Install the ALCF with:

     pipx install alcf
     mkdir -p ~/.local/share/man/man1
     ln -s ~/.local/pipx/venvs/alcf/share/man/man1/alcf*.1 ~/.local/share/man/man1/
    

    Make sure that $HOME/.local/bin is in the PATH environment variable.

Windows

It is recommended to run ALCF on Linux.

Installation on Windows is possible under the “Windows Subsystem for Linux”.

  1. Install “Ubuntu” from the Microsoft Store. You might have to enable “Windows Subsystem for Linux” under “Windows Features” first.

  2. Open “Ubuntu” from the Start Menu. Update packages with:

     apt update
     apt upgrade
    

    Follow the instructions above for installation on Linux (Debian-based distributions).

    Note: You can use cd /mnt/c/Users/<user>, where <user> is your Windows user name to change the current directory to your home directory, and ls to list the directory contents.

macOS

The installation has been tested on macOS Ventura on Intel. Installation on Apple ARM (M1 CPUs and later) may be possible but is untested.

  1. Install MacPorts.

  2. Install required MacPorts packages:

     sudo port install gcc12 hdf5 netcdf netcdf-fortran ecCodes ncarg
     sudo port select --set gcc mp-gcc12
    
  3. Install required Python packages:

     python3 -m pip install numpy
    
  4. Install the ALCF:

     python3 -m pip install alcf
    

Make sure that /Users/<user>/Library/Python/<version>/bin is included in the PATH environment variable if not already, where <user> is your system user name and <version> is the Python version. This path should be printed by the above command. This can be done by adding this line to the file .zprofile in your home directory and restart the Terminal:

PATH="$PATH:/Users/<user>/Library/Python/<version>/bin"

On macOS, the default shell zsh does not work with the command-line syntax of the ALCF. It is highly recommeded to run any alcf commands in the bash shell:

% bash
bash-3.2$ alcf
...

Post-installation

After completing the installation, you should be able to run the ALCF in the terminal:

alcf

should output:

alcf -- Tool for processing of automatic lidar and ceilometer (ALC) data and intercomparison with atmospheric models.
====

Synopsis
--------

    alcf <cmd> [<options>]
    alcf <cmd> --help

Arguments
---------

- `cmd`: See Commands below.
- `options`: Command options.

Commands
--------

- `auto`: Peform automatic processing of model or lidar data.
- `calibrate`: Calibrate lidar backscatter.
- `convert`: Convert input instrument or model data to the ALCF standard NetCDF.
- `lidar`: Process lidar data.
- `model`: Extract model data at a point or along a track.
- `plot`: Plot lidar data.
- `simulate`: Simulate lidar measurements from model data using COSP.
- `stats`: Calculate cloud occurrence statistics.

Options
-------

- `--help`: Print help for command.
- `--debug`: Enable debugging information.

How to uninstall ALCF

To uninstall ALCF on Linux:

pipx uninstall alcf
rm ~/.local/share/man/man1/alcf*.1

To uninstall ALCF on macOS with Anaconda:

pip uninstall alcf

Building from the source code

If you want to build the ALCF from the source code, run the following commands in the source code directory:

./download_cosp
pipx install . # Replace pipx with pip for installation in Anaconda.
mkdir -p ~/.local/share/man/man1
ln -s ~/.local/pipx/venvs/alcf/share/man/man1/alcf*.1 ~/.local/share/man/man1/

This will download and unpack ALCF-COSP (a version of COSP with support for ground-based lidars), and compile and install the ALCF. Use this option if you want to customise any parts of the ALCF.

Preparing a source distribution

To prepare a source distribution:

./download_cosp
python3 setup.py sdist

You might have to replace python3 with python depending on the Python distribution.

The resulting package can be found in dist/alcf-<version>.tar.gz. This is the type of package available on PyPI.

Releases

Below is a list of releases of the ALCF. The version numbers follow the Semantic Versioning. Installation instructions have been changing with versions. Please follow the installation instructions in the documentation of the particular version.

1.6.0 (2024-02-28)

Release notes
  • alcf model: Support for overriding year for situations when model time is different from observation time.
  • alcf model: Support for AMPS GRIB format converted to NetCDF with ncl_convert2nc.
  • alcf model: Support for ICON via Intake-ESM on HEALPix grid (icon_intake_healpix).
  • alcf model: Add support for caching of vgrid files in the icon module.
  • alcf convert: Support for converting AMPS GRIB to NetCDF with ncl_convert2nc.
  • alcf convert: Support for more file extensions, such as .asc for CL31.
  • alcf lidar: New cloud_threshold_exp option for exponentially varying cloud detection threshold.
  • alcf lidar: New options bsd and bsd_z for specifying explicit standard deviation of noise for cloud detection.
  • alcf lidar: New align_output option.
  • alcf lidar: New keep_vars option for keeping specified lidar variables.
  • alcf lidar: Fixed backscatter coupling.
  • alcf lidar: Fixed reading of lon and lat in the default lidar driver.
  • alcf stats: Added time_total output variable.
  • alcf stats: Specifying multiple filters is now supported.
  • Improved documentation.
  • eccodes is now a required package (required by alcf convert jra55).
  • mpl2nc is now a required package (required by alcf convert mpl).

1.5.2 (2023-09-09)

Release notes
  • Fixed installation of manual pages.
  • Fixed output aggregation with short time intervals.
  • Fixed installation on macOS Ventura.
  • aclf lidar: Fixed an array type error with certain values of zres.

1.5.1 (2023-08-19)

Release notes
  • alcf plot: New render option.
  • alcf plot: Fixed closing of temporary plot objects.
  • alcf plot: Fixed time axis tick locations.
  • alcf stats: Support for filtering by file.
  • alcf lidar: Support for adding near-range noise.
  • Fixes related to short output time intervals.
  • Support for installation with pipx.
  • Fixed output metadata.
  • Imporved documentation.

1.5.0 (2023-04-22)

Release notes
  • Fixed calculation of time and time bounds in model drivers.
  • Require ds-format 3.6.1, which fixes an issue with readdir.
  • Support for index reading in the AMPS model driver.
  • Less verbose output from the COSP simulator.

1.4.1 (2023-04-21)

Release notes
  • Fixed a missing function argument in model drivers.

1.4.0 (2023-04-21)

Release notes
  • IMPORTANT: Fixed temperature reading in the AMPS driver. Temperature was read as 32 K colder than it should, resulting in a relatively small but potentially significant error in lidar cloud occurrence calculated from AMPS data.
  • Fixed skip option handling.
  • Added support for ARM format of CL51 data.
  • Parallel processing of model input.
  • Partial time limiting support in lidar drivers using the time option.
  • Faster interpolation implemented in Cython.
  • Speed improvements in the ICON driver.
  • Improvements in documentation.

1.3.1 (2023-03-16)

Release notes
  • Fixed issues with a new version of NumPy.
  • Minor improvements in the documentation.

1.3.0 (2023-03-15)

Release notes
  • Support for the ICON model.
  • Minor improvements in the documentation.

1.2.1 (2023-01-30)

Release notes
  • Support for reading Vaisala CL61 data files with zero-dimensional elevation variables.

1.2.0 (2022-11-22)

Release notes
  • Recursive directory processing options in alcf convert, alcf lidar and alcf model.
  • Support for double-dash command line delimiter.
  • cl51: Reading of files with arbitrary time units.
  • alcf simulate NetCDF matadata.
  • Improved documentation.

1.1.4 (2021-12-12)

Release notes
  • Use proleptic Gregorian calendar for time variables.
  • Include required fonts.

1.1.2 (2021-11-30)

Release notes
  • Simplified installation by removing a dependency on CMOR.

1.1.0 (2021-06-29) [documentation] [DOI: 10.5281/zenodo.5153867]

Release notes
  • Support for Vaisala CL61 and NetCDF files produced by BL-VIEW.
  • Improved documentation.

1.0.1 (2021-02-24) [documentation] [DOI: 10.5281/zenodo.5036683]

Release notes
  • Fixed download links to dependencies (udunits archive was removed upstream).

1.0.0 (2021-01-02) [documentation] [DOI: 10.5281/zenodo.4411633]

Release notes
  • First stable release. No change from 1.0.0-beta.3.

1.0.0-beta.3 (2020-10-15) [documentation] [DOI: 10.5281/zenodo.4088217]

Release notes
  • alcf lidar option for coupling of observed data with simulated molecular backscatter.
  • Removal of molecular backscatter in plots (if available).
  • alcf stats filter option now supports "night" and "day" and passing of multiple arguments.
  • New lidar type "default" for re-processing of already processed lidar data.
  • Support for plotting of model cloud liquid water, ice content and cloud fraction.
  • Calculation of lidar ratio changed to effective lidar ratio.
  • Backscatter plots now show effective lidar ratio and cloud mask by default.
  • Changed default vlim for backscatter plots to { 0.1 200 } and default sigma to 5.
  • Output files names are now without colons, which are not compatible with Windows.
  • More accurate plot labels.
  • Improved time sampling: exact profile time bounds are used from weighting.
  • Improved handling of errors and stopping with Ctrl+C.
  • Improved NetCDF metadata.
  • Improved compatibility with newer versions of matplotlib.
  • Fixed clearing of figures in alcf plot backscatter.

1.0.0-beta.2 (2020-05-01) [documentation] [DOI: 10.5281/zenodo.3779518]

Release notes
  • Initial beta release.