NAWDEX Analysis Tools

NAWDEX Analysis Toolset

General Description

This is a set of tools developed for the analysis of ICON simulations during the NAWDEX campaign. The study is done together with Aiko Voigt (KIT) and will focus on observed and simulated cloud radiative effects in the Northern Atlantic and their dependence on grid spacing and microphysics parameterization.

Data

Two types of data go into the study: (i) observational data derived from Meteosat SEVIRI including cloud mask, cloud typing and long- and short-wave radiative fluxes, and (ii) simulation data obtained from differently configured ICON runs.

Method

The simulation data will be transfered to observation space. Thereafter, synthetic cloud mask and typing is derived via the NWCSAF software. Cloud radiative effects are calculated. For observations, simulated clear-sky fluxes are used. More details about the workflow are described here.

Paper

The paper

Senf, F., A. Voigt, N. Clerbaux, A. Hünerbein, and H. Deneke, Does increasing resolution and resolving convection improve simulated cloud-radiative effects over the North Atlantic?, J. Geophys. Res. Atmos., submitted in Jan. 2020

uses the package nawdex_analysis

  • to read (level-1.5) observation and simulation data (radiation fluxes and cloud types)
  • to derived regridded (satellite grid) data to be analyzed (level-2)
  • to performed statistical analysis (level-3)
  • to make the plots / plot styles shown in the paper

The notebooks used for the paper plot are provide here

Typical Workflow

We have two target quantities which will be derived from the analysis

  • net TOA radiation fields including the so-called cloud-radiative effect (CRE)
  • cloud typing (also based on TOA radiation observation

Both analysis targets will be derived from observations (esp. using Meteosat data) nad from simulation using ICON-NWP data.

Analysis of Simulation Data

Simulation data are given on a icosahedral grid with different grid spacings. The following analysis steps are explained

For the simulation data, the cloud typing aspect is more complex.

Analysis of Observation Data

The observational data are derived from Meteosat. The following analysis steps are explained

For the observation data, the CRE aspect is more complex.

Observed Cloud Typing

Simulation Data (on mistral)

General File Location

All simulation files are saved on mistral under

/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP

The following list is available:

find /work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP -type d -iname 'nawdexnwp*mis*'| sort
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0011
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0012
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0011
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0012
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0011
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0012
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0001_latbc
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0001_with_icon-2.1.00
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0002_latbc
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0010

Georeference Mapping

A mapper has been written that generates the georeference file name based on the forecast / simulation filename. All ICON geo-ref files are collected at

/work/bm0834/b380459/NAWDEX/grids

Synthetic Satellite Data Calculation

The ICON simulation data have been transfered to DKRZ Mistral by Aiko Voigt. The “Synsat” forward operator has been applied to the simulation data using the script

/pf/b/b380352/proj/2015-12_synsat/synsats4icon_nawdex/synsat_icon-nawdex.py

The script takes the fg filename as argument. The calculation are restricted to a satellite zenith angle of <75 degrees. The synsat data are saved as brightness temperatures for several infrared SEVIRI bands into hdf5 files. They are vectors using the mask = zen < 75 and are discretized as 0.01 Kelvin. Hence, the georeference and the masking condition is needed to recover the synsat data on the native model grid (which is also a vector in our case).

One example is e.g.

cd data/synsat/nawdex/nawdexnwp-20km-mis-0001/

h5ls -r synsat_nawdexnwp-20km-mis-0001_2016092200_fg_DOM01_ML_0001_0_88576.h5

giving the output

/                        Group
/mcfarq_rescale_noccthresh Group
/mcfarq_rescale_noccthresh/IR_087 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/IR_108 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/IR_120 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/IR_134 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/WV_062 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/WV_073 Dataset {1, 88576}

Thus, six different channels are included.

SynSat Data Regridding

For regridding of synsat data, two steps are needed.

  1. Model Georeference is input and satellite zenith mask is calculated.
  2. Model grid is mapped to SEVIRI grid.

Step one seems to be easy. Step two is a bit more complicated. One could generate and store a nearest neighbor index for the coarser model grids, but for the finer grid an other method should be applied. If several model grid box centers fall into one SEVIRI pixel then all radiances should be averaged (similar to an rediation-based averaging a satellite sensor would do).

Two sets of functions have been developed to regrid data, all are contained in the package reprojassessed by

>>>import nawdex_analysis.io.reproj as reproj
NN interpolation

The first set treats the nearest-neighbor-interpolation problem. Two steps are needed. First an interpolation index is calculated with

>>>index =  reproj.get_vector2msevi_index( vgeo )

where vgeo is a dictionary containing the geo reference. And second, the index (and some masking) is applied to a dataset (also dict).

>>>dset_nn_reproj =  reproj.nn_reproj_with_index( dset, ind )

The index must only be calculated once and can be applied thereafter in a very efficient way.

Box-average interpolation

The second set treats the box-average interpolation which is not trivial because the target grid is not rectangular. The calculation is also divided into two steps. First, reprojection parameters (depending on the grid) are calculated (similar to the nn index) with

>>>rparam = reproj.get_vector2msevi_rparam( vgeo )

where vgeo is again a dictionary containing the geo reference. A field is reprojected with

>>>fint = reproj.reproj_field(f, rparam)
Combined interpolation

The box averaging only fills grid boxes of the target grid that contain values from the initial simulation grid. For a coarse initial simulation grid, box averging results in a sparse dataset with missing values filled with Nans. Therefore, a combination of NN interpolation and box averaging is applied to a data set (again dict) using

>>>dset_inter = reproj.combined_reprojection( dset, ind, rparam )

Observational Data (on TROPOS altair)

TOA Radiation fluxes

TOA Radiation fluxes have been provided by Nicola Clerbaux (Belgium) as hdf5 files. The data are saved as H5T_STD_I16BE which means signed 16-bit integer (big endian). On TROPOS altair, TOA radiation data are saved under

/vols/talos/home/fabian/data/gerb-like

A NaN values seems to be set to -32767 (and need to be masked) and all other values have to be multiplied by 0.25 get to radiation fluxes with unit Wm**(-2).

Input of Data

Functions to read and cutout TOA radiation flux data are available under pyhton package nawdex_analysis.io.input_obs.

The calls

>>>import nawdex_analysis.io.input_obs as ioobs
>>>lwf, swf_net = ioobs.read_radiation_fluxes( t )

read observed long-wave fluxes (lwf) and net short-wave flux (swf_net) scaled to SEVIRI grid.

Cutout of data

We only use a certain SEVIRI cutout for data analysis. This variable is saved under nawdex_analysis.config. An option to input cutted data on-the-fly is incorporated into ioobs.read_radiation_fluxes( t ).

Output daily stacks to netcdf

To ease later analysis, daily stacks of TOA radiation flux data have been saved into netcdf format using

>>>import nawdex_analysis.io.output_obs as oobs
>>>oobs.save_radflux_tstack( date )

SEVIRI BT data

SEVIRI BT are directly read out of the TROPOS archive at altair using the ioobs.read_msevi(t1,t2) where t1 and t2 are start and end time of the BT data stack.

The finally daily BT fields are output into netcdf files which are generated by

>>>import nawdex_analysis.io.output_obs as oobs
>>>oobs.save_meteosat_tstack( date )

where date is a date string.

Simulated Cloud Typing

Synthetic Satellite Data Calculation (Mistral)

The ICON simulation data have been transfered to DKRZ Mistral by Aiko Voigt. The “Synsat” forward operator has been applied to the simulation data using the script

/pf/b/b380352/proj/2015-12_synsat/synsats4icon_nawdex/synsat_icon-nawdex.py

The script takes the fg filename as argument. The calculation are restricted to a satellite zenith angle of <75 degrees. The synsat data are saved as brightness temperatures for several infrared SEVIRI bands into hdf5 files. They are vectors using the mask = zen < 75 and are discretized as 0.01 Kelvin. Hence, the georeference and the masking condition is needed to recover the synsat data on the native model grid (which is also a vector in our case).

One example is e.g.

cd ~/data/synsat/nawdex/nawdexnwp-20km-mis-0001/

h5ls -r synsat_nawdexnwp-20km-mis-0001_2016092200_fg_DOM01_ML_0001_0_88576.h5

giving the output

/                        Group
/mcfarq_rescale_noccthresh Group
/mcfarq_rescale_noccthresh/IR_087 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/IR_108 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/IR_120 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/IR_134 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/WV_062 Dataset {1, 88576}
/mcfarq_rescale_noccthresh/WV_073 Dataset {1, 88576}

Thus, six different channels are included.

SynSat Data Regridding

Regridding of synsat data is done with the tools described here.

Similar to the radiation fluxes, synthetic sat data are regridded onto MSG grid using a python script on Mistral:

cd ~/proj/2017-07_nawdex_analysis/inout
./save_reproj_synsat.py /pf/b/b380352/data/synsat/nawdex/nawdexnwp-80km-mis-0001

Applying Cloud Typing to the Synthetic Sat Data (TROPOS)

Synsat @ TROPOS

The synsat data have been transfered to TROPOS and are stored under

ls -1 /vols/talos/home/fabian/data/icon/nawdex/synsat/synsat*nc
synsat-nawdexnwp-10km-mis-0001.nc
synsat-nawdexnwp-10km-mis-0002.nc
synsat-nawdexnwp-10km-mis-0003.nc
synsat-nawdexnwp-10km-mis-0004.nc
synsat-nawdexnwp-10km-mis-0005.nc
synsat-nawdexnwp-10km-mis-0006.nc
synsat-nawdexnwp-10km-mis-0007.nc
synsat-nawdexnwp-10km-mis-0008.nc
synsat-nawdexnwp-10km-mis-0009.nc
synsat-nawdexnwp-10km-mis-0010.nc
synsat-nawdexnwp-10km-mis-0011.nc
synsat-nawdexnwp-10km-mis-0012.nc
synsat-nawdexnwp-20km-mis-0001.nc
synsat-nawdexnwp-20km-mis-0002.nc
synsat-nawdexnwp-20km-mis-0003.nc
synsat-nawdexnwp-20km-mis-0004.nc
synsat-nawdexnwp-20km-mis-0005.nc
synsat-nawdexnwp-20km-mis-0006.nc
synsat-nawdexnwp-20km-mis-0007.nc
synsat-nawdexnwp-20km-mis-0008.nc
synsat-nawdexnwp-20km-mis-0009.nc
synsat-nawdexnwp-20km-mis-0010.nc
synsat-nawdexnwp-2km-mis-0001.nc
synsat-nawdexnwp-2km-mis-0002.nc
synsat-nawdexnwp-2km-mis-0003.nc
synsat-nawdexnwp-2km-mis-0004.nc
synsat-nawdexnwp-2km-mis-0005.nc
synsat-nawdexnwp-2km-mis-0006.nc
synsat-nawdexnwp-2km-mis-0007.nc
synsat-nawdexnwp-2km-mis-0008.nc
synsat-nawdexnwp-2km-mis-0009.nc
synsat-nawdexnwp-2km-mis-0010.nc
synsat-nawdexnwp-2km-mis-0011.nc
synsat-nawdexnwp-2km-mis-0012.nc
synsat-nawdexnwp-40km-mis-0001.nc
synsat-nawdexnwp-40km-mis-0002.nc
synsat-nawdexnwp-40km-mis-0003.nc
synsat-nawdexnwp-40km-mis-0004.nc
synsat-nawdexnwp-40km-mis-0005.nc
synsat-nawdexnwp-40km-mis-0006.nc
synsat-nawdexnwp-40km-mis-0007.nc
synsat-nawdexnwp-40km-mis-0008.nc
synsat-nawdexnwp-40km-mis-0009.nc
synsat-nawdexnwp-40km-mis-0010.nc
synsat-nawdexnwp-5km-mis-0001.nc
synsat-nawdexnwp-5km-mis-0002.nc
synsat-nawdexnwp-5km-mis-0003.nc
synsat-nawdexnwp-5km-mis-0004.nc
synsat-nawdexnwp-5km-mis-0005.nc
synsat-nawdexnwp-5km-mis-0006.nc
synsat-nawdexnwp-5km-mis-0007.nc
synsat-nawdexnwp-5km-mis-0008.nc
synsat-nawdexnwp-5km-mis-0009.nc
synsat-nawdexnwp-5km-mis-0010.nc
synsat-nawdexnwp-5km-mis-0011.nc
synsat-nawdexnwp-5km-mis-0012.nc
synsat-nawdexnwp-80km-mis-0001.nc
synsat-nawdexnwp-80km-mis-0002.nc
synsat-nawdexnwp-80km-mis-0003.nc
synsat-nawdexnwp-80km-mis-0004.nc
synsat-nawdexnwp-80km-mis-0005.nc
synsat-nawdexnwp-80km-mis-0006.nc
synsat-nawdexnwp-80km-mis-0007.nc
synsat-nawdexnwp-80km-mis-0008.nc
synsat-nawdexnwp-80km-mis-0009.nc
synsat-nawdexnwp-80km-mis-0010.nc

An example content is

ncdump -h /vols/talos/home/fabian/data/icon/nawdex/synsat/synsat-nawdexnwp-80km-mis-0010.nc
netcdf synsat-nawdexnwp-80km-mis-0010 {
dimensions:
    time = 96 ;
    rows = 1004 ;
    cols = 2776 ;
    ndim = 2 ;
variables:
    short bt120(time, rows, cols) ;
        bt120:_FillValue = 0s ;
        bt120:units = "K" ;
        bt120:long_name = "Synthetic MSG SEVIRI Brightness Temperatures at 12.0 um" ;
        bt120:coordinates = "lat lon" ;
        bt120:scale_factor = 0.01 ;
    int64 msevi_region(ndim, ndim) ;
        msevi_region:long_name = "msg_seviri_region_cutout" ;
        msevi_region:description = "((row1, row2), (col1, col2))" ;
    short bt134(time, rows, cols) ;
        bt134:_FillValue = 0s ;
        bt134:units = "K" ;
        bt134:long_name = "Synthetic MSG SEVIRI Brightness Temperatures at 13.4 um" ;
        bt134:coordinates = "lat lon" ;
        bt134:scale_factor = 0.01 ;
    double zen(rows, cols) ;
        zen:units = "degrees" ;
        zen:lon0 = 0. ;
        zen:long_name = "satellite_zenith_angle" ;
        zen:coordinates = "lat lon" ;
    int64 nwcsaf_region(ndim, ndim) ;
        nwcsaf_region:long_name = "nwcsaf_region_cutout" ;
        nwcsaf_region:description = "((row_center, col_center), (nrows, ncols))" ;
    short bt087(time, rows, cols) ;
        bt087:_FillValue = 0s ;
        bt087:units = "K" ;
        bt087:long_name = "Synthetic MSG SEVIRI Brightness Temperatures at 8.7 um" ;
        bt087:coordinates = "lat lon" ;
        bt087:scale_factor = 0.01 ;
    short bt108(time, rows, cols) ;
        bt108:_FillValue = 0s ;
        bt108:units = "K" ;
        bt108:long_name = "Synthetic MSG SEVIRI Brightness Temperatures at 10.8 um" ;
        bt108:coordinates = "lat lon" ;
        bt108:scale_factor = 0.01 ;
    short bt073(time, rows, cols) ;
        bt073:_FillValue = 0s ;
        bt073:units = "K" ;
        bt073:long_name = "Synthetic MSG SEVIRI Brightness Temperatures at 7.3 um" ;
        bt073:coordinates = "lat lon" ;
        bt073:scale_factor = 0.01 ;
    short bt062(time, rows, cols) ;
        bt062:_FillValue = 0s ;
        bt062:units = "K" ;
        bt062:long_name = "Synthetic MSG SEVIRI Brightness Temperatures at 6.2 um" ;
        bt062:coordinates = "lat lon" ;
        bt062:scale_factor = 0.01 ;
    double lat(rows, cols) ;
        lat:units = "degrees_north" ;
        lat:long_name = "latitude" ;
    double lon(rows, cols) ;
        lon:units = "degrees_east" ;
        lon:long_name = "longitude" ;
    double time(time) ;
        time:units = "day as %Y%m%d.%f" ;
        time:long_name = "Time" ;
        time:calendar = "proleptic_gregorian" ;

// global attributes:
        :description = "Synthetic Infrared MSG-SEVIRI images from the Prime Service" ;
        :title = "Synthetic MSG-SEVIRI Brightness Temperatures" ;
        :institution = "Leibniz Institute for Tropospheric Research" ;
        :author = "Fabian Senf (senf@tropos.de)" ;
}

NWCSAF patch

The cloud tpying is based on a patched version of the NWCSAF software version 2013. The patch

  • reads netcdf data (BTs)
  • output fake HRIT data with the considered BTs embedded
  • runs NWCSAF at permanent mid-night
  • emulates BTs from the 3.9 um channel (which are contaminated by sun light during day, but required by NWCSAF in night-mode)

The patching is done with the toolset hosted under https://gitea.tropos.de/senf/proj.synNWCSAF

Further details are explained there.

Synthetic cloud types

Cloud type data are stored under

ls -1 /vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf*nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0001.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0002.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0003.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0004.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0005.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0006.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0007.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0008.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0009.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0010.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0011.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-10km-mis-0012.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0001.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0002.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0003.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0004.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0005.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0006.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0007.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0008.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0009.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-20km-mis-0010.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0001.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0002.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0003.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0004.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0005.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0006.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0007.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0008.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0009.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0010.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0011.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-2km-mis-0012.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0001.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0002.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0003.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0004.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0005.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0006.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0007.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0008.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0009.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-40km-mis-0010.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0001.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0002.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0003.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0004.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0005.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0006.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0007.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0008.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0009.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0010.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0011.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-5km-mis-0012.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0001.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0002.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0003.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0004.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0005.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0006.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0007.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0008.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0009.nc
/vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0010.nc

Example content is

ncdump -h /vols/talos/home/fabian/data/icon/nawdex/synsat/nwcsaf_synsat-nawdexnwp-80km-mis-0010.nc
netcdf nwcsaf_synsat-nawdexnwp-80km-mis-0010 {
dimensions:
    ndim = 2 ;
    time = 96 ;
    rows = 1004 ;
    cols = 2776 ;
variables:
    int64 msevi_region(ndim, ndim) ;
        msevi_region:long_name = "msg_seviri_region_cutout" ;
        msevi_region:description = "((row1, row2), (col1, col2))" ;
    int64 nwcsaf_region(ndim, ndim) ;
        nwcsaf_region:long_name = "nwcsaf_region_cutout" ;
        nwcsaf_region:description = "((row_center, col_center), (nrows, ncols))" ;
    short CTTH_HEIGHT(time, rows, cols) ;
        CTTH_HEIGHT:_FillValue = -1s ;
        CTTH_HEIGHT:units = "km" ;
        CTTH_HEIGHT:long_name = "cloud top height" ;
        CTTH_HEIGHT:description = "" ;
        CTTH_HEIGHT:coordinates = "lat lon" ;
        CTTH_HEIGHT:scale_factor = 200LL ;
    short CMa(time, rows, cols) ;
        CMa:_FillValue = -1s ;
        CMa:units = "km" ;
        CMa:long_name = "cloud top height" ;
        CMa:description = "" ;
        CMa:coordinates = "lat lon" ;
        CMa:scale_factor = 1LL ;
    short CT(time, rows, cols) ;
        CT:_FillValue = -1s ;
        CT:units = "km" ;
        CT:long_name = "cloud top height" ;
        CT:description = "" ;
        CT:coordinates = "lat lon" ;
        CT:scale_factor = 1LL ;
    double lat(rows, cols) ;
        lat:_FillValue = NaN ;
        lat:units = "degrees_north" ;
        lat:long_name = "latitutde" ;
    double lon(rows, cols) ;
        lon:_FillValue = NaN ;
        lon:units = "degrees_east" ;
        lon:long_name = "longitude" ;
    double time(time) ;
        time:_FillValue = NaN ;
        time:units = "day as %Y%m%d.%f" ;
        time:long_name = "Time" ;
        time:calendar = "proleptic_gregorian" ;

// global attributes:
        :description = "Based on synthetic Meteosat-SEVIRI images from the Prime Service" ;
        :title = "NWCSAF cloud products" ;
        :institution = "Leibniz Institute for Tropospheric Research" ;
        :author = "Fabian Senf (senf@tropos.de)" ;
}

Observed TOA Radiation Flux Data and CRE (on TROPOS altair)

TOA allsky Radiation Fluxes

TOA Radiation fluxes have been provided by Nicola Clerbaux (Belgium) as hdf5 files. The data are saved as H5T_STD_I16BE which means signed 16-bit integer (big endian). On TROPOS altair, TOA radiation data are saved under

/vols/talos/home/fabian/data/gerb-like

A NaN values seems to be set to -32767 (and need to be masked) and all other values have to be multiplied by 0.25 get to radiation fluxes with unit Wm**(-2).

Input of Data

Functions to read and cutout TOA radiation flux data are available under pyhton package nawdex_analysis.io.input_obs.

The calls

>>>import nawdex_analysis.io.input_obs as ioobs
>>>lwf, swf_net = ioobs.read_radiation_fluxes( t )

read observed long-wave fluxes (lwf) and net short-wave flux (swf_net) scaled to SEVIRI grid.

Cutout of data

We only use a certain SEVIRI cutout for data analysis. This variable is saved under nawdex_analysis.config. An option to input cutted data on-the-fly is incorporated into ioobs.read_radiation_fluxes( t ).

Output daily stacks to netcdf

To ease later analysis, daily stacks of TOA radiation flux data have been saved into netcdf format using

>>>import nawdex_analysis.io.output_obs as oobs
>>>oobs.save_radflux_tstack( date )

Run Script for Regridding and NetCDF Output

A python script on TROPOS is used for regridding and output of observed radiation fluxes:

cd /pf/b/b380352/proj/2017-07_nawdex_analysis/inout/
./save_toa_radflux4nawdex.py 20160930

Final Allsky Radiation Data

These daily data stacks are saved under:

ls -1 /vols/talos/home/fabian/data/icon/nawdex/gerb-like/
toa_radflux-nawdex-20160915.nc
toa_radflux-nawdex-20160916.nc
toa_radflux-nawdex-20160917.nc
toa_radflux-nawdex-20160918.nc
toa_radflux-nawdex-20160919.nc
toa_radflux-nawdex-20160920.nc
toa_radflux-nawdex-20160921.nc
toa_radflux-nawdex-20160922.nc
toa_radflux-nawdex-20160923.nc
toa_radflux-nawdex-20160924.nc
toa_radflux-nawdex-20160925.nc
toa_radflux-nawdex-20160926.nc
toa_radflux-nawdex-20160927.nc
toa_radflux-nawdex-20160928.nc
toa_radflux-nawdex-20160929.nc
toa_radflux-nawdex-20160930.nc
toa_radflux-nawdex-20161001.nc
toa_radflux-nawdex-20161002.nc
toa_radflux-nawdex-20161003.nc
toa_radflux-nawdex-20161004.nc
toa_radflux-nawdex-20161005.nc
toa_radflux-nawdex-20161006.nc
toa_radflux-nawdex-20161007.nc
toa_radflux-nawdex-20161008.nc
toa_radflux-nawdex-20161009.nc
toa_radflux-nawdex-20161010.nc
toa_radflux-nawdex-20161011.nc
toa_radflux-nawdex-20161012.nc
toa_radflux-nawdex-20161013.nc
toa_radflux-nawdex-20161014.nc

An example is

ncdump -h /vols/talos/home/fabian/data/icon/nawdex/gerb-like/toa_radflux-nawdex-20160930.nc
netcdf toa_radflux-nawdex-20160930 {
dimensions:
    rows = 1004 ;
    cols = 2776 ;
    time = 24 ;
variables:
    short swf_up(time, rows, cols) ;
        swf_up:_FillValue = 0s ;
        swf_up:units = "W m**(-2)" ;
        swf_up:long_name = "TOA short-wave upwelling radiation flux" ;
        swf_up:coordinates = "lat lon" ;
        swf_up:scale_factor = 0.25 ;
    short swf_net(time, rows, cols) ;
        swf_net:_FillValue = 0s ;
        swf_net:units = "W m**(-2)" ;
        swf_net:long_name = "TOA short-wave net radiation flux" ;
        swf_net:coordinates = "lat lon" ;
        swf_net:scale_factor = 0.25 ;
    short lwf(time, rows, cols) ;
        lwf:_FillValue = 0s ;
        lwf:units = "W m**(-2)" ;
        lwf:long_name = "TOA long-wave radiation flux" ;
        lwf:coordinates = "lat lon" ;
        lwf:scale_factor = 0.25 ;
    float lat(rows, cols) ;
        lat:_FillValue = NaNf ;
        lat:units = "degrees_north" ;
        lat:long_name = "latitude" ;
    float lon(rows, cols) ;
        lon:_FillValue = NaNf ;
        lon:units = "degrees_east" ;
        lon:long_name = "longitude" ;
    double time(time) ;
        time:_FillValue = NaN ;
        time:units = "day as %Y%m%d.%f" ;
        time:long_name = "Time" ;
        time:calendar = "proleptic_gregorian" ;

// global attributes:
        :description = "GERB-like TOA radiation flux derived from Meteosat SEVIRI and obtained from Nicolas Clerbaux" ;
        :title = "TOA Radiation Fluxes" ;
        :institution = "Leibniz Institute for Tropospheric Research" ;
        :author = "Fabian Senf (senf@tropos.de)" ;
}

TOA clearsky Radiation Fluxes

Clearsky is challenging for the observation because the investigation area is a very cloudy place and the time record (~one month) is rather short. Hence, methods that rely on temporal stacking and finding a close clearsky pixel are rather inaccurate!

We are going top use simulated clearsky instead (chosen the 10km grid spacing runs) with the drawback that bias corrections have to be applied.

Bias Correction for Shortwave (on Mistral)

Two important points

  • It seems to be that the simulated clearsky ocean is too bright, i.e. outgoing SWF is too large for the simulations. Therefore, we apply a scale factor to reduce the bias.
  • The incoming SWF sensitively depends on the accurate time. The model time is exactly at 0 UTC, but the observational time is delayed by ~12 minutes due to the scanning strategy (from South to North). Therefore, the incoming flux is taken from the observational data (luckily included) and used as net incoming flux.

A general function exists that inputs simulated clearsky fluxes:

import nawdex_analysis.io.input_sim as isim
isim.read_simulated_clearsky_flux??

Python scripts are located on mistral and can be run via

cd /pf/b/b380352/proj/2017-07_nawdex_analysis/inout
./save_retrieved_clearsky_SWF.py $EXPNAME

Bias Correction for Longwave

Bias correction for the longwave is done on the fly (at the moment - for simplicity). An offset of-2 W/m**2 is substracted from the simulation bases estimate. That is, simulaiton is slightly too warm.

Calculation of Average CRE

A general script exists to calculate the average CRE depending on cloud type (hence cloud type has be input as well). It can be run at TROPOS via

cd /vols/talos/home/fabian/proj/2017-07_icon-nawdex/inout
./save_ave_cre.py /vols/talos/home/fabian/data/icon/nawdex/gerb-like/toa_radflux-nawdex-20160922.nc -scaled

The script generates netcdf files which are stored under

ls -1 /vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled*nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160915.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160916.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160917.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160918.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160919.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160920.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160921.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160922.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160923.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160924.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160925.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160926.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160927.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160928.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160929.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160930.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161001.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161002.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161003.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161004.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161005.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161006.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161007.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161008.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161009.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161010.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161011.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161012.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161013.nc
/vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20161014.nc

Example content:

ncdump -h /vols/talos/home/fabian/data/icon/nawdex/statistics/ave_cre-scaled_meteosat-nawdex-20160922.nc
netcdf ave_cre-scaled_meteosat-nawdex-20160922 {
dimensions:
    time = 24 ;
    ct = 11 ;
    string18 = 18 ;
variables:
    int64 time(time) ;
        time:units = "hours since 2016-09-22 00:00:00" ;
        time:calendar = "proleptic_gregorian" ;
    double lcre_ave(time, ct) ;
        lcre_ave:_FillValue = NaN ;
        lcre_ave:units = "W m^{-2}" ;
        lcre_ave:longname = "area-average longwave CRE " ;
    double afrac(time, ct) ;
        afrac:_FillValue = NaN ;
        afrac:units = "%" ;
        afrac:longname = "relative area fractions per cloud type" ;
    double scre_ave(time, ct) ;
        scre_ave:_FillValue = NaN ;
        scre_ave:units = "W m^{-2}" ;
        scre_ave:longname = "area-average shortwave CRE " ;
    char ct(ct, string18) ;
}

Simulated TOA Radiation and Cloud-Radiative Effects

Original Simulation Data (on mistral)

General File Location

All simulation files are saved on mistral under

/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP

The following list is available:

find /work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP -type d -iname 'nawdexnwp*mis*'| sort
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0011
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-10km-mis-0012
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-20km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0011
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0012
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-40km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0010
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0011
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-5km-mis-0012
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0001
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0001_latbc
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0001_with_icon-2.1.00
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0002
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0002_latbc
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0003
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0004
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0005
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0006
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0007
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0008
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0009
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-80km-mis-0010

Radiation Variables

The simulation data provide allsky radiation variable for up- and downwelling shortwave as well as longwave at TOA. Variable keys are

['sod_t', 'sou_t', 'thb_t']

Moreover, access to net clearsky variables is possible.

['swtoaclr', 'lwtoaclr']

The lev1 functions provide direct access to the data:

>>> import nawdex_analysis.io.input_sim as isim
>>> isim.read_icon_rad_vector??

Georeference Mapping

A mapper has been written that generates the georeference file name based on the forecast / simulation filename. All ICON geo-ref files are collected at

/work/bm0834/b380459/NAWDEX/grids

Data Regridding

The simulation data will be regridded onto MSG grid for easy analysis and comparison. For regridding, two steps are needed.

  1. Model Georeference is input and satellite zenith mask is calculated.
  2. Model grid is mapped to SEVIRI grid.

Step one seems to be easy. Step two is a bit more complicated. One could generate and store a nearest neighbor index for the coarser model grids, but for the finer grid an other method should be applied. If several model grid box centers fall into one SEVIRI pixel then all radiances should be averaged (similar to an rediation-based averaging a satellite sensor would do).

Two sets of functions have been developed to regrid data, all are contained in the package reprojassessed by

>>>import nawdex_analysis.io.reproj as reproj
NN interpolation

The first set treats the nearest-neighbor-interpolation problem. Two steps are needed. First an interpolation index is calculated with

>>>index =  reproj.get_vector2msevi_index( vgeo )

where vgeo is a dictionary containing the geo reference. And second, the index (and some masking) is applied to a dataset (also dict).

>>>dset_nn_reproj =  reproj.nn_reproj_with_index( dset, ind )

The index must only be calculated once and can be applied thereafter in a very efficient way.

Box-average interpolation

The second set treats the box-average interpolation which is not trivial because the target grid is not rectangular. The calculation is also divided into two steps. First, reprojection parameters (depending on the grid) are calculated (similar to the nn index) with

>>>rparam = reproj.get_vector2msevi_rparam( vgeo )

where vgeo is again a dictionary containing the geo reference. A field is reprojected with

>>>fint = reproj.reproj_field(f, rparam)
Combined interpolation

The box averaging only fills grid boxes of the target grid that contain values from the initial simulation grid. For a coarse initial simulation grid, box averging results in a sparse dataset with missing values filled with Nans. Therefore, a combination of NN interpolation and box averaging is applied to a data set (again dict) using

>>>dset_inter = reproj.combined_reprojection( dset, ind, rparam )

Final Regridding Loops

The final regridding is done on mistral via the script:

cd ${HOME}/proj/2017-07_nawdex_analysis/inout/
./save_reproj_sim_raddata.py /work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0001

The simulation path is the input argument and the final regridded data are saved under:

ls -1 ${HOME}/data/nawdex/sim-toarad/
toa_clear_radflux-nawdexnwp-10km-mis-0001.nc
toa_clear_radflux-nawdexnwp-10km-mis-0002.nc
toa_clear_radflux-nawdexnwp-10km-mis-0003.nc
toa_clear_radflux-nawdexnwp-10km-mis-0004.nc
toa_clear_radflux-nawdexnwp-10km-mis-0005.nc
toa_clear_radflux-nawdexnwp-10km-mis-0006.nc
toa_clear_radflux-nawdexnwp-10km-mis-0007.nc
toa_clear_radflux-nawdexnwp-10km-mis-0008.nc
toa_clear_radflux-nawdexnwp-10km-mis-0009.nc
toa_clear_radflux-nawdexnwp-10km-mis-0010.nc
toa_clear_radflux-nawdexnwp-10km-mis-0011.nc
toa_clear_radflux-nawdexnwp-10km-mis-0012.nc
toa_clear_radflux-nawdexnwp-20km-mis-0001.nc
toa_clear_radflux-nawdexnwp-20km-mis-0002.nc
toa_clear_radflux-nawdexnwp-20km-mis-0003.nc
toa_clear_radflux-nawdexnwp-20km-mis-0004.nc
toa_clear_radflux-nawdexnwp-20km-mis-0005.nc
toa_clear_radflux-nawdexnwp-20km-mis-0006.nc
toa_clear_radflux-nawdexnwp-20km-mis-0007.nc
toa_clear_radflux-nawdexnwp-20km-mis-0008.nc
toa_clear_radflux-nawdexnwp-20km-mis-0009.nc
toa_clear_radflux-nawdexnwp-20km-mis-0010.nc
toa_clear_radflux-nawdexnwp-2km-mis-0001.nc
toa_clear_radflux-nawdexnwp-2km-mis-0002.nc
toa_clear_radflux-nawdexnwp-2km-mis-0003.nc
toa_clear_radflux-nawdexnwp-2km-mis-0004.nc
toa_clear_radflux-nawdexnwp-2km-mis-0005.nc
toa_clear_radflux-nawdexnwp-2km-mis-0006.nc
toa_clear_radflux-nawdexnwp-2km-mis-0007.nc
toa_clear_radflux-nawdexnwp-2km-mis-0008.nc
toa_clear_radflux-nawdexnwp-2km-mis-0009.nc
toa_clear_radflux-nawdexnwp-2km-mis-0010.nc
toa_clear_radflux-nawdexnwp-2km-mis-0011.nc
toa_clear_radflux-nawdexnwp-2km-mis-0012.nc
toa_clear_radflux-nawdexnwp-40km-mis-0001.nc
toa_clear_radflux-nawdexnwp-40km-mis-0002.nc
toa_clear_radflux-nawdexnwp-40km-mis-0003.nc
toa_clear_radflux-nawdexnwp-40km-mis-0004.nc
toa_clear_radflux-nawdexnwp-40km-mis-0005.nc
toa_clear_radflux-nawdexnwp-40km-mis-0006.nc
toa_clear_radflux-nawdexnwp-40km-mis-0007.nc
toa_clear_radflux-nawdexnwp-40km-mis-0008.nc
toa_clear_radflux-nawdexnwp-40km-mis-0009.nc
toa_clear_radflux-nawdexnwp-40km-mis-0010.nc
toa_clear_radflux-nawdexnwp-5km-mis-0001.nc
toa_clear_radflux-nawdexnwp-5km-mis-0002.nc
toa_clear_radflux-nawdexnwp-5km-mis-0003.nc
toa_clear_radflux-nawdexnwp-5km-mis-0004.nc
toa_clear_radflux-nawdexnwp-5km-mis-0005.nc
toa_clear_radflux-nawdexnwp-5km-mis-0006.nc
toa_clear_radflux-nawdexnwp-5km-mis-0007.nc
toa_clear_radflux-nawdexnwp-5km-mis-0008.nc
toa_clear_radflux-nawdexnwp-5km-mis-0009.nc
toa_clear_radflux-nawdexnwp-5km-mis-0010.nc
toa_clear_radflux-nawdexnwp-5km-mis-0011.nc
toa_clear_radflux-nawdexnwp-5km-mis-0012.nc
toa_clear_radflux-nawdexnwp-80km-mis-0001.nc
toa_clear_radflux-nawdexnwp-80km-mis-0002.nc
toa_clear_radflux-nawdexnwp-80km-mis-0003.nc
toa_clear_radflux-nawdexnwp-80km-mis-0004.nc
toa_clear_radflux-nawdexnwp-80km-mis-0005.nc
toa_clear_radflux-nawdexnwp-80km-mis-0006.nc
toa_clear_radflux-nawdexnwp-80km-mis-0007.nc
toa_clear_radflux-nawdexnwp-80km-mis-0008.nc
toa_clear_radflux-nawdexnwp-80km-mis-0009.nc
toa_clear_radflux-nawdexnwp-80km-mis-0010.nc
toa_radflux-nawdexnwp-10km-mis-0001.nc
toa_radflux-nawdexnwp-10km-mis-0002.nc
toa_radflux-nawdexnwp-10km-mis-0003.nc
toa_radflux-nawdexnwp-10km-mis-0004.nc
toa_radflux-nawdexnwp-10km-mis-0005.nc
toa_radflux-nawdexnwp-10km-mis-0006.nc
toa_radflux-nawdexnwp-10km-mis-0007.nc
toa_radflux-nawdexnwp-10km-mis-0008.nc
toa_radflux-nawdexnwp-10km-mis-0009.nc
toa_radflux-nawdexnwp-10km-mis-0010.nc
toa_radflux-nawdexnwp-10km-mis-0011.nc
toa_radflux-nawdexnwp-10km-mis-0012.nc
toa_radflux-nawdexnwp-20km-mis-0001.nc
toa_radflux-nawdexnwp-20km-mis-0002.nc
toa_radflux-nawdexnwp-20km-mis-0003.nc
toa_radflux-nawdexnwp-20km-mis-0004.nc
toa_radflux-nawdexnwp-20km-mis-0005.nc
toa_radflux-nawdexnwp-20km-mis-0006.nc
toa_radflux-nawdexnwp-20km-mis-0007.nc
toa_radflux-nawdexnwp-20km-mis-0008.nc
toa_radflux-nawdexnwp-20km-mis-0009.nc
toa_radflux-nawdexnwp-20km-mis-0010.nc
toa_radflux-nawdexnwp-2km-mis-0001.nc
toa_radflux-nawdexnwp-2km-mis-0002.nc
toa_radflux-nawdexnwp-2km-mis-0003.nc
toa_radflux-nawdexnwp-2km-mis-0004.nc
toa_radflux-nawdexnwp-2km-mis-0005.nc
toa_radflux-nawdexnwp-2km-mis-0006.nc
toa_radflux-nawdexnwp-2km-mis-0007.nc
toa_radflux-nawdexnwp-2km-mis-0008.nc
toa_radflux-nawdexnwp-2km-mis-0009.nc
toa_radflux-nawdexnwp-2km-mis-0010.nc
toa_radflux-nawdexnwp-2km-mis-0011.nc
toa_radflux-nawdexnwp-2km-mis-0012.nc
toa_radflux-nawdexnwp-40km-mis-0001.nc
toa_radflux-nawdexnwp-40km-mis-0002.nc
toa_radflux-nawdexnwp-40km-mis-0003.nc
toa_radflux-nawdexnwp-40km-mis-0004.nc
toa_radflux-nawdexnwp-40km-mis-0005.nc
toa_radflux-nawdexnwp-40km-mis-0006.nc
toa_radflux-nawdexnwp-40km-mis-0007.nc
toa_radflux-nawdexnwp-40km-mis-0008.nc
toa_radflux-nawdexnwp-40km-mis-0009.nc
toa_radflux-nawdexnwp-40km-mis-0010.nc
toa_radflux-nawdexnwp-5km-mis-0001.nc
toa_radflux-nawdexnwp-5km-mis-0002.nc
toa_radflux-nawdexnwp-5km-mis-0003.nc
toa_radflux-nawdexnwp-5km-mis-0004.nc
toa_radflux-nawdexnwp-5km-mis-0005.nc
toa_radflux-nawdexnwp-5km-mis-0006.nc
toa_radflux-nawdexnwp-5km-mis-0007.nc
toa_radflux-nawdexnwp-5km-mis-0008.nc
toa_radflux-nawdexnwp-5km-mis-0009.nc
toa_radflux-nawdexnwp-5km-mis-0010.nc
toa_radflux-nawdexnwp-5km-mis-0011.nc
toa_radflux-nawdexnwp-5km-mis-0012.nc
toa_radflux-nawdexnwp-80km-mis-0001.nc
toa_radflux-nawdexnwp-80km-mis-0002.nc
toa_radflux-nawdexnwp-80km-mis-0003.nc
toa_radflux-nawdexnwp-80km-mis-0004.nc
toa_radflux-nawdexnwp-80km-mis-0005.nc
toa_radflux-nawdexnwp-80km-mis-0006.nc
toa_radflux-nawdexnwp-80km-mis-0007.nc
toa_radflux-nawdexnwp-80km-mis-0008.nc
toa_radflux-nawdexnwp-80km-mis-0009.nc
toa_radflux-nawdexnwp-80km-mis-0010.nc

Output is splitted between all- and clear-sky.

Regridded Simulation Data (TROPOS store)

Data have been transfered to TROPOS and are actually saved at

${LHOME}/data/icon/nawdex/sim-toarad/

Example Output

An example of the TOA radiation file is given here:

ncdump -h ${LHOME}/data/icon/nawdex/sim-toarad/toa_radflux-nawdexnwp-10km-mis-0001.nc
netcdf toa_radflux-nawdexnwp-10km-mis-0001 {
dimensions:
    time = 96 ;
    rows = 1004 ;
    cols = 2776 ;
variables:
    short swf_up(time, rows, cols) ;
        swf_up:_FillValue = 0s ;
        swf_up:units = "W m**(-2)" ;
        swf_up:long_name = "TOA  short-wave upwelling radiation flux" ;
        swf_up:coordinates = "lat lon" ;
        swf_up:scale_factor = 0.25 ;
    short swf_net(time, rows, cols) ;
        swf_net:_FillValue = 0s ;
        swf_net:units = "W m**(-2)" ;
        swf_net:long_name = "TOA  short-wave net radiation flux" ;
        swf_net:coordinates = "lat lon" ;
        swf_net:scale_factor = 0.25 ;
    short lwf(time, rows, cols) ;
        lwf:_FillValue = 0s ;
        lwf:units = "W m**(-2)" ;
        lwf:long_name = "TOA  long-wave radiation flux" ;
        lwf:coordinates = "lat lon" ;
        lwf:scale_factor = 0.25 ;
    double lat(rows, cols) ;
        lat:units = "degrees_north" ;
        lat:long_name = "latitude" ;
    double lon(rows, cols) ;
        lon:units = "degrees_east" ;
        lon:long_name = "longitude" ;
    double time(time) ;
        time:units = "day as %Y%m%d.%f" ;
        time:long_name = "Time" ;
        time:calendar = "proleptic_gregorian" ;

// global attributes:
        :description = "instantaneous  TOA radiation fluxes simulated with ICON" ;
        :title = "TOA  Radiation Fluxes" ;
        :institution = "Leibniz Institute for Tropospheric Research" ;
        :author = "Fabian Senf (senf@tropos.de)" ;
}

Generic Input

The level2 function

import nawdex_analysis.io.input_lev2 as ilev2
ilev2.read_data_field??

provides the functionality to input these data together with georef and regional masks.

Input for CRE analysis

There is a second input functionality (collector) that reads allsky together with clearsky together with cloud type. This helps for the CRE calculations

ilev2.collect_data4cre_sim??

Average CRE

The analysis toolset

import  nawdex_analysis.analysis.ave_cre as acre
acre.ave_cre_from_radname
acre.ave_radfluxes_from_radname

provides functions to calculate domain-average CRE and radiation fluxes depending on cloud type as function of time. Looping over time index within each radfile is also possible.

Quick Restart

This document should help to quickly redo all analysis steps when new simulations are available.

Synsats (Mistral)

Calculation

  • Go to the synsat scripts directory

    cd ${HOME}/proj/2015-12_synsat/synsats4icon_nawdex

  • Make a list (also change listname…)

    EXPDIR=/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0002-shcon/
find ${EXPDIR} -iname '*fg*nc' | sort > lists/nawdexnwp-2km-mis-0002-shcon.lst

  • Split list into parts if runs should be distributed across several mistralpp nodes

    split -d -l 48 nawdexnwp-2km-mis-0002-shcon.lst nawdexnwp-2km-mis-0002-shcon.lst

  • Open screen

    screen -S synsat
. ~/.profile

  • Parallel synsat runs

    start_proc_from_list -n 4 ./synsat_icon-nawdex.py lists/nawdexnwp-2km-mis-0001-shcon.lst01

  • Wait

Regridding

After all synsat runs are finished: Regridding is done in parts (memory and speed issue)

  • go to the dir

    cd ${HOME}/proj/2017-07_nawdex_analysis/inout
./save_reproj_synsat.py ${HOME}/data/synsat/nawdex/nawdexnwp-80km-mis-0001

  • make a list file that looks like this (default: 6 lines = 6 chunks). vim regrid_synsat10.lst

    /pf/b/b380352/data/synsat/nawdex/nawdexnwp-2km-mis-0010-shcon 0
/pf/b/b380352/data/synsat/nawdex/nawdexnwp-2km-mis-0010-shcon 1
/pf/b/b380352/data/synsat/nawdex/nawdexnwp-2km-mis-0010-shcon 2
/pf/b/b380352/data/synsat/nawdex/nawdexnwp-2km-mis-0010-shcon 3
/pf/b/b380352/data/synsat/nawdex/nawdexnwp-2km-mis-0010-shcon 4
/pf/b/b380352/data/synsat/nawdex/nawdexnwp-2km-mis-0010-shcon 5

  • open a screen screen -S regrid_synsat10

  • run

    start_proc_from_list -n 6 ./save_reproj_synsat_in_parts.py regrid_synsat10.lst

  • goto the parts output dir

    cd ~/data/nawdex/synsat-parts/

  • combine file parts with cdo

    cdo -z zip_4 cat synsat-nawdexnwp-2km-mis-0010-shcon-part?.nc synsat-nawdexnwp-2km-mis-0010-shcon.nc

  • move the combine into the ../synsat directory

Combined Radiation Fluxes

Regridding (Mistral)

Similar to synsat regridding!!!

Regridding is done in parts (memory and speed issue)

  • go to the dir

    cd ${HOME}/proj/2017-07_nawdex_analysis/inout/

  • make a list file that looks like this (default: 6 lines = 6 chunks). vim regrid_radflux10.lst

    /work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0010-shcon 0
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0010-shcon 1
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0010-shcon 2
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0010-shcon 3
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0010-shcon 4
/work/bm0834/b380459/NAWDEX/ICON_OUTPUT_NWP/nawdexnwp-2km-mis-0010-shcon 5

  • open a screen screen -S regrid_radflux10

  • run

    start_proc_from_list -n 6 ./save_reproj_sim_raddata_in_parts.py regrid_radflux10.lst

  • goto the parts output dir

    cd ~/data/nawdex/sim-toarad-parts/

  • combine file parts with cdo

    cdo -z zip_4 cat toa_clear_radflux-nawdexnwp-2km-mis-0010-shcon-part?.nc toa_clear_radflux-nawdexnwp-2km-mis-0010-shcon.nc
cdo -z zip_4 cat toa_radflux-nawdexnwp-2km-mis-0010-shcon-part?.nc toa_radflux-nawdexnwp-2km-mis-0006-shcon.nc

  • move the combine into the ../sim-toarad directory

Transfer to TROPOS (Gauss)

Copy data to Gauss

cd ${LHOME}/data/icon/nawdex/sim-toarad
scp -r b380352@mistralpp.dkrz.de:/pf/b/b380352/data/nawdex/sim-toarad/*shcon.nc .

Cloud Typing (Gauss)

Transfer of Synsats to Gauss

Copy data to Gauss

cd  ${LHOME}/data/icon/nawdex/synsat/
scp -r b380352@mistralpp.dkrz.de:/pf/b/b380352/data/nawdex/synsat/*shcon.nc .

Run Cloud Typing

cd ${LHOME}/proj/2019-08_NWCSAF/synNWCSAF/scripts
./run_NWCSAF4synsat ${LHOME}/data/icon/nawdex/synsat/synsat-nawdexnwp-2km-mis-0001.nc

CRE (Gauss)

If regridded Radflux files & Synsat files are complete, then derived average rad-fluxes per CT (level3 data)

cd ${LHOME}/proj/2017-07_icon-nawdex/inout
./save_ave_radfluxes.py ${LHOME}/data/icon/nawdex/sim-toarad/toa_radflux-nawdexnwp-2km-mis-0001-shcon.nc

where the full filename refers to the regidded radflux file.

Adjusting the Naming Scheme

The new expermients might have new names, e.g. nawdexnwp-2km-mis-0001-shcon. The new experiment need to be mapped into experiment categories.

nawdex_analysis.io.selector has to be modified.

  • Go to the nawdex_analysis package development path

    cd ${LHOME}/proj/2018-05_pypackage_devel/nawdex_analysis/nawdex_analysis/io

  • Modify selector.py

    emacs selector.py

  • Change Version Number

    cd ${LHOME}/proj/2018-05_pypackage_devel/nawdex_analysis/nawdex_analysis
vim _version.py

  • Install new package version into thew system

    cd ..
pip install --upgrade .

nawdex_analysis.analysis package

The package collects two types of analysis

  • a general tools to perform averaging and histogram anaylsis
  • a specialized tools to calculate average CREs

Functions Overview

nawdex_analysis.analysis.ave_cre

A summary of functions that are contained in the nawdex_analysis.analysis.ave_cre module.

ave_cre_from_radname Calculate Cloud-radiative effect (CRE) for different cloud types.
ave_cre_from_radname_tloop Calculate Cloud-radiative effect (CRE) for different cloud types (time loop).
ave_radfluxes_from_radname Calculate average radiation fluxes for different cloud types.
radflux2cre Converts radiation fluxes to CRE.

nawdex_analysis.analysis.averaging

A summary of functions that are contained in the nawdex_analysis.analysis.averaging module.

area_fractions Calculates area fractions of variable values in certain intervals.
area_weighted_binwise_averages Calculates area fractions of variable values in certain intervals.

nawdex_analysis.io package

The package controlls input fields to be analyzed and also give some tools to write output.

A distinction is made between

  • primary input fields that might have very different grids
  • second analysis fields (on the same SEVIRI grid), level-2 data

Functions Overview

nawdex_analysis.io.collector

A summary of functions that are contained in the nawdex_analysis.io.collector module.

collect_sim_ave_cre4set Collects average CRE data for a selected set.
get_obs_cre4time_list Collects average CRE data for a selected set.
get_cre4set Collects average CRE data for a selected set.
get_radflux4set Collects average radiation flux data for a selected set.
get_stat4set Collects average CRE data for a selected set.
get_stat4all Read and stack all input sets (CRE or Radiation Fluxes).

nawdex_analysis.io.input_lev2

A summary of functions that are contained in the nawdex_analysis.io.input_lev2 module.

read_mask Read region mask.
read_data_field Reads “level2” data for analysis and plotting.
radname2ctname Name converter: Converts the standard filename of TOA allsky files to corresponding cloud type files.
collect_data4cre_obs Collects a set of observed data fields for cloud-radiative effect analysis.
collect_data4cre_sim Collects a set of simulated data fields for cloud-radiative effect analysis.
collect_data4cre Collects a set of data fields (observed or simulated) for cloud-radiative effect analysis.

nawdex_analysis.io.input_obs

A summary of functions that are contained in the nawdex_analysis.io.input_obs module.

msevi_setting Create a set of standard keywords for MSG SEVIRI input.
read_msevi Reads a time stack of MSG SEVIRI infrared channels.
scale_radiation The small helper function scales observed radiation data.
read_radiation_fluxes Reads and scales radiation flux data based on GERB-like SEVIRI retrievals.
read_solar_flux Reads and scales incoming solar radiation flux data based on GERB-like SEVIRI retrievals.
read_cc_from_fluxdata Reads and scales cloud cover based on GERB-like SEVIRI retrievals.
read_radiation_flux_tstack Reads and scales radiation flux data based on GERB-like SEVIRI retrievals.

nawdex_analysis.io.input_sim

A summary of functions that are contained in the nawdex_analysis.io.input_sim module.

subdir_from_fname Gets the sub-directory name from filename.
get_grid_filename Returns the name of a NAWDEX gridfile depending on the sub directory name.
get_synsat_basename Recovers simulation filename on which synsat forward operator was applied to.
read_georef Reads geo reference of simulation.
get_zen_mask Calculates a satellite zenith angle mask.
read_synsat_vector Input of Synsat BT vector (given at original ICON grid).
read_iconvar_vector Input of ICON variable vector (given at original ICON grid and with applied zenith angle mask).
read_icon_rad_vector Reads ICON TOA radiation vector.
read_time Reads time object from netcdf file.
read_generic_sim_data_flist Reads ICON field vectors from filelist as time stack.
read_radiation_flux_flist Reads ICON TOA radiation vectors from filelist as time stack.
read_synsat_flist Reads ICON Synsat vectors from filelist as time stack.
read_simulated_clearsky_flux Reads simulated solar clearsky fluxes.

nawdex_analysis.io.output_obs

A summary of functions that are contained in the nawdex_analysis.io.output_obs module.

save_meteosat_tstack Saves full time stack of daily Meteosat BT data to netcdf file for nawdex region.
save_meteosat_bt2nc Saves Meteosat BTs to netcdf.
save_radflux_tstack Saves full time stack of daily TOA radiation flux data to netcdf file for nawdex region.
save_rad2nc Saves Radiaiton fluxes to netcdf.

nawdex_analysis.io.output_sim

A summary of functions that are contained in the nawdex_analysis.io.output_sim module.

save_synsat_flist Saves full time stack of simulated BT data to netcdf file.
save_synsat_bt2nc Saves Synsat BTs to netcdf.
save_radflux_flist Saves full time stack of simulated TOA radiation flux data to netcdf file.
save_rad2nc Saves Radiaiton fluxes to netcdf.
save_retrieved_clearsky_swf2nc Saves retrieved clearsky radiation fluxes to netcdf.

nawdex_analysis.io.reproj

A summary of functions that are contained in the nawdex_analysis.io.reproj module.

set_msevi_proj Geostationary projection object for Meteosat.
msevi_ll2xy Returns line/column numbers in geostationary satellite projection.
msevi_xy2ij Converts msevi xy coordinates into indices.
msevi_ij2xy Converts msevi indices into xy coordinates.
msevi_ij2ll Converts msevi indices into xy coordinates.
msevi_lonlat Calcualtions MSG longitude and latitude for a certain cutout.
slice2nwcsaf_region Function that gets the NWCSAF region definition from a region slice (row1, row2), (col1, col2).
nwcsaf_region2slice The region defintion of the NWCSAF (based on size and center coordinates) is converted to a region slice.
get_vector2msevi_index Calculates nearest neighbor index for the conversion between ICON output vectors and the Msevi grid.
get_msevi2vector_index Calculates nearest neighbor index for the conversion between MSevi grid and ICON output vectors.
nn_reproj_with_index Reprojects data field intpo Meteosat SEVIRI projection.
get_reproj_param Preparation of reprojection parameters.
reproj_field Reprojection field using reprojection parameters (and grid box avareging)
get_vector2msevi_rparam Calculates reprojection parameters for the conversion between ICON output vectors and the Msevi grid.
combined_reprojection Combine nearest neighbor (nn) and box-average interpolation.

nawdex_analysis.io.selector

A summary of functions that are contained in the nawdex_analysis.io.selector module.

check_if_nc_has_varname Checks if netcdf file contains variable with a certain name.
check_if_nc_has_time Checks if netcdf file contains variable name AND time.
check_varname_and_time_in_nc Checks if netcdf file contains time.
make_filetime_index The function generates an time index listing all filenames that contain selected time and variable name.
set_selector Selects mis numbers based on (per-defined) experiment set.
set_initdate Selects init date based on (per-defined) experiment set.
gather_simset Collects a list of experiment names based on a selected set.
extra_experiments Add extra sets.
set_dateslices Sets the time range to be analyzed for a selected experiment set.
expname2conf_str Based on experiment name of configuration ID string is generated.
convert_explist2idlist Converts expname lists into ID string lists.

nawdex_analysis.io.tools

A summary of functions that are contained in the nawdex_analysis.io.tools module.

convert_time Utility converts between two time formats A->B or B->A:
convert_timevec Utility converts between an array of two time formats A->B or B->A:
roundTime Round a datetime object to any time laps in seconds.
round2day Rounds a np.datetime64 object to one day.
lonlat2azizen Calculates satellite zenith and azimuth given lon / lat.

nawdex_analysis.plot package

The plots package contains several utlilites to make it easier to get synchronized plots with the same style.

We have function

  • for standardize plot color, style and order
  • maps over the Atlantic domain
  • vertically stacked symbol plots
  • horizontally stacked bar plots

Functions Overview

nawdex_analysis.plot.exp_layout

A summary of functions that are contained in the nawdex_analysis.plot.exp_layout module.

get_exp_kws Wrapper for plotting keywords for different experiments.
get_exp_kws_lines Plotting keywords for different experiments - lines.
get_exp_kws_points Plotting keywords for different experiments - points.
get_exp_kws_bars Plotting keywords for different experiments - lines.
get_exp_kws_lines_expname Plotting keywords for different experiments - lines.
get_exp_kws_points_expname Plotting keywords for different experiments - points.
get_plotting_order Define a static order in which all different experiments are plotted.

nawdex_analysis.plot.legend

A summary of functions that are contained in the nawdex_analysis.plot.legend module.

legend_renamer A function to rename labels for plotting them into a legend.
plegend Convenience function for easier legend plotting.

nawdex_analysis.plot.nawdex_map

A summary of functions that are contained in the nawdex_analysis.plot.nawdex_map module.

nawdex_map Draws a map in cylindric projection.
nawdex_bt_plot Plots Brightness Temperature (BT) for NAWDEX Analysis.
nawdex_rad_plot Plots TOA Radiation Fluxes for NAWDEX Analysis.
nawdex_nwcsaf_plot Plots TOA Radiation Fluxes for NAWDEX Analysis.
nwcsaf_product_colorbar Plots TOA Radiation Fluxes for NAWDEX Analysis.

nawdex_analysis.plot.plot_nawdex_nwcsaf

A summary of functions that are contained in the nawdex_analysis.plot.plot_nawdex_nwcsaf module.

plot_prods Plots a selected NWCSAF product given a netcdf filename.
all_nwcsaf_plots_for_file Make NWCSAF plots for a file.

nawdex_analysis.plot.stacked_analysis_plots

A summary of functions that are contained in the nawdex_analysis.plot.stacked_analysis_plots module.

vert_stacked_exp_plot A generic function that makes a colorful plot with categories stacked in the vertical.
plot_scre_lcre4set Convenience function to easily plot shortwave vs.
plot_net_cre_contrib4set Convenience function to easily plot net CRE comparision.
hor_barplot_exp_plot A generic function that makes a colorful plot with categories stacked in the vertical.
stacker4flamingo Makes a cumulative data stack for flamingo plot.
hor_flamingo_plots Make a horizontal flamingo / snake plot.

Indices and tables