Welcome

Global models provide the big picture. A high-resolution model uses these data to simulate cloud processes that a global model cannot resolve, following clouds as they move and change.

Explore the cloud evolution observed by satellite (large panel on the left) and simulated with three high-resolution simulations (small panels on the right). The colored squares in the satellite view mark the regions followed by the simulations as the cloud field moves north-west:

Comparisons with satellite and aircraft measurements show that this approach can realistically reproduce clouds and their evolution, providing a tool for studying how they work.

Read the research paper: Kazil et al. (2021).

Weather and climate research often begins with data scattered across archives, stored in different formats, and sampled at different times and locations.

These Python tools automate the retrieval and preparation of numerical weather forecasts and surface observations, turning them into datasets that are easier to compare, analyze, visualize, and use in machine learning.

hrrr-data

Accessing and processing high-resolution weather forecasts

The High-Resolution Rapid Refresh (HRRR) is a numerical weather prediction model operated by the National Oceanic and Atmospheric Administration (NOAA). hrrr-data provides tools to retrieve its forecasts from public cloud archives and prepare them for analysis.

Key capabilities:

• Download forecasts for selected dates, initialization times, and forecast lead times.
• Access forecast data stored in GRIB2 or Zarr archives.
• Extract atmospheric variables such as temperature, humidity, wind, and precipitation into NetCDF files.
• Generate maps of selected forecast variables.

Why it is useful: Retrieving and processing numerical weather forecasts can involve substantial repetitive work. hrrr-data automates these steps, making it easier to assemble consistent forecast datasets for model evaluation, visualization, and machine-learning applications.

lcd-v2-data

Turning weather station observations into consistent time series

NOAA’s Local Climatological Data version 2 (LCD v2) provides surface weather observations from stations across the United States. The observations are not necessarily recorded at regular hourly intervals.

lcd-v2-data downloads and processes these observations into consistently sampled time series.

Key capabilities:

• Retrieve observations for individual stations, U.S. states and territories, or selected electricity-market regions.
• Convert observation timestamps to Coordinated Universal Time (UTC).
• Construct full-hourly time series of temperature, dew point, relative humidity, and wind components.
• Save the processed observations in NetCDF format, with optional diagnostic plots.

Why it is useful: Weather observations and model forecasts must be aligned in time before they can be meaningfully compared. lcd-v2-data automates much of this preparation, producing datasets suitable for forecast verification, regional analysis, and machine-learning development.

madis-data

Preparing surface observations from weather station networks

The Meteorological Assimilation Data Ingest System (MADIS) provides observations from multiple weather station networks, including Mesonet and METAR stations.

madis-data retrieves and processes these observations, creating datasets for selected stations, geographic regions, and time periods.

Key capabilities:

• Retrieve MADIS Mesonet and METAR observations.
• Select individual stations, U.S. states and territories, or larger geographic regions.
• Process temperature, dew point, and wind observations, together with solar radiation where available from Mesonet stations.
• Construct hourly time series and save them in NetCDF files.
• Generate maps of station locations and reuse previously processed data.

Why it is useful: MADIS brings together observations from numerous sources, providing valuable information about local weather conditions. madis-data simplifies preparing these observations for comparisons with weather forecasts, regional studies, and the development and evaluation of predictive models.

isd-lite-data

Accessing historical weather observations around the world

NOAA’s Integrated Surface Database Lite (ISD-Lite) provides hourly surface weather observations from stations in many countries.

isd-lite-data retrieves and organizes these observations into analysis-ready datasets.

Key capabilities:

• Retrieve observations for individual stations, countries, U.S. states and territories, or selected regions.
• Select stations based on geographic location and data availability.
• Process hourly observations of temperature, humidity, wind, atmospheric pressure, precipitation, and sky conditions.
• Save observations in NetCDF files for further analysis.

Why it is useful: Historical weather observations are essential for examining atmospheric conditions, evaluating numerical forecasts, and developing data-driven models. isd-lite-data makes it easier to assemble regional or international observational datasets without repeatedly handling the original archive files.

Status update (2026): NOAA discontinued updates to ISD-Lite after August 24, 2025, and announced the retirement of its original download service. Historical data remain available through the NOAA Open Data Dissemination (NODD) archive. For current observations and more extensive historical records, NOAA recommends the Global Historical Climatology Network–Hourly (GHCNh) dataset. See the official NOAA notice for details.

Training a machine-learning model is only part of its development. Models and optimizers need to be saved and restored, training progress needs to be monitored, and prediction errors need to be examined.

The following software addresses these practical aspects of machine-learning development.

torch-tk

Tools for training and evaluating PyTorch models

torch-tk is a Python toolkit built around PyTorch. It simplifies recurring tasks in machine-learning development, including model training, checkpoint management, and performance diagnostics.

Key capabilities:

• Model management: Save models derived from the toolkit’s model base class together with the information required to reconstruct them, including their configuration and trained parameters.
• Checkpoint management: Save and restore model and optimizer states, allowing training to continue from a saved checkpoint. Reconstruction requires importable model and optimizer classes that expose the toolkit’s configuration and state interfaces.
• Training: Train models using PyTorch data loaders or tensors, with facilities for recording training loss and execution time.
• Missing data: Train models whose target datasets contain missing values represented by NaN, using masked loss functions that exclude invalid targets and skipping batches with no valid targets.
• Diagnostics: Examine prediction losses for individual samples, identify poorly predicted cases, and visualize loss distributions.
• Analysis: Save diagnostic results in NetCDF format for subsequent investigation.

Why it is useful: Developing a neural network often involves repeated experiments, interruptions, parameter adjustments, and comparisons between different model configurations. torch-tk provides reusable tools for managing these activities, reducing repetitive implementation work and making it easier to investigate model performance.

It builds on standard PyTorch functionality rather than replacing it, allowing developers to retain familiar modeling and optimization workflows.