³⁶⁰⁴ views

Kernel: Python 2

THREDDS Data Server: Data access

The following information will demonstrate:

- Discover some data through NCI's Geonetwork page

- Direct download of a file

- OPeNDAP services:

Using OPeNDAP to request/convert subset to an ASCII file
Using OPeNDAP access and subsetting with Python modules

- NetcdfSubset service

Using THREDDS NetcdfSubset to request/convert subset another netCDF file

Browse for data

Begin by going to NCI's Geonetwork page: http://geonetwork.nci.org.au/

This page contains the metadata records for NCI Data Collections as well as information on where to find the data.

In this example, we will search for Landsat data:

If we click on the first result, we see a brief overview of the metadata record. Note: For the full record, navigate to the upper-right corner of your browser to change to the "Full view" (eyeball icon).

One of the options under Download and links is the NCI THREDDS Data Server Catalog page:

By navigating to this link, the available (public) data subcollections and datasets will be visible:

In this example, let's navigate to the ** LANDSAT data: scenes and tiles/ tiles/ EPSG3577/ LS8_OLI_TIRS_NBAR/ ** dataset:

To access the data we have several options. First select a data file:

1. File download

To download the full file, click the link titled: HTTPServer and the download should commence.

2. THREDDS OPeNDAP

On the THREDDS catalog page for the file, click the link titled: OPeNDAP

This will direct to the 'OPeNDAP Dataset Access Form'

This page will list information on the file contents, including file metadata and variables. Each variable will also contain a list of variable-level metadata (or attributes).

Request subset in ASCII format:

1. To request a subset with the OPeNDAP Dataset Access Form, begin by selecting the desired subset.

2. To select a variable:

Tick the box to the left of the variable name
Provide subset range for each dimension of the variable

3. Navigate to the top of the page and select "Get ASCII"

4. This should return the subset in a new window that can be copied/pasted or saved as a new file.

Use OPeNDAP for remote data access with Python

1. Return to the OPeNDAP Dataset Access Form page

2. Copy the URL in the field titled "Data URL"

3. This URL can be used in the same manner as the path to a local file within Python.

The following demonstrates how to access and subset with the common netCDF Python module "netCDF4".

In [1]:

from netCDF4 import Dataset

In [2]:

data_url = "http://dapds00.nci.org.au/thredds/dodsC/rs0/tiles/EPSG3577/LS8_OLI_TIRS_NBART/LS8_OLI_TIRS_NBART_3577_-10_-28_2013.nc"

Open file

This action opens the file but no data is yet loaded (or decompressed) until requested. Once the file is open, the following actions in Python are identical to those used with local files.

In [3]:

f = Dataset(data_url, 'r')

Browse file contents

In [17]:

## Display file dimensions
for item in f.dimensions:
    print item, f.dimensions[item].size

Out[17]:

maxStrlen64 64
time 61
x 4000
y 4000

In [16]:

## Look at variables
vars = f.variables.keys()
for item in vars:
    print 'Variable: \t', item
    print 'Dimensions: \t', f[item].dimensions
    print 'Shape:    \t', f[item].shape, '\n'

Out[16]:

Variable: 	y
Dimensions: 	(u'y',)
Shape:    	(4000,) 

Variable: 	x
Dimensions: 	(u'x',)
Shape:    	(4000,) 

Variable: 	time
Dimensions: 	(u'time',)
Shape:    	(61,) 

Variable: 	crs
Dimensions: 	()
Shape:    	() 

Variable: 	extra_metadata
Dimensions: 	(u'time', u'maxStrlen64')
Shape:    	(61, 64) 

Variable: 	band_6
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_7
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_4
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_5
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_2
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_3
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Variable: 	band_1
Dimensions: 	(u'time', u'y', u'x')
Shape:    	(61, 4000, 4000) 

Extract data subset examples (using indices)

In [19]:

# Extract the full spatial (x, y) range for the first time step:
band2_t0 = f.variables['band_2'][0, :, :]

# Extract all of time for a small spatial subset:
band2_tyx_subset = f.variables['band_2'][:, 2000:2500, 1000:1500]

3. THREDDS NetcdfSubset

On the THREDDS catalog page for the file, click the link titled: NetcdfSubset

This will direct to the 'NetcdfSubset' page

This page will list the available variables along with the spatial and temporal information. To request a subset, select the variable(s) of interest (and/or spatial/temporal range) and then select 'Submit'.