Otherworldly Gate: February 2008

Friday, February 29, 2008

Example PolarGrid Web Feeds

To get all existing entries: http://pg3.ucs.indiana.edu/cgi-bin/getPolarGridFeed

To experiment GeoRSS feeds in Google Maps, point http://pg3.ucs.indiana.edu/cgi-bin/gm_getPolarGridFeed?entries=10 in the search bar, and click on "Search Maps". The value for entries is adjustable with the maximum 64 limit imposed by Google Maps.

To play with KML in Google Earth, point "Add->Network Link" to http://pg3.ucs.indiana.edu/cgi-bin/gm_getPolarGridKML?entries=10, where entries is similarly adjustable. The KML works in Google Maps too.

Perl with Atom Feed

While working on the PolarGrid web feed generator, I found a helpful document on how to create an Atom feed in Perl. But it's slightly out of date, and here's a short summary of my journey:

The latest version of Perl CPAN module XML::Atom supports Atom 1.0, and the default Atom 0.3 version can be overwritten in either the Version parameter of XML::Atom::Feed and XML::Atom::Entry, or the global variable $XML::Atom::DefaultVersion.

The Feed Validator is a very helpful debugging tool. In my case, besides everything I wanted to display in a feed, id, updated, author, self_link for the feed, and id, updated for an entry are also required. In addition, note the specific DateTime and id formats.

To add GeoRSS encoding in the feed, don't forget to include the corresponding XML namespace: xmlns:georss="http://www.georss.org/georss"

Use the following to output/return the feed as a HTTP response:print "Content-type: application/atom+xml; charset=utf-8\n\n"; print $feed->as_xml

GeoRSS vs. KML

From my limited exposure to both, I have but one comment: Both are straightforward enough for basic functions, however, a simple point in GeoRSS is represented as
<georss:point>latitude longitude</georss:point>
while in KML it's
<Point> <coordinates>longitude,latitude</coordinates> </Point>

So everyone definitely needs his own favorite separator, but can't we all play nice with each other, and agree on something consistent with the coordinate system?

Anyway, the lesson is if you have a GeoRSS feed that displays well in Google Maps, don't forget to swap the coordinates when converting it into KML for Google Earth. Does anyone else also wish they make up a mind already?!

Monday, February 18, 2008

Example PolarGrid Microformat Atom Feed Entry


<div>
<span class="pg:description">MCRDS 2007 09 17</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Sampling</span>
  <span class="pg:value">720</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:averages">32</span>
<br>
<span class="waveforms">2</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">2</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">4</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">3</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">4</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:data-acquisition-unit-array">
  <span class="pg:array-size">4</span>
  <span class="data-acquisition-unit">
    <span class="pg:id">1</span>
    <span class="pg:samples">9346</span>
    <span class="pg:delay>
      <span class="pg:value">0</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">nan</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">2</span>
    <span class="pg:samples">9346</span>
    <span class="pg:delay>
      <span class="pg:value">0</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">nan</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">3</span>
    <span class="pg:samples">9346</span>
    <span class="pg:delay>
      <span class="pg:value">0</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">nan</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">4</span>
    <span class="pg:samples">9346</span>
    <span class="pg:delay>
      <span class="pg:value">0</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">nan</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
</span>
<br>
<span class="pg:dsp-mode">coherent</span>

<p>
<span class="pg:waveform">
  <span class="pg:id">1</span>
</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Start</span>
  <span class="pg:value">140</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Stop</span>
  <span class="pg:value">160</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:pulse-width">
  <span class="pg:value">3.016670e+00</span>
  <span class="pg:unit">us</span>
</span>
<br>
<span class="pg:zero-pi-mode">1</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">2</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">4</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">3</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">4</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:data-acquisition-unit-array">
  <span class="pg:array-size">4</span>
  <span class="data-acquisition-unit">
    <span class="pg:id">1</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">2</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">3</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">4</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
</span>

<p>
<span class="pg:waveform">
  <span class="pg:id">2</span>
</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Start</span>
  <span class="pg:value">140</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Stop</span>
  <span class="pg:value">160</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:pulse-width">
  <span class="pg:value">1.001670e+01</span>
  <span class="pg:unit">us</span>
</span>
<br>
<span class="pg:zero-pi-mode">1</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">2</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">4</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">3</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">4</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:data-acquisition-unit-array">
  <span class="pg:array-size">4</span>
  <span class="data-acquisition-unit">
    <span class="pg:id">1</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">16</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">15</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">2</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">16</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">15</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">3</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">16</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">15</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">4</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">16</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">15</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
</span>

<p>
<span class="pg:data-chunk">
  <span class="pg:name">Start</span>
  <span class="pg:utc-timestamp">1190029164.278480</span>
  <span class="geo">
    <span class="longitude">-59.524211</span>
    <span class="latitude">77.601132</span>
  </span>
</span>
<georss:where>
  <georss:point>77.601132 -59.524211</georss:point>
</georss:where>
<br>
<span class="pg:data-chunk">
  <span class="pg:name">Stop</span>
  <span class="pg:utc-timestamp">1190029165.986734</span>
  <span class="geo">
    <span class="longitude">-59.506412</span>
    <span class="latitude">77.607323</span>
  </span>
</span>
<georss:where>
  <georss:point>77.607323 -59.506412</georss:point></georss:where>

<p>
<span class="pg:data-file">  <span class="pg:name">MCRDS_20070917_113924_fk_image_wf_01_tx_01_rx_01.jpg</span></span>
<br>
<span class="pg:data-file">  <span class="pg:name">MCRDS_20070917_113924_fk_image_wf_01_tx_01_rx_02.jpg</span></span>
<br>
<span class="pg:data-file">  <span class="pg:name">MCRDS_20070917_113924_fk_image_wf_02_tx_01_rx_03.jpg</span></span>
<br>
<span class="pg:data-file">  <span class="pg:name">MCRDS_20070917_113924_fk_image_wf_02_tx_01_rx_04.jpg</span></span>
<br>
</div>

Sunday, February 17, 2008

My PolarGrid

A PolarGrid fan in my house wrote a small program to randomly grab pictures from Flickr One Letter image group, and generated this picture to show his support.

Microformat for PolarGrid -- Amendment

Thanks to William for a concrete example, and the request of including units in microformats, here comes the revision:

Simple descriptive metadata:
InSAR 2008 01 22 1 1 coherent 3

Pulse width:
 3 us

Frequency, used in both overall system sampling and start/stop property of individual waveforms:
 Sampling/Start/Stop 720 mhz

Antenna:
 0 TX/RX 0

Array of antennas: (TX or RX)
 2 0 TX 0 1 TX 0

Sample delay:
 0 us

Blanking time:
 5 us

Data acquisition unit:
 0 50000 0 us 5 us

Array of data acquisition units:
 2 0 50000 0 us 5 us 1 50000 0 us 5 us

Spatial and temporal information of a data processing chunk. Note that existing geo microformat is used, which is outside our pg namespace:
 Start/Stop 1202755351.892651 -2.193 52.686

Individual waveform identification:
 1

Individual (jpg) file:
 MCRDS_20070917_113924_fk_image_wf_01_tx_01_rx_01.jpg

Tuesday, February 12, 2008

Microformat for PolarGrid

Prelude this interesting discussion of very human and machine-readable data formats, here is my contribution to the google ranking of PolarGrid (a.k.a. Polar Grid) project.

Background
Over the weekend William gave us an overview of metadata generated from the Matlab data processing, which will be published through web feeds. Much of the following summary comes from email excerpts.

A processed dataset corresponds to one jpg image, which consists of processed measurement values from many radar samples. Fundamental properties of each processed dataset are defined by three parameters forming a triplet: waveform (wf), transmit antenna (tx), and receive antenna (rx).

Each radar system can transmit several different waveforms, and a waveform can be transmitted and received on one or more antennas. So an example jpg filename would be TYPE_YYYYMMDD_HHMMSS_image_wf_##_tx_##_rx_##.jpg, where TYPE describes the radar data type such as InSAR, YYYYMMDD_HHMMSS corresponds to the UTC timestamp of the first sample in the dataset, <wf, tx, rx> triplet specifies the particular waveform transmitted and received on a particular antenna, and the word "image" distinguishes these data from position or reference data.
Additionally there may be pulse compressed and f-k migrated images, which correspond to
"image_pc"or "image_fk", and are different types of processed images for the same track. Word "comb" would replace wf_##_tx_##_rx_## in case of combined images too.

A plain text file will be associated with each jpg image, containing three levels of metadata.
1. Overall system configuration and setup:
 Description: "InSAR 2008 01 22 " Sampling Frequency (MHz): 720 Number of Averages (Samples): 1 Number of Waveforms: 1 TX Attenuation: [0 0] RX Attenuation: [0 0] Number of Samples: [20000 20000] DSP Mode: coherent System Delay (us): 0
 Each TX/RX Attenuation value corresponds to a tx/rx antenna. Hence in case of 2 tx antennas and 8 rx antennas, the values could be
 TX Attenuation: [10 10] RX Attenuation: [5 5 5 5 5 5 5 5]
 Each "Number of Samples" value corresponds to a data acquisition unit (daq), the total number of which is always less than or equal to the total number of rx antennas. Hence in case of 4 daqs for 8 rx antennas, the value could be
 Number of Samples: [20000 20000 20000 20000]
 For each daq, the "Number of Samples" value is the sum of the corresponding value described in individual waveform metadata.
2. Spatial and temporal information of data processing chunks:
 Start lon: (Degrees) Stop lon: (Degrees) Start lat: (Degrees) Stop lat: (Degrees) Start UTC time: (seconds.useconds) Stop UTC time: (seconds.useconds)
 The complete expedition consists of multiple (~30) data chunks, and all <wf, tx, rx> triplets in the same chunk share the same spatial and temporal chunk metadata.
3. Information about individual waveforms. Continuing the example of 2 tx antennas, 8 rx antennas, and 4 daqs, 2 waveforms can be described as:
 Start Frequency (MHz): 120 Stop Frequency (MHz): 300 Pulse Width (us): 10 Zero/Pi Mode: 1 TX Attenuation: [0 0] RX Attenuation: [0 0 0 0 0 0 0 0] Number of Samples: [15000 15000 15000 15000] Sample Delay (us): [20 20 20 20] Blanking Time (us): [20 20 20 20] Start Frequency (MHz): 120 Stop Frequency (MHz): 300 Pulse Width (us): 3 Zero/Pi Mode: 1 TX Attenuation: [5 5] RX Attenuation: [7 7 7 7 7 7 7 7] Number of Samples: [5000 5000 5000 5000] Sample Delay (us): [0 0 0 0] Blanking Time (us): [5 5 5 5]
 For each waveform, each daq also corresponds to one Sample Delay and one Blanking Time.
4. A fourth level of metadata associated with each jpg image can be deduced from its file name as discussed earlier, including radar type, specific <wf, tx, rx> triplet value, processed image type, and a timestamp.

Microformat Proposal
We propose the following set of microformats to describe the above metadata, and they can be composed to construct web feeds for individual jpg images. All class names in the proposed microformat start with "pg:" denoting the PolarGrid namespace.

Simple descriptive metadata:
InSAR 2008 01 22 1 1 coherent 3 3

Frequency, used in both overall system sampling and start/stop property of individual waveforms:
 Sampling/Start/Stop 720

Antenna:
 0 TX/RX 0

Array of antennas: (TX or RX)
 2 0 TX 0 1 TX 0

Data acquisition unit:
 0 50000 0 5

Array of data acquisition units:
 2 0 50000 0 5 1 50000 0 5

Spatial and temporal information of a data processing chunk. Note that existing geo microformat is used, which is outside our pg namespace:
 Start/Stop 1202755351.892651 -2.193 52.686

Individual waveform identification:
 1

Missing pieces: we are yet to understand the relationship between a waveform description and individual jpg images better to supply microformats describing the fourth level of metadata, especially the location of jpg images to be provided in the final web feeds.

Comments
The above proposal is a very preliminary draft, please criticize, suggest, and comment. Every single click improves the internet fame of our project too.

Otherworldly Gate