Otherworldly Gate: 2008

Friday, June 13, 2008

Apache Security

When hosting an Apache web server for very concerned customers, authentication, authorization, and access control can all flood in the way. Some may even ask for host-based methods. Just keep in mind that Apache has its own way for them all, including where to write logs. Don't go mess with usual files like /etc/hosts, /etc/hosts.allow, /etc/hosts.deny, or /var/log/messages. Unless, of course, your boss told you to.

Tuesday, June 3, 2008

MySQL in Non-standard Locations

Sometimes one ends up with a machine containing multiple MySQL installations, and no clue how and why it happened in the first place. Besides normal which password to use for which server confusions, it also requires some attention when installing packages like Perl DBD-mysql module. Be sure to specify the correct -I and -L paths at "perl Makefile.PL" step of the installation.

For example:


perl Makefile.PL --cflags="-I/usr/local/mysql/include -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fwrapv" --libs="-L/usr/local/mysql/lib -lmysqlclient -lz -lcrypt -lnsl -lm -L/usr/lib64 -lssl -lcrypto"

A Perl DBI client connection string can specify the non-standard socket location like this:


"DBI:mysql:mydb:localhost:mysql_socket=/tmp/mysql.sock"

Friday, May 30, 2008

What CoG Really Wants

I ran into various error messages from the CoG Kit when attempting a fresh install of GridFTP enabled CIMA portal. After figuring out and fixing all mishaps on my part, I wonder if anyone else also find those messages cryptic.

Using wrong Java.
$ java --version java version "1.5.0" gij (GNU libgcj) version 4.1.2 20070925 (Red Hat 4.1.2-27) $ ./cog-myproxy -h myproxy.teragrid.org -p 7514 -l username -S anonget Enter MyProxy Pass Phrase: Error: MyProxy get failed. [Caused by: Authentication failed [Caused by: Miscellaneous failure. [Caused by: Bad certificate (java.security.SignatureException: MD5/RSA/PKCS#1: Not initialized)]]]
Should have used the following one from Sun instead:
$ java -version java version "1.5.0_14" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_14-b03) Java HotSpot(TM) Client VM (build 1.5.0_14-b03, mixed mode, sharing)

Missing the remote host/server CA certificate from $HOME/.globus/certificates. In the following example, it's missing the NCSA certificate, CA who signed the BigRed cluster. Insiders may like this reference.
$ ./cog-file-transfer -s gsiftp://gridftp.bigred.iu.teragrid.org/N/dc/projects/cima/lustre_iumsc_frames/07108/a_071081.001 -d file:///tmp/a_071081.001 ERROR - File transfer failed Authentication failed. Caused by Failure unspecified at GSS-API level. Caused by COM.claymoresystems.ptls.SSLThrewAlertException: Unknown CA at COM.claymoresystems.ptls.SSLConn.alert(SSLConn.java:235) at COM.claymoresystems.ptls.SSLHandshake.recvCertificate(SSLHandshake.java:304) at COM.claymoresystems.ptls.SSLHandshakeClient.processTokens(SSLHandshakeClient.java:128) at COM.claymoresystems.ptls.SSLHandshake.processHandshake(SSLHandshake.java:135) at org.globus.gsi.gssapi.GlobusGSSContextImpl.initSecContext(GlobusGSSContextImpl.java:488) at org.globus.ftp.extended.GridFTPControlChannel.authenticate(GridFTPControlChannel.java:208) at org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:104) at org.globus.ftp.GridFTPClient.authenticate(GridFTPClient.java:89) at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.start(FileResourceImpl.java:97) at org.globus.cog.abstraction.impl.file.gridftp.FileResourceImpl.start(FileResourceImpl.java:47) at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.startResource(DelegatedFileTransferHandler.java:415) at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.prepareService(DelegatedFileTransferHandler.java:191) at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:434) at java.lang.Thread.run(Thread.java:595)

Missing the NCSA MyProxy CA certificate.
$ ./cog-file-transfer -s gsiftp://gridftp.bigred.iu.teragrid.org/N/dc/projects/cima/lustre_iumsc_frames/07108/a_071081.001 -d file:///tmp/a_071081.001 ERROR - File transfer failed org.globus.cog.abstraction.impl.file.FileResourceException: Exception in getFile at org.globus.cog.abstraction.impl.file.gridftp.old.FileResourceImpl.getFile(FileResourceImpl.java:312) at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.doSource(DelegatedFileTransferHandler.java:251) at org.globus.cog.abstraction.impl.fileTransfer.DelegatedFileTransferHandler.run(DelegatedFileTransferHandler.java:438) at java.lang.Thread.run(Thread.java:595) Caused by: org.globus.ftp.exception.ServerException: Server refused performing the request. Custom message: (error code 1) [Nested exception message: Custom message: Unexpected reply: 451 active connection to server failed org.globus.common.ChainedIOException: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]] Authentication failed. Caused by Failure unspecified at GSS-API level. Caused by COM.claymoresystems.ptls.SSLThrewAlertException: Unknown CA at COM.claymoresystems.ptls.SSLConn.alert(SSLConn.java:235) at COM.claymoresystems.ptls.SSLHandshake.recvCertificate(SSLHandshake.java:304) at COM.claymoresystems.ptls.SSLHandshakeClient.processTokens(SSLHandshakeClient.java:128) at COM.claymoresystems.ptls.SSLHandshake.processHandshake(SSLHandshake.java:135) at org.globus.gsi.gssapi.GlobusGSSContextImpl.initSecContext(GlobusGSSContextImpl.java:488) at org.globus.gsi.gssapi.net.GssSocket.authenticateClient(GssSocket.java:107) at org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:145) at org.globus.gsi.gssapi.net.GssSocket.getOutputStream(GssSocket.java:166) at org.globus.ftp.extended.GridFTPServerFacade.authenticate(GridFTPServerFacade.java:550) at org.globus.ftp.dc.GridFTPActiveConnectTask.execute(GridFTPActiveConnectTask.java:86) at org.globus.ftp.dc.TaskThread.run(TaskThread.java:71) at java.lang.Thread.run(Thread.java:595) ] [Nested exception is org.globus.ftp.exception.UnexpectedReplyCodeException: Custom message: Unexpected reply: 451 active connection to server failed org.globus.common.ChainedIOException: Authentication failed [Caused by: Failure unspecified at GSS-API level [Caused by: Unknown CA]] Authentication failed. Caused by Failure unspecified at GSS-API level. Caused by COM.claymoresystems.ptls.SSLThrewAlertException: Unknown CA at COM.claymoresystems.ptls.SSLConn.alert(SSLConn.java:235) at COM.claymoresystems.ptls.SSLHandshake.recvCertificate(SSLHandshake.java:304) at COM.claymoresystems.ptls.SSLHandshakeClient.processTokens(SSLHandshakeClient.java:128) at COM.claymoresystems.ptls.SSLHandshake.processHandshake(SSLHandshake.java:135) at org.globus.gsi.gssapi.GlobusGSSContextImpl.initSecContext(GlobusGSSContextImpl.java:488) at org.globus.gsi.gssapi.net.GssSocket.authenticateClient(GssSocket.java:107) at org.globus.gsi.gssapi.net.GssSocket.startHandshake(GssSocket.java:145) at org.globus.gsi.gssapi.net.GssSocket.getOutputStream(GssSocket.java:166) at org.globus.ftp.extended.GridFTPServerFacade.authenticate(GridFTPServerFacade.java:550) at org.globus.ftp.dc.GridFTPActiveConnectTask.execute(GridFTPActiveConnectTask.java:86) at org.globus.ftp.dc.TaskThread.run(TaskThread.java:71) at java.lang.Thread.run(Thread.java:595) ] at org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java:101) at org.globus.ftp.exception.ServerException.embedUnexpectedReplyCodeException(ServerException.java:110) at org.globus.ftp.vanilla.TransferMonitor.run(TransferMonitor.java:195) ... 1 more

Maybe they don't look as cryptic now as before, when I was scratching my head over them...

Friday, February 29, 2008

Example PolarGrid Web Feeds

To get all existing entries: http://pg3.ucs.indiana.edu/cgi-bin/getPolarGridFeed

To experiment GeoRSS feeds in Google Maps, point http://pg3.ucs.indiana.edu/cgi-bin/gm_getPolarGridFeed?entries=10 in the search bar, and click on "Search Maps". The value for entries is adjustable with the maximum 64 limit imposed by Google Maps.

To play with KML in Google Earth, point "Add->Network Link" to http://pg3.ucs.indiana.edu/cgi-bin/gm_getPolarGridKML?entries=10, where entries is similarly adjustable. The KML works in Google Maps too.

Perl with Atom Feed

While working on the PolarGrid web feed generator, I found a helpful document on how to create an Atom feed in Perl. But it's slightly out of date, and here's a short summary of my journey:

The latest version of Perl CPAN module XML::Atom supports Atom 1.0, and the default Atom 0.3 version can be overwritten in either the Version parameter of XML::Atom::Feed and XML::Atom::Entry, or the global variable $XML::Atom::DefaultVersion.

The Feed Validator is a very helpful debugging tool. In my case, besides everything I wanted to display in a feed, id, updated, author, self_link for the feed, and id, updated for an entry are also required. In addition, note the specific DateTime and id formats.

To add GeoRSS encoding in the feed, don't forget to include the corresponding XML namespace: xmlns:georss="http://www.georss.org/georss"

Use the following to output/return the feed as a HTTP response:print "Content-type: application/atom+xml; charset=utf-8\n\n"; print $feed->as_xml

GeoRSS vs. KML

From my limited exposure to both, I have but one comment: Both are straightforward enough for basic functions, however, a simple point in GeoRSS is represented as
<georss:point>latitude longitude</georss:point>
while in KML it's
<Point> <coordinates>longitude,latitude</coordinates> </Point>

So everyone definitely needs his own favorite separator, but can't we all play nice with each other, and agree on something consistent with the coordinate system?

Anyway, the lesson is if you have a GeoRSS feed that displays well in Google Maps, don't forget to swap the coordinates when converting it into KML for Google Earth. Does anyone else also wish they make up a mind already?!

Monday, February 18, 2008

Example PolarGrid Microformat Atom Feed Entry


<div>
<span class="pg:description">MCRDS 2007 09 17</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Sampling</span>
  <span class="pg:value">720</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:averages">32</span>
<br>
<span class="waveforms">2</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">2</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">4</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">3</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">4</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:data-acquisition-unit-array">
  <span class="pg:array-size">4</span>
  <span class="data-acquisition-unit">
    <span class="pg:id">1</span>
    <span class="pg:samples">9346</span>
    <span class="pg:delay>
      <span class="pg:value">0</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">nan</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">2</span>
    <span class="pg:samples">9346</span>
    <span class="pg:delay>
      <span class="pg:value">0</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">nan</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">3</span>
    <span class="pg:samples">9346</span>
    <span class="pg:delay>
      <span class="pg:value">0</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">nan</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">4</span>
    <span class="pg:samples">9346</span>
    <span class="pg:delay>
      <span class="pg:value">0</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">nan</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
</span>
<br>
<span class="pg:dsp-mode">coherent</span>

<p>
<span class="pg:waveform">
  <span class="pg:id">1</span>
</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Start</span>
  <span class="pg:value">140</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Stop</span>
  <span class="pg:value">160</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:pulse-width">
  <span class="pg:value">3.016670e+00</span>
  <span class="pg:unit">us</span>
</span>
<br>
<span class="pg:zero-pi-mode">1</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">2</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">4</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">3</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">4</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:data-acquisition-unit-array">
  <span class="pg:array-size">4</span>
  <span class="data-acquisition-unit">
    <span class="pg:id">1</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">2</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">3</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">4</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">1</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
</span>

<p>
<span class="pg:waveform">
  <span class="pg:id">2</span>
</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Start</span>
  <span class="pg:value">140</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:frequency">
  <span class="pg:name">Stop</span>
  <span class="pg:value">160</span>
  <span class="pg:unit">MHz</span>
</span>
<br>
<span class="pg:pulse-width">
  <span class="pg:value">1.001670e+01</span>
  <span class="pg:unit">us</span>
</span>
<br>
<span class="pg:zero-pi-mode">1</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">2</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">TX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:antenna-array">
  <span class="pg:array-size">4</span>
  <span class="pg:antenna">
    <span class="pg:id">1</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">2</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">3</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
  <span class="pg:antenna">
    <span class="pg:id">4</span>
    <span class="pg:type">RX</span>
    <span class="pg:attenuation">0</span>
  </span>
</span>
<br>
<span class="pg:data-acquisition-unit-array">
  <span class="pg:array-size">4</span>
  <span class="data-acquisition-unit">
    <span class="pg:id">1</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">16</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">15</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">2</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">16</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">15</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">3</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">16</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">15</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
  <span class="data-acquisition-unit">
    <span class="pg:id">4</span>
    <span class="pg:samples">4673</span>
    <span class="pg:delay>
      <span class="pg:value">16</span>
      <span class="pg:unit">us</span>
    </span>
    <span class="pg:blanking">
      <span class="pg:value">15</span>
      <span class="pg:unit">us</span>
    </span>
  </span>
</span>

<p>
<span class="pg:data-chunk">
  <span class="pg:name">Start</span>
  <span class="pg:utc-timestamp">1190029164.278480</span>
  <span class="geo">
    <span class="longitude">-59.524211</span>
    <span class="latitude">77.601132</span>
  </span>
</span>
<georss:where>
  <georss:point>77.601132 -59.524211</georss:point>
</georss:where>
<br>
<span class="pg:data-chunk">
  <span class="pg:name">Stop</span>
  <span class="pg:utc-timestamp">1190029165.986734</span>
  <span class="geo">
    <span class="longitude">-59.506412</span>
    <span class="latitude">77.607323</span>
  </span>
</span>
<georss:where>
  <georss:point>77.607323 -59.506412</georss:point></georss:where>

<p>
<span class="pg:data-file">  <span class="pg:name">MCRDS_20070917_113924_fk_image_wf_01_tx_01_rx_01.jpg</span></span>
<br>
<span class="pg:data-file">  <span class="pg:name">MCRDS_20070917_113924_fk_image_wf_01_tx_01_rx_02.jpg</span></span>
<br>
<span class="pg:data-file">  <span class="pg:name">MCRDS_20070917_113924_fk_image_wf_02_tx_01_rx_03.jpg</span></span>
<br>
<span class="pg:data-file">  <span class="pg:name">MCRDS_20070917_113924_fk_image_wf_02_tx_01_rx_04.jpg</span></span>
<br>
</div>

Sunday, February 17, 2008

My PolarGrid

A PolarGrid fan in my house wrote a small program to randomly grab pictures from Flickr One Letter image group, and generated this picture to show his support.

Microformat for PolarGrid -- Amendment

Thanks to William for a concrete example, and the request of including units in microformats, here comes the revision:

Simple descriptive metadata:
InSAR 2008 01 22 1 1 coherent 3

Pulse width:
 3 us

Frequency, used in both overall system sampling and start/stop property of individual waveforms:
 Sampling/Start/Stop 720 mhz

Antenna:
 0 TX/RX 0

Array of antennas: (TX or RX)
 2 0 TX 0 1 TX 0

Sample delay:
 0 us

Blanking time:
 5 us

Data acquisition unit:
 0 50000 0 us 5 us

Array of data acquisition units:
 2 0 50000 0 us 5 us 1 50000 0 us 5 us

Spatial and temporal information of a data processing chunk. Note that existing geo microformat is used, which is outside our pg namespace:
 Start/Stop 1202755351.892651 -2.193 52.686

Individual waveform identification:
 1

Individual (jpg) file:
 MCRDS_20070917_113924_fk_image_wf_01_tx_01_rx_01.jpg

Tuesday, February 12, 2008

Microformat for PolarGrid

Prelude this interesting discussion of very human and machine-readable data formats, here is my contribution to the google ranking of PolarGrid (a.k.a. Polar Grid) project.

Background
Over the weekend William gave us an overview of metadata generated from the Matlab data processing, which will be published through web feeds. Much of the following summary comes from email excerpts.

A processed dataset corresponds to one jpg image, which consists of processed measurement values from many radar samples. Fundamental properties of each processed dataset are defined by three parameters forming a triplet: waveform (wf), transmit antenna (tx), and receive antenna (rx).

Each radar system can transmit several different waveforms, and a waveform can be transmitted and received on one or more antennas. So an example jpg filename would be TYPE_YYYYMMDD_HHMMSS_image_wf_##_tx_##_rx_##.jpg, where TYPE describes the radar data type such as InSAR, YYYYMMDD_HHMMSS corresponds to the UTC timestamp of the first sample in the dataset, <wf, tx, rx> triplet specifies the particular waveform transmitted and received on a particular antenna, and the word "image" distinguishes these data from position or reference data.
Additionally there may be pulse compressed and f-k migrated images, which correspond to
"image_pc"or "image_fk", and are different types of processed images for the same track. Word "comb" would replace wf_##_tx_##_rx_## in case of combined images too.

A plain text file will be associated with each jpg image, containing three levels of metadata.
1. Overall system configuration and setup:
 Description: "InSAR 2008 01 22 " Sampling Frequency (MHz): 720 Number of Averages (Samples): 1 Number of Waveforms: 1 TX Attenuation: [0 0] RX Attenuation: [0 0] Number of Samples: [20000 20000] DSP Mode: coherent System Delay (us): 0
 Each TX/RX Attenuation value corresponds to a tx/rx antenna. Hence in case of 2 tx antennas and 8 rx antennas, the values could be
 TX Attenuation: [10 10] RX Attenuation: [5 5 5 5 5 5 5 5]
 Each "Number of Samples" value corresponds to a data acquisition unit (daq), the total number of which is always less than or equal to the total number of rx antennas. Hence in case of 4 daqs for 8 rx antennas, the value could be
 Number of Samples: [20000 20000 20000 20000]
 For each daq, the "Number of Samples" value is the sum of the corresponding value described in individual waveform metadata.
2. Spatial and temporal information of data processing chunks:
 Start lon: (Degrees) Stop lon: (Degrees) Start lat: (Degrees) Stop lat: (Degrees) Start UTC time: (seconds.useconds) Stop UTC time: (seconds.useconds)
 The complete expedition consists of multiple (~30) data chunks, and all <wf, tx, rx> triplets in the same chunk share the same spatial and temporal chunk metadata.
3. Information about individual waveforms. Continuing the example of 2 tx antennas, 8 rx antennas, and 4 daqs, 2 waveforms can be described as:
 Start Frequency (MHz): 120 Stop Frequency (MHz): 300 Pulse Width (us): 10 Zero/Pi Mode: 1 TX Attenuation: [0 0] RX Attenuation: [0 0 0 0 0 0 0 0] Number of Samples: [15000 15000 15000 15000] Sample Delay (us): [20 20 20 20] Blanking Time (us): [20 20 20 20] Start Frequency (MHz): 120 Stop Frequency (MHz): 300 Pulse Width (us): 3 Zero/Pi Mode: 1 TX Attenuation: [5 5] RX Attenuation: [7 7 7 7 7 7 7 7] Number of Samples: [5000 5000 5000 5000] Sample Delay (us): [0 0 0 0] Blanking Time (us): [5 5 5 5]
 For each waveform, each daq also corresponds to one Sample Delay and one Blanking Time.
4. A fourth level of metadata associated with each jpg image can be deduced from its file name as discussed earlier, including radar type, specific <wf, tx, rx> triplet value, processed image type, and a timestamp.

Microformat Proposal
We propose the following set of microformats to describe the above metadata, and they can be composed to construct web feeds for individual jpg images. All class names in the proposed microformat start with "pg:" denoting the PolarGrid namespace.

Simple descriptive metadata:
InSAR 2008 01 22 1 1 coherent 3 3

Frequency, used in both overall system sampling and start/stop property of individual waveforms:
 Sampling/Start/Stop 720

Antenna:
 0 TX/RX 0

Array of antennas: (TX or RX)
 2 0 TX 0 1 TX 0

Data acquisition unit:
 0 50000 0 5

Array of data acquisition units:
 2 0 50000 0 5 1 50000 0 5

Spatial and temporal information of a data processing chunk. Note that existing geo microformat is used, which is outside our pg namespace:
 Start/Stop 1202755351.892651 -2.193 52.686

Individual waveform identification:
 1

Missing pieces: we are yet to understand the relationship between a waveform description and individual jpg images better to supply microformats describing the fourth level of metadata, especially the location of jpg images to be provided in the final web feeds.

Comments
The above proposal is a very preliminary draft, please criticize, suggest, and comment. Every single click improves the internet fame of our project too.

Tuesday, January 22, 2008

Much Ado

About seemingly trivial stuff in the last couple of weeks, like how to redirect an URL in JSF, how to completely wipe off sensitive, crucial information that were accidently, mistakenly checked into CVS, etc. But as the happy ending, I finished my part of transforming CIMA portal to use GridFTP for data transfer services. (Of course I have to mention CIMA portal often to maintain my seat in the Google search competition.)

In the coming couple of weeks, besides the PolarGrid meeting and a quick trip to the west, I expect to be watching videos and surfing webs, hoping to find something in the cloud.

Wednesday, January 16, 2008

Accessing JSF Backing Beans from JSP

As sophisticated as it sounds, I don't really have much clue about what I'm talking here. Starting with very little knowledge on either JSF or JSP, I struggled with one simple problem for almost a week. That much searching in darkness at least deserves a blog entry I guess.

To make the long story short, in one of my CIMA portal renovations, I have a backing bean that takes a string parameter from the JSF/JSP front end, and generates an URL to be dispatched/forwarded/redirected by JSF/JSP. Stunningly, I couldn't find appropriate JSF tags that meet exactly what I need. After studying all possibilities listed in a very good summary given my restricted environment, I gave up and took the brute force approach of accessing the backing bean attribute from JSP directly:


<%
    cima.jsf.managedbeans.MyBean mbean = (cima.jsf.managedbeans.MyBean) request.getAttribute("MyBean");
    String url = mbean.get_url();
    response.sendRedirect(url);
%>

However the above code gives NULL pointer exception because MyBean was not initialized for request, likely because JSF <managed-beans> declaration in faces-config.xml didn't fall though JSP invocation path -- I encountered some statements like that out on the web. Finally I settled with a simple solution buried in a helpful discussing thread, which is to insert the following line before trying to grab MyBean object out of request:

 
javax.faces.context.FacesContext.getCurrentInstance().getApplication().getVariableResolver().resolveVariable(javax.faces.context.FacesContext.getCurrentInstance(), "MyBean");

Worked like charm, if only I understood every piece of the complete mystery.

Sunday, January 6, 2008

Short Week

with long meetings started off the year. CIMA project plans were discussed, and I messed with Log4J for a while to (slightly) improve GridFTP portlet performance further. Also spent some time on the CDI pre proposal, we'll see how far this one goes.

Friday, January 4, 2008

Log4J with Tomcat and Gridsphere

For a couple of days I've been trying to tune up (actually down) the logging level of some Gridsphere portlets, which use the Java cog kit to GridFTP files. Excessive security handshaking and encryption messages were showing up in catalina.out, even though the logging level was set to "ERROR" in log4j.properties. Obviously understanding better the Tomcat Logging Documentation or some interesting online discussions is the proper start, but here are my quick idioms:

Gridsphere puts a log4j.xml file in ${CATALINA_HOME}/common/classes/, to make Tomcat honor the Log4J settings defined there, commons-logging and log4j jars need to present in ${CATALINA_HOME}/common/lib/.

For the log4j.properties file in ${CATALINA_HOME}/webapps/some-portlet/WEB-INF/classes/ to be honored, commons-logging and log4j jars are required in the corresponding WEB-INF/lib/ directory. Those in ${CATALINA_HOME}/shared/lib/ didn't seem to count, and putting the two jars in individual web app directories incidentally got rid of some annoying log4j warning about "No appenders could be found for logger (org.apache.catalina.session.ManagerBase)" as well.

Experience drew from Tomcat 5.5.25 and Gridsphere 2.2.10.

Otherworldly Gate