Otherworldly Gate

Sunday, February 17, 2008

My PolarGrid

A PolarGrid fan in my house wrote a small program to randomly grab pictures from Flickr One Letter image group, and generated this picture to show his support.

Microformat for PolarGrid -- Amendment

Thanks to William for a concrete example, and the request of including units in microformats, here comes the revision:

Simple descriptive metadata:
InSAR 2008 01 22 1 1 coherent 3

Pulse width:
 3 us

Frequency, used in both overall system sampling and start/stop property of individual waveforms:
 Sampling/Start/Stop 720 mhz

Antenna:
 0 TX/RX 0

Array of antennas: (TX or RX)
 2 0 TX 0 1 TX 0

Sample delay:
 0 us

Blanking time:
 5 us

Data acquisition unit:
 0 50000 0 us 5 us

Array of data acquisition units:
 2 0 50000 0 us 5 us 1 50000 0 us 5 us

Spatial and temporal information of a data processing chunk. Note that existing geo microformat is used, which is outside our pg namespace:
 Start/Stop 1202755351.892651 -2.193 52.686

Individual waveform identification:
 1

Individual (jpg) file:
 MCRDS_20070917_113924_fk_image_wf_01_tx_01_rx_01.jpg

Tuesday, February 12, 2008

Prelude this interesting discussion of very human and machine-readable data formats, here is my contribution to the google ranking of PolarGrid (a.k.a. Polar Grid) project.

Background
Over the weekend William gave us an overview of metadata generated from the Matlab data processing, which will be published through web feeds. Much of the following summary comes from email excerpts.

A processed dataset corresponds to one jpg image, which consists of processed measurement values from many radar samples. Fundamental properties of each processed dataset are defined by three parameters forming a triplet: waveform (wf), transmit antenna (tx), and receive antenna (rx).

Each radar system can transmit several different waveforms, and a waveform can be transmitted and received on one or more antennas. So an example jpg filename would be TYPE_YYYYMMDD_HHMMSS_image_wf_##_tx_##_rx_##.jpg, where TYPE describes the radar data type such as InSAR, YYYYMMDD_HHMMSS corresponds to the UTC timestamp of the first sample in the dataset, <wf, tx, rx> triplet specifies the particular waveform transmitted and received on a particular antenna, and the word "image" distinguishes these data from position or reference data.
Additionally there may be pulse compressed and f-k migrated images, which correspond to
"image_pc"or "image_fk", and are different types of processed images for the same track. Word "comb" would replace wf_##_tx_##_rx_## in case of combined images too.

A plain text file will be associated with each jpg image, containing three levels of metadata.
1. Overall system configuration and setup:
 Description: "InSAR 2008 01 22 " Sampling Frequency (MHz): 720 Number of Averages (Samples): 1 Number of Waveforms: 1 TX Attenuation: [0 0] RX Attenuation: [0 0] Number of Samples: [20000 20000] DSP Mode: coherent System Delay (us): 0
 Each TX/RX Attenuation value corresponds to a tx/rx antenna. Hence in case of 2 tx antennas and 8 rx antennas, the values could be
 TX Attenuation: [10 10] RX Attenuation: [5 5 5 5 5 5 5 5]
 Each "Number of Samples" value corresponds to a data acquisition unit (daq), the total number of which is always less than or equal to the total number of rx antennas. Hence in case of 4 daqs for 8 rx antennas, the value could be
 Number of Samples: [20000 20000 20000 20000]
 For each daq, the "Number of Samples" value is the sum of the corresponding value described in individual waveform metadata.
2. Spatial and temporal information of data processing chunks:
 Start lon: (Degrees) Stop lon: (Degrees) Start lat: (Degrees) Stop lat: (Degrees) Start UTC time: (seconds.useconds) Stop UTC time: (seconds.useconds)
 The complete expedition consists of multiple (~30) data chunks, and all <wf, tx, rx> triplets in the same chunk share the same spatial and temporal chunk metadata.
3. Information about individual waveforms. Continuing the example of 2 tx antennas, 8 rx antennas, and 4 daqs, 2 waveforms can be described as:
 Start Frequency (MHz): 120 Stop Frequency (MHz): 300 Pulse Width (us): 10 Zero/Pi Mode: 1 TX Attenuation: [0 0] RX Attenuation: [0 0 0 0 0 0 0 0] Number of Samples: [15000 15000 15000 15000] Sample Delay (us): [20 20 20 20] Blanking Time (us): [20 20 20 20] Start Frequency (MHz): 120 Stop Frequency (MHz): 300 Pulse Width (us): 3 Zero/Pi Mode: 1 TX Attenuation: [5 5] RX Attenuation: [7 7 7 7 7 7 7 7] Number of Samples: [5000 5000 5000 5000] Sample Delay (us): [0 0 0 0] Blanking Time (us): [5 5 5 5]
 For each waveform, each daq also corresponds to one Sample Delay and one Blanking Time.
4. A fourth level of metadata associated with each jpg image can be deduced from its file name as discussed earlier, including radar type, specific <wf, tx, rx> triplet value, processed image type, and a timestamp.

Microformat Proposal
We propose the following set of microformats to describe the above metadata, and they can be composed to construct web feeds for individual jpg images. All class names in the proposed microformat start with "pg:" denoting the PolarGrid namespace.

Simple descriptive metadata:
InSAR 2008 01 22 1 1 coherent 3 3

Frequency, used in both overall system sampling and start/stop property of individual waveforms:
 Sampling/Start/Stop 720

Antenna:
 0 TX/RX 0

Array of antennas: (TX or RX)
 2 0 TX 0 1 TX 0

Data acquisition unit:
 0 50000 0 5

Array of data acquisition units:
 2 0 50000 0 5 1 50000 0 5

Spatial and temporal information of a data processing chunk. Note that existing geo microformat is used, which is outside our pg namespace:
 Start/Stop 1202755351.892651 -2.193 52.686

Individual waveform identification:
 1

Missing pieces: we are yet to understand the relationship between a waveform description and individual jpg images better to supply microformats describing the fourth level of metadata, especially the location of jpg images to be provided in the final web feeds.

Comments
The above proposal is a very preliminary draft, please criticize, suggest, and comment. Every single click improves the internet fame of our project too.

Tuesday, January 22, 2008

Much Ado

About seemingly trivial stuff in the last couple of weeks, like how to redirect an URL in JSF, how to completely wipe off sensitive, crucial information that were accidently, mistakenly checked into CVS, etc. But as the happy ending, I finished my part of transforming CIMA portal to use GridFTP for data transfer services. (Of course I have to mention CIMA portal often to maintain my seat in the Google search competition.)

In the coming couple of weeks, besides the PolarGrid meeting and a quick trip to the west, I expect to be watching videos and surfing webs, hoping to find something in the cloud.

Wednesday, January 16, 2008

Accessing JSF Backing Beans from JSP

As sophisticated as it sounds, I don't really have much clue about what I'm talking here. Starting with very little knowledge on either JSF or JSP, I struggled with one simple problem for almost a week. That much searching in darkness at least deserves a blog entry I guess.

To make the long story short, in one of my CIMA portal renovations, I have a backing bean that takes a string parameter from the JSF/JSP front end, and generates an URL to be dispatched/forwarded/redirected by JSF/JSP. Stunningly, I couldn't find appropriate JSF tags that meet exactly what I need. After studying all possibilities listed in a very good summary given my restricted environment, I gave up and took the brute force approach of accessing the backing bean attribute from JSP directly:


<%
    cima.jsf.managedbeans.MyBean mbean = (cima.jsf.managedbeans.MyBean) request.getAttribute("MyBean");
    String url = mbean.get_url();
    response.sendRedirect(url);
%>

However the above code gives NULL pointer exception because MyBean was not initialized for request, likely because JSF <managed-beans> declaration in faces-config.xml didn't fall though JSP invocation path -- I encountered some statements like that out on the web. Finally I settled with a simple solution buried in a helpful discussing thread, which is to insert the following line before trying to grab MyBean object out of request:

 
javax.faces.context.FacesContext.getCurrentInstance().getApplication().getVariableResolver().resolveVariable(javax.faces.context.FacesContext.getCurrentInstance(), "MyBean");

Worked like charm, if only I understood every piece of the complete mystery.

Sunday, January 6, 2008

Short Week

with long meetings started off the year. CIMA project plans were discussed, and I messed with Log4J for a while to (slightly) improve GridFTP portlet performance further. Also spent some time on the CDI pre proposal, we'll see how far this one goes.

Friday, January 4, 2008

Log4J with Tomcat and Gridsphere

For a couple of days I've been trying to tune up (actually down) the logging level of some Gridsphere portlets, which use the Java cog kit to GridFTP files. Excessive security handshaking and encryption messages were showing up in catalina.out, even though the logging level was set to "ERROR" in log4j.properties. Obviously understanding better the Tomcat Logging Documentation or some interesting online discussions is the proper start, but here are my quick idioms:

Gridsphere puts a log4j.xml file in ${CATALINA_HOME}/common/classes/, to make Tomcat honor the Log4J settings defined there, commons-logging and log4j jars need to present in ${CATALINA_HOME}/common/lib/.

For the log4j.properties file in ${CATALINA_HOME}/webapps/some-portlet/WEB-INF/classes/ to be honored, commons-logging and log4j jars are required in the corresponding WEB-INF/lib/ directory. Those in ${CATALINA_HOME}/shared/lib/ didn't seem to count, and putting the two jars in individual web app directories incidentally got rid of some annoying log4j warning about "No appenders could be found for logger (org.apache.catalina.session.ManagerBase)" as well.

Experience drew from Tomcat 5.5.25 and Gridsphere 2.2.10.