Microformat for PolarGrid

Prelude this interesting discussion of very human and machine-readable data formats, here is my contribution to the google ranking of PolarGrid (a.k.a. Polar Grid) project.

Background
Over the weekend William gave us an overview of metadata generated from the Matlab data processing, which will be published through web feeds. Much of the following summary comes from email excerpts.

• A processed dataset corresponds to one jpg image, which consists of processed measurement values from many radar samples. Fundamental properties of each processed dataset are defined by three parameters forming a triplet: waveform (wf), transmit antenna (tx), and receive antenna (rx).


• Each radar system can transmit several different waveforms, and a waveform can be transmitted and received on one or more antennas. So an example jpg filename would be TYPE_YYYYMMDD_HHMMSS_image_wf_##_tx_##_rx_##.jpg, where TYPE describes the radar data type such as InSAR, YYYYMMDD_HHMMSS corresponds to the UTC timestamp of the first sample in the dataset, <wf, tx, rx> triplet specifies the particular waveform transmitted and received on a particular antenna, and the word "image" distinguishes these data from position or reference data.
  Additionally there may be pulse compressed and f-k migrated images, which correspond to
  "image_pc"or "image_fk", and are different types of processed images for the same track. Word "comb" would replace wf_##_tx_##_rx_## in case of combined images too.
  


• A plain text file will be associated with each jpg image, containing three levels of metadata.
  
  
  1. Overall system configuration and setup:
     
Description: "InSAR 2008 01 22 "
Sampling Frequency (MHz): 720
Number of Averages (Samples): 1
Number of Waveforms: 1
TX Attenuation: [0 0]
RX Attenuation: [0 0]
Number of Samples: [20000 20000]
DSP Mode: coherent
System Delay (us): 0

     Each TX/RX Attenuation value corresponds to a tx/rx antenna. Hence in case of 2 tx antennas and 8 rx antennas, the values could be
     
TX Attenuation: [10 10]
RX Attenuation: [5 5 5 5 5 5 5 5]

     Each "Number of Samples" value corresponds to a data acquisition unit (daq), the total number of which is always less than or equal to the total number of rx antennas. Hence in case of 4 daqs for 8 rx antennas, the value could be
     
Number of Samples: [20000 20000 20000 20000]

     For each daq, the "Number of Samples" value is the sum of the corresponding value described in individual waveform metadata.
  
  
  2. Spatial and temporal information of data processing chunks:
     
Start lon: (Degrees)
Stop lon: (Degrees)
Start lat: (Degrees)
Stop lat: (Degrees)
Start UTC time: (seconds.useconds)
Stop UTC time: (seconds.useconds)

     The complete expedition consists of multiple (~30) data chunks, and all <wf, tx, rx> triplets in the same chunk share the same spatial and temporal chunk metadata.
  
  
  3. Information about individual waveforms. Continuing the example of 2 tx antennas, 8 rx antennas, and 4 daqs, 2 waveforms can be described as:
     
Start Frequency (MHz): 120
Stop Frequency (MHz): 300
Pulse Width (us): 10
Zero/Pi Mode: 1
TX Attenuation: [0 0]
RX Attenuation: [0 0 0 0 0 0 0 0]
Number of Samples: [15000 15000 15000 15000]
Sample Delay (us): [20 20 20 20]
Blanking Time (us): [20 20 20 20]

Start Frequency (MHz): 120
Stop Frequency (MHz): 300
Pulse Width (us): 3
Zero/Pi Mode: 1
TX Attenuation: [5 5]
RX Attenuation: [7 7 7 7 7 7 7 7]
Number of Samples: [5000 5000 5000 5000]
Sample Delay (us): [0 0 0 0]
Blanking Time (us): [5 5 5 5]

     For each waveform, each daq also corresponds to one Sample Delay and one Blanking Time.
  
  
  4. A fourth level of metadata associated with each jpg image can be deduced from its file name as discussed earlier, including radar type, specific <wf, tx, rx> triplet value, processed image type, and a timestamp.
  
  

Microformat Proposal
We propose the following set of microformats to describe the above metadata, and they can be composed to construct web feeds for individual jpg images. All class names in the proposed microformat start with "pg:" denoting the PolarGrid namespace.

• Simple descriptive metadata:
  
<span class="pg:description">InSAR 2008 01 22</span>
<span class="pg:averages">1</span>
<span class="pg:waveforms">1</span>
<span class="pg:dsp-mode">coherent</span>
<span class="pg:pulse-width">3</span>
<span class="pg:zero-pi-mode">3</span>



• Frequency, used in both overall system sampling and start/stop property of individual waveforms:
  
<span class="pg:frequency>
<span class="pg:name">Sampling/Start/Stop</span>
<span class="pg:mhz">720</span>
</span>



• Antenna:
  
<span class="pg:antenna">
<span class="pg:id">0</span>
<span class="pg:type">TX/RX</span>
<span class="pg:attenuation">0</span>
</span>



• Array of antennas: (TX or RX)
  
<span class="pg:antenna-array">
<span class="pg:array-size">2</span>
<span class="pg:antenna">
<span class="pg:id">0</span>
<span class="pg:type">TX</span>
<span class="pg:attenuation">0</span>
</span>
<span class="pg:antenna">
<span class="pg:id">1</span>
<span class="pg:type">TX</span>
<span class="pg:attenuation">0</span>
</span>
</span>



• Data acquisition unit:
  
<span class="pg:data-acquisition-unit">
<span class="pg:id">0</span>
<span class="pg:samples">50000</span>
<span class="pg:delay">0</span>
<span class="pg:blanking">5<span>
</span>



• Array of data acquisition units:
  
<span class="pg:data-acquisition-unit-array">
<span class="pg:array-size">2</span>
<span class="pg:data-acquisition-unit">
<span class="pg:id">0</span>
<span class="pg:samples">50000</span>
<span class="pg:delay">0</span>
<span class="pg:blanking">5<span>
</span>
<span class="pg:data-acquisition-unit">
<span class="pg:id">1</span>
<span class="pg:samples">50000</span>
<span class="pg:delay">0</span>
<span class="pg:blanking">5<span>
</span>
</span>



• Spatial and temporal information of a data processing chunk. Note that existing geo microformat is used, which is outside our pg namespace:
  
<span class="pg:data-chunk">
<span class="pg:name">Start/Stop</span>
<span class="pg:utc-timestamp">1202755351.892651</span>
<span class="geo">
<span class="longitude">-2.193</span>
<span class="latitude">52.686</span>
</span>
</span>



• Individual waveform identification:
  
<span class="pg:waveform">
<span class="pg:id">1</span>
</span>



• Missing pieces: we are yet to understand the relationship between a waveform description and individual jpg images better to supply microformats describing the fourth level of metadata, especially the location of jpg images to be provided in the final web feeds.

Comments
The above proposal is a very preliminary draft, please criticize, suggest, and comment. Every single click improves the internet fame of our project too.

Much Ado

About seemingly trivial stuff in the last couple of weeks, like how to redirect an URL in JSF, how to completely wipe off sensitive, crucial information that were accidently, mistakenly checked into CVS, etc. But as the happy ending, I finished my part of transforming CIMA portal to use GridFTP for data transfer services. (Of course I have to mention CIMA portal often to maintain my seat in the Google search competition.)

In the coming couple of weeks, besides the PolarGrid meeting and a quick trip to the west, I expect to be watching videos and surfing webs, hoping to find something in the cloud.

Accessing JSF Backing Beans from JSP

As sophisticated as it sounds, I don't really have much clue about what I'm talking here. Starting with very little knowledge on either JSF or JSP, I struggled with one simple problem for almost a week. That much searching in darkness at least deserves a blog entry I guess.

To make the long story short, in one of my CIMA portal renovations, I have a backing bean that takes a string parameter from the JSF/JSP front end, and generates an URL to be dispatched/forwarded/redirected by JSF/JSP. Stunningly, I couldn't find appropriate JSF tags that meet exactly what I need. After studying all possibilities listed in a very good summary given my restricted environment, I gave up and took the brute force approach of accessing the backing bean attribute from JSP directly:
<%
cima.jsf.managedbeans.MyBean mbean = (cima.jsf.managedbeans.MyBean) request.getAttribute("MyBean");
String url = mbean.get_url();
response.sendRedirect(url);
%>


However the above code gives NULL pointer exception because MyBean was not initialized for request, likely because JSF <managed-beans> declaration in faces-config.xml didn't fall though JSP invocation path -- I encountered some statements like that out on the web. Finally I settled with a simple solution buried in a helpful discussing thread, which is to insert the following line before trying to grab MyBean object out of request:
javax.faces.context.FacesContext.getCurrentInstance().getApplication().getVariableResolver().resolveVariable(javax.faces.context.FacesContext.getCurrentInstance(), "MyBean");


Worked like charm, if only I understood every piece of the complete mystery.

Short Week

with long meetings started off the year. CIMA project plans were discussed, and I messed with Log4J for a while to (slightly) improve GridFTP portlet performance further. Also spent some time on the CDI pre proposal, we'll see how far this one goes.

Log4J with Tomcat and Gridsphere

For a couple of days I've been trying to tune up (actually down) the logging level of some Gridsphere portlets, which use the Java cog kit to GridFTP files. Excessive security handshaking and encryption messages were showing up in catalina.out, even though the logging level was set to "ERROR" in log4j.properties. Obviously understanding better the Tomcat Logging Documentation or some interesting online discussions is the proper start, but here are my quick idioms:

• Gridsphere puts a log4j.xml file in ${CATALINA_HOME}/common/classes/, to make Tomcat honor the Log4J settings defined there, commons-logging and log4j jars need to present in ${CATALINA_HOME}/common/lib/.


• For the log4j.properties file in ${CATALINA_HOME}/webapps/some-portlet/WEB-INF/classes/ to be honored, commons-logging and log4j jars are required in the corresponding WEB-INF/lib/ directory. Those in ${CATALINA_HOME}/shared/lib/ didn't seem to count, and putting the two jars in individual web app directories incidentally got rid of some annoying log4j warning about "No appenders could be found for logger (org.apache.catalina.session.ManagerBase)" as well.

Experience drew from Tomcat 5.5.25 and Gridsphere 2.2.10.

Around and Beyond SC07

Before I even noticed, two whole months flew around that emotional and exhausting conference in Reno. I was unintentionally and indirectly reminded the other day that anxious readers have been waiting for news updates, here comes the chronicle summary I guess.

• CIMA High Availability prototype finished before SC07. Several application level bugs and protocol limitations have been discovered, fixed, or planned for improvements. After enormous physical cable pulling tests in the machine room, we were finally confident with what to expect in various failure cases, and set up the two failover servers in geographically distributed locations. Well, 50 miles apart may not be much, but it's better than 5 inches.


• SC07 had been quite eventful: attended GCE07 because my name incidentally appeared in one of the papers; gave a booth presentation on the CIMA High Availability project; served as the mobile communication center between reno booth and CIMA east coast headquarter office; met old friends scattered in booths around the show floor; and most important of all, witnessed the great conquer of data capacitor winning the bandwidth challenge, which makes us all proud.


• Heading back and onto the portal world, we started a project to replace open direct data links in CIMA portal with GridFTP on demand transferring services. The idea is rather straightforward, while sorting through complex code structures resulted from a blend of Windows and Linux developers over the years had not been. A prototype is now in place after all the funs with jars, and the portlet is about twice slower than the original. Our clients cheered whole-heartedly, "Impressive! Much faster than I expected!" Though paying a performance price in exchange for security is expected, we are still seeking ways to bargain.


• iCenter project that collaborates with the biology department to establish data management infrastructures for a new light-microscopy core facility is steadily progressing. A couple of productive meetings acquiring user requirements have been held.


• WIYN ODI Data Pipeline and Distribution project is also marching forward, and we had a whole day planning and discussion meeting with visitors from LSST and NOAO just recently.


• Now talking about meetings, there had been a few others too, like final TeraGrid weekly status meeting, status meeting with glorious data capacitor, and strategic meeting preparation meeting. Does Skype video conferencing count?

Looking forward, a couple of other portal projects are in the lineup too, and I'm hoping to work/write more about the high availability project with DRBD and heartbeat in my copious free time, before it's completely spaced.

With the holiday season right around the corner, I'm very attempted to make a resolution about more dedicated blogging in the coming new year, via a youtube-banned video. So long, 2007.

Leopard widget repair DIY

After (thankfully) upgrading to Leopard, a couple of my favorite dashboard widgets started to have problems. Obviously it's hard for me to live without knowing the current weather outside, or the current date in Lunar calendar, I started my hunting trip of those little tiny lines of codes that broke under the latest Safari/Leopard, and here's a summary of how:

• Sharp the weapon: in Developer Tools that came with Leopard, Dashcode is a nice debugging environment that one can view, modify and test run widgets from source. Once installed, it can be found under /Developer/Applications/
  


• Know the enemy: all downloaded third-party widgets are installed in ./Library/Widgets/ under user's home directory, and the widget source can be viewed via "Show Package Contents" option with right mouse click.


• Onto the weather: after downloading WeatherBug Local Weather version 1.1.0.9 released on Nov. 7, 2007, Leopard complains that it can't be installed because it's not a widget. Dashcode revealed the truth that file Info.plist is missing. Actually it was not missing, but named as info.plist, with the small "i". That's all needed to get radar maps back on board.


• Finally the date: China Calendar widget stopped reacting to mouse clicks on any buttons under Leopard, and its "confirm" button appeared to have misalignment on edges. Only file mycal.html needs to be modified for complete repair: replacing the parameter this.tag with this.id in all mouse movement functions fixes the former, and specifying source of genericButton.js to use the system one at file:///System/Library/WidgetResources/button/genericButton.js fixes the later.

One less thing to be unhappy about Leopard.