Thursday, December 31, 2009

Moving to the Cloud Part 2: Email

In my last post, I mentioned my goal of moving all of my important data out of local storage and into the cloud. First, I dealt with my photos. My next order of business was email.

Traditionally, I used Outlook to organize my email and contacts. So I had years' worth of messages organized into personal folders on a local hard drive. Whenever I wanted to move to a new machine, I would export all of these as .pst files and re-import them in a new location. That was annoying, risky, and error-prone.

Now that I am paying for all of this Google storage (as discussed in my last post), it seemed obvious that Gmail was the place to move all of my old email. I wasn't sure off the top of my head exactly how to do that, but a quick search turned up many variations on a simple trick -- enable IMAP in Gmail settings, bring the Gmail account into Outlook via IMAP, and drag-and-drop folders and messages. Perfect!

As for my contacts, I imported them into the Google space (via Google Voice) a long time ago. And I never did a lot of personal calendaring with Outlook. (Online solutions are far better for that now anyway.)

Overall, I was very pleased at how easy it was to migrate away from Outlook. There are two more things that I care about enough to want to move to the cloud -- my music and my schoolwork/teaching/research files. I will cover those in future posts.

Wednesday, December 30, 2009

Moving to the Cloud Part 1: Photos

Well, it's been a long time since I've posted. Without going into a lot of details, I've had a number of life changes in the last few months. Hopefully I can get back into blogging a bit.

One goal I've made for 2010 is to care a lot less about what, exactly, is on the fleet of machines and disk drives sitting around my house. That means moving content into the cloud. I decided to start with what is most important to me, namely, my family photos.

I was a pretty early adopter of digital photography, getting my first camera in 1997. So I have a lot of pictures. I've used Picasa for a long time (well before Google bought it), so it seemed like a natural fit to upload everything into Picasa Web Albums. And of course, there is a web API, so in the future, I can manage my photos with custom scripts and applications.

The Picasa fat client application makes the upload process incredibly easy. (I used the beta version for Linux, which now runs via the WINE emulator.) One thing I had to make sure to do was force it to upload the full-size images for each album, lest I lose some JPEG quality for more recent years' photos.

Another thing I had to do was upgrade my Google Account storage, because the free space wasn't going to be nearly enough. Conveniently, there is currently a deal to get a free Eye-Fi card with a year's worth of 200G of storage. I went for that. I don't really need anywhere near 200G just for my photos, but I will use some more of that space in future blog posts. And I can't wait for my new Eye-Fi card to arrive in the mail.

Saturday, October 24, 2009

Testing Exception Messages, JUnit 4.7 Edition

Historically, JUnit has not elegantly supported verification of expected-exception messages in unit tests. Some time ago, I wrote a post about techniques for handling this situation. Brian Brooks (clearly a diligent web searcher) found that post and was kind enough to send me a message pointing out that the latest version of JUnit, 4.7, finally includes support for this incredibly important and common testing scenario.

In honor of this exciting occasion, I wanted to post an example. Here is the class I'm going to test:
public class PositiveInteger {
 private int num;

 public PositiveInteger(int num) {
  if(num <= 0) {
   throw new IllegalArgumentException("Number must be positive");
  }
  this.num = num;
 }
}
Let's test -- hmmm -- the constructor, I guess. I created two test cases -- one which passes, and one which intentionally fails (just to show the resulting output):
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;


public class TestPositiveNumber {

 @Rule
 public ExpectedException thrown = ExpectedException.none();

 @Test
 public void testConstructorInvalid_PASS() {
  thrown.expect(IllegalArgumentException.class);
  thrown.expectMessage("Number must be positive");
  new PositiveInteger(-5);
 }

 /**
  * Intentionally-failing test to demonstrate the results
  * when checking an exception message.
  */
 @Test
 public void testConstructorInvalid_FAIL() {
  thrown.expect(IllegalArgumentException.class);
  thrown.expectMessage("Not gonna pass");
  new PositiveInteger(-5);
 }
}
JUnit now supports a style embodied in "rules", which get marked with the @Rule annotation and provide a means by which the test execution behavior can be modified. You can see that here with the "thrown" field, of type "ExpectedException". It would be great if a) that field didn't have to be public, and b) the functionality could be implemented using a local variable, but unfortunately the Java language won't currently support such an implementation. So a public field it is. In a test, you can configure the rule. Here I do that with calls to the "expect" and "expectMessage" methods. When JUnit executes the test case, it checks the result against the rule and fails if the rule is not matched. In the test case that does not pass, the failure takes this form:
java.lang.AssertionError: 
Expected: (exception with message a string containing "Not gonna pass" and an instance of java.lang.IllegalArgumentException)
     got: 

 at org.junit.Assert.assertThat(Assert.java:778)
 at org.junit.Assert.assertThat(Assert.java:736)
Very useful. Thanks to Kent Beck for adding this support.

Saturday, September 26, 2009

URLEncoder Demonstrates API Design Hazards

I had to use java.net.URLEncoder yesterday. This experience was a great reminder of how annoying this API is. It demonstrates a couple of issues, so I thought I would comment on it here.

The first thing you can see here is how important it is to get the API right the first time, lest things go downhill from there. In its original form, URLEncoder provided only one static method which encoded a string as "application/x-www-form-urlencoded" (suitable for issuing an HTTP request). This method has the signature encode(String), and it used the platform's default encoding. Unfortunately, this is technically hazardous, because URL encodings are supposed to always be UTF-8. Therefore, wiser heads built and used replacement classes (such as URLCodec from Apache Commons).

Java 1.4 finally "fixed" this deficiency in the platform with a new encode(String,String) overload, the second parameter being the name of a character encoding to use. The old encode(String) method is marked as deprecated. The documentation itself states that: Note: The World Wide Web Consortium Recommendation states that UTF-8 should be used. Not doing so may introduce incompatibilites.

I don't know, but I'm guessing that the powers-that-be would have liked to make encode(String) do UTF-8 encoding. But they felt constrained by the legacy decision to use the platform default, in the name of not breaking any existing code. Never mind the substantial odds that any such existing code is already broken in this case wherever the default encoding is not UTF-8.

But wait, it gets worse. The encode(String,String) method also throws UnsupportedEncodingException whenever (surprise!) the second parameter does not refer to a valid encoding. I'm no rocket scientist, but it seems to me that if you're following the W3C recommendation and always passing UTF-8, you don't have to worry about this exception being thrown. And yet because the exception is checked, you have no choice but to handle it and clutter up your code, perhaps with a catch block that just does assert(false).

This all begs the question, if the second parameter should always be "UTF-8" for some definition of "always", why should the API require a second parameter at all? I submit that a better approach would have been to create, instead of or in addition to encode(String,String), an encodeUTF8(String) method that always did UTF-8 encoding without the added hassle of passing a (constant) parameter and dealing with a checked exception...

Sunday, September 13, 2009

"How to Think About Algorithms" Book

I have been gradually making my way through "How to Think About Algorithms" by Jeff Edmonds.

I have recently recommended this book to several people, and I wanted to mention it here, too. Edmonds uses what I'll call a more intuitive approach to algorithms that I think would be appealing to those with an engineering mindset. Here I'm constrasting it with the texts that I've used as a student in algorithms courses -- the classic Sedgewick and CLRS books.
(Maybe more importantly in this economy, though, the subject matter and exercises in this book are a rich source of review for those preparing for tech interviews!)

Wednesday, September 2, 2009

Tailing Inside Eclipse

I'm currently working on a project that writes to a log file inside my workspace when I test-execute it. Obviously it's useful to actually view that log file once in awhile, but to do so inside Eclipse requires regularly F5-refreshing it in the Project Explorer. I started to wish that Eclipse would prompt me to automatically reload the file when it detected modification, like some text editors do. But then it occurred to me how convenient it would be if I could actually tail the log from inside Eclipse itself. After all,

The less I leave the IDE, The more productive I can be.

(That rhymes.)

This seemed like an obvious enough desire that somebody would have built a plugin for it already, and sure enough, a search turned up a couple of different examples. The one that I went with is called NTail. It works like a champ. It also has a few extra nice features, like the ability to highlight lines in the file based on a regular expression. If this sounds useful to you, I'd encourage you to check it out.

Saturday, August 8, 2009

Changing Return Values to Exceptions, Java/AspectJ Edition

In the last couple of posts, I've been talking about APIs that communicate errors via return values, and how to map those to exceptions. As we saw most recently, one reasonably effective approach in C++ makes use of macros, but it requires a degree of development discipline, and the resulting code remains somewhat unsatisfactory.

In this post, I will present an approach for Java that mitigates these issues. Specifically, you can neatly map return values to exceptions using AspectJ.

(Note that I am using Eclipse for this work. It's great that AspectJ is now an Eclipse project, because it can be incredibly frustrating to use without solid IDE support.)

I'll start with some Java math utility routines that are roughly equivalent to what I was working with in C++. One difference is that in Java, I don't need a custom sqrt() wrapper because the Math.sqrt() method already uses a return value, Double.NaN, to indicate error.
package com.scottmcmaster365.math;

public class MyMathUtils
{
  public static double radiusFromArea(double area)
  {
    return Math.sqrt(area / 3.14);
  }

  public static double circumferenceFromRadius(double radius)
  {
    return 2 * 3.14 * radius;
  }
}
I want to wire up an aspect that runs when these functions are returning, checks for NaN, and throws an exception. With a bit of understanding of AspectJ, this is quite simple:
package com.scottmcmaster365.aspects;

import com.scottmcmaster365.math.MyMathUtils;

/**
 * This aspect maps double NaN return values to runtime exceptions.
 * (For instructional purposes only.)
 * @author Scott McMaster
 *
 */
public aspect MapNaNResultToExceptionAspect
{
  pointcut mathClass() : within(MyMathUtils);
 
  pointcut mathMethods() : mathClass() && execution(* *(..));
 
  after () returning (double d) : mathMethods()
  {
    if( Double.isNaN(d) )
    {
      throw new RuntimeException("Result is NaN");
    }
  }
}
Hopefully even if you're not familiar with AspectJ, that makes at least a little sense to you. Otherwise, I suggest checking out a good tutorial. But in a nutshell, the pointcuts pick out the methods in the MyMathUtils class, and the after advice grabs the return value and allows us to inject code at compile-time.

With this aspect in effect, the client for finding the circumference given the area of a circle will throw a RuntimeException during the call to radiusFromArea():
final double TESTAREA = -1.0;
double radius = MyMathUtils.radiusFromArea(TESTAREA);
double circumference = MyMathUtils.circumferenceFromRadius(radius);
System.out.println(circumference);
That's exactly what I was after: Clean code that doesn't have to check return values explicitly or via macros. Score one for AspectJ. In fact, by making a very minor change to the pointcuts and advice spec, I can even fire the aspect at the Math.sqrt() call site, which (in a more complicated example) narrow the defect down even further. I'll leave this as an exercise for the reader unless someone comments that they want to see it.

Thursday, August 6, 2009

Changing Return Values to Exceptions, C++ Edition

In my last post, I discussed one reason why exceptions are superior to return values from the perspective of an API client. Despite this, you might still run into APIs (albeit hopefully legacy ones) that heavily use status codes and magic return values to indicate errors, and it may not be within your sphere of influence to change. A friend of mine asked me how to deal with this situation.

Ideally we would like to map error results to exceptions. One approach that works in C++ is to leverage the preprocessor via a "failure-checking" macro.

Consider my example from the last post which uses a couple of simple functions to calculate the circumference of a circle given the area. Here is the original client code:
float area = TEST_AREA;
float radius = radiusFromArea(area);
float circumference = circumferenceFromRadius(radius);
cout << circumference << endl;
Let's say that "radiusFromArea" uses the original version of the "mysqrt" function which returns -1 when it receives invalid input, and let's also stipulate that we cannot change the API for any of these functions. We'd like to present an exception to our callers as soon as we detect a bad value returned from one of the functions. We could do that with some spaghetti-ish code:
float area = TEST_AREA;
float radius = radiusFromArea(area);
if( radius < 0 )
{
  throw runtime_error("cannot be negative");
}
float circumference = circumferenceFromRadius(radius);
if( circumference < 0 )
{
  throw runtime_error("cannot be negative");
}
cout << circumference << endl;
(Ideally we would get the exception from "mysqrt" like I demonstrated last time, but recall that for the purposes of this post, we're pretending that we cannot change that function. And yes, in this contrived situation, we could easily check the negativeness of "area", but that would defeat the instructional value of the example.)

As far as isolating where a negative value might be coming from, that works. But it's annoying and ugly and it takes too much typing. That's why C/C++ programmers traditionally tend to be pretty bad about this type of error-checking. We can clean it up considerably with a macro:
#define IfNegativeThrow(x) { \
    if(x < 0 ) \
      throw runtime_error("cannot be negative"); \
    }
The new client code becomes:
float area = TEST_AREA;
float radius;
IfNegativeThrow(radius = radiusFromArea(area));
float circumference;
IfNegativeThrow(circumference = circumferenceFromRadius(radius));
cout << circumference << endl;
This is functionally equivalent to the immediately-preceding example, but it is more concise and (in my opinion) cleaner. You can potentially improve on it by adding an extra macro parameter for the exception message. Macros similar to IfNegativeThrow() can be found in many industrial-strength C++ codebases. This pattern is especially prevalent in old-school COM programming, where it is often used to deal with the infamous HRESULT.

(Shout-out to anybody who still does COM programming like that on a regular basis. Last time I did, I think Bill Clinton was still President.)

There remain some significant drawbacks here. Doing those assignments inside macros still leaves you with less-than-aesthetic code. You still have to maintain discipline about using the macros. And if you are in a language that doesn't support the preprocessor, things get more difficult and/or annoying. (You may have to implement IfNegativeThrow() as a full-fledged method, for example, and incur the requisite overhead.) In a later post, I'll address these issues.

Tuesday, August 4, 2009

Exceptions Versus Return Values

I'm alarmed by the number of programmers I've been talking to lately -- not just C/C++ programmers, but also Java and C# programmers -- who are inclined to forgo the throwing of exceptions and instead use magic return values and/or status codes to indicate an API failure.

Consequenty I sat down to write a blog post about this. As a general rule, before I write something, I check to see if someone else has done it already, and sure enough, a number of people have taken on this issue competently. So instead of imitating their work, I wanted to point you to the best example, namely Ned Batchelder's.

One slight addition I wanted to make was to point out that Ned's article is primarily focused on the use of return values as status codes. I'm actually more concerned about overloading the return value with a sentinel value that is intended to indicate failure (a practice that Ned covers in the section on "Valuable channels"). In my opinion, this is far more insidious than the correct application of the status code idiom because it can time- and space-shift errors far from where they originate, which can unnecessarily complicate debugging.

Take the following square-root function as an example. Let's use C++.
float mysqrt(float n)
{
  if( n < 0 )
  {
    return -1.0;
  }
  return sqrt(n);
}
Here, -1 is being used as a sentinel value to indicate what any high school math student knows, namely, that you can't take the square root of a negative number. Consider what a client of this function must go through. Say I'm starting with the area of a circle, and I want to calculate the circumference using a couple of convenient helper methods:
float radiusFromArea(float area)
{
  return mysqrt(area / 3.14);
}

float circumferenceFromRadius(float radius)
{
  return 2 * 3.14 * radius;
}
Here is the client code:
float area = TEST_AREA;
float radius = radiusFromArea(area);
float circumference = circumferenceFromRadius(radius);
cout << circumference << endl;
If TEST_AREA is negative, we notice that the circumference is whacked (-6.28) when we are all done. That is, hopefully we notice and don't propagate the bad value forward, but I'll give the author of this code the benefit of the doubt for now. Even so, we still have no idea if the error is in radiusFromArea, circumferenceFromRadius, or some unknown function that one of them calls. Now condider what happens if mysqrt is written to throw an exception rather than return a "magic" error value:
float mysqrt(float n)
{
  if( n < 0 )
  {
    throw invalid_argument("can't take sqrt of negative number");
  }
  return sqrt(n);
}
When this code runs on a negative area for input, you see the error immediately in the form of an unhandled exception, with a stack trace pointing right to where mysqrt is called with the bogus value. This is much preferable for debugging, even in this simplistic example. Hopefully this will help convince you that exceptions are superior to the alternatives.

Wednesday, June 24, 2009

Download Eclipse 3.5 Galileo With Amazon S3 and CloudFront

One piece of exciting news today is that Eclipse 3.5 (Galileo) is available. But the best part is that you can download it via Amazon S3 and CloudFront. As of earlier today at least, the "old-fashioned" mirrors were dragging, but the S3 download was extremely snappy.

Unfortunately, the S3 link isn't very well-advertised on the download page, so I wanted to mention it here. When you get to the download page for the distro you want, look down the page for "Download from Amazon Web Services". I bet you'll be impressed.

Saturday, June 20, 2009

Mock Responses with Optional Elements in soapUI

SoapUI will probably always be one of my favorite development and testing tools. Recently I had the opportunity to explore its web service simulation and mocking capabilities.

I needed to create some fairly-detailed mock responses for a service where pretty much every element in schema for the response was optional. By default, soapUI does not add optional elements to the mock response XML. This was causing me a significant amount of pain. I noticed in the Preferences, WSDL Settings, that there was an "Include Optional" option that says "always include optional schema elements when creating requests": I discovered that this option not only applies to creating requests, but also to creating mock responses. Problem solved. But I couldn't find this documented anywhere, so I wanted to point it out.

Wednesday, June 17, 2009

An Example of Why You Shouldn't Test in Production

If you ever need a concrete example of why it is worthwhile to set up a proper testing environment for your online or data-based applications, consider this case. According to the Chicago Tribune, Illinois resident Tom Feddor, pround owner of license plate number of "0" has received some 170 bogus parking tickets because

A glitch occurred at the Chicago Department of Revenue involving Feddor's 0 plates being used during tests of ticketing equipment.

The article continues:

It turned out that some city parking-enforcement aides punched in 0 when testing their electronic ticket-issuing devices, Revenue Department spokesman Ed Walsh said. Officials weren't aware there was a 0 plate or that Feddor was receiving tickets, Walsh said in response to the Tribune inquiry.

But thank goodness:

"But we are taking steps to rectify the situation so in the future an actual registered plate number will not be used to do the testing," Walsh said.

Before you laugh too hard, be honest: How many of you have run "test" transactions through your live/production systems exactly in this manner? And what precautions did you have in place?

Thursday, June 11, 2009

Why We Automate Tests

James Whittaker, who is now blogging at the Google Testing Blog, talked a bit about manual versus automated testing. First he implied that testers are unjustifiably infatuated with their test automation frameworks:
Too often we are solution focused and marry ourselves to our automation rather than the product we are testing. Whenever I see testers that seem more vested in their automation than in the product they are shipping I worry the pendulum has swung too far.
He also tried to relate this question back to good test design.
Let's shift the debate to good test design. Whether those tests ultimately end up being executed manually or via a tool is a moot point, let's just make them good ones that solve real testing problems.
I'm all for good test design and solving real testing problems, but how tests are run is not a moot point, and I'd like to give all of those "solution-focused" testers a little more credit than that. Let me start with some simple logic:
  1. Any series of steps that you perform more than once on a computer should be automated if at all possible.
  2. A test case is (at least in some sense) a series of steps that you perform on a computer.
  3. You want to run your test cases more than once (i.e. to perform regression testing).
  4. Therefore, you should automate your test cases.
Occasionally, I'll meet an engineer that seems satisfied to autonomically do the same things over and over, day after day, but for the most part, testers understand my line of reasoning better than most people.

If automating a test case was as simple as running that test case in the first place, there would be nothing more to say, and this post would be finished. Of course, that's not reality, and therein lies the rub. But I would argue that you're still better off automating far more often than not. Automation yields significant benefits to both cost and quality.

Imagine that you design a test case and execute it the first time for a cost of x. In my experience, if a test case is ever going to detect a bug, there is a >90% chance that will occur the first time you run it. But if it really is a good test case like James is arguing for, you want to run it again and again, ideally with each new build of the software to look for regressions and related bugs.

Let's say that the cost to automate a test case is 10x what it takes to run it manually. If you only expect to run that test case five times over the life of the software, you should stick with manual testing. But if you expect to run it more than ten times over the life of the software, it is in your long-term financial interest to automate it.

Now let's say that you do expect to run the test case ten times, and you decide to forgo automation. If you want achieve the same quality assurances with your testing, you are committing to nine additional manual executions of that test. This is where quality risks start to creep into the process. How likely is it that you will really get all nine of those executions in given your busy schedule? How timely will they be? Will you perform them consistently and correctly, or will you make some mistakes and miss some bugs? Automation solves those problems.

Finally, what if you can build a tool that drives the cost differential between manual and automated testing from 10x down to 5x? And you can leverage that tool over the long-term not for one but rather n test cases? Granted, I am ignoring a couple of important factors, like the cost of automated-test-case maintenance and the magnitude of x relative to the release schedule. But I submit that the gist of the reasoning still holds. In many cases, we can build tools that work this well or better, maybe even for less than the 5x cost in this example. That is why testers work so hard at building test automation tools.

Sunday, May 24, 2009

Streaming Office 2007 Documents Requires New Content Types

Streaming out content that opens in Microsoft Office applications is a fairly common feature in web applications. This involves setting a Content-Type header to the appropriate MIME type. The new XML/ZIP-based file formats in Office 2007 bring with them new MIME types. There is a complete list of those here.

Another common technique is to stream XML or HTML and basically lie about the content type to get data presented in Word or (especially) Excel. After all, generating true native Office-formatted files on a web server generally involves third-party components, and depending on your platform and specific needs, those can get expensive. But Excel 2007 creates a gotcha for this use case called "extension hardening", where Excel actually verifies that a file format matches the extension it claims to be before opening it, and if not, gives an annoying, scary prompt to the user.

I first experienced this last week, with Excel 2007 trying to open an Excel 2007 document from a web site which set the content type to "application/vnd.ms-excel" -- i.e. an Excel 2007 document claiming to be an Excel 2003 document. This is bothersome, but at least it's easy to work around by changing the content type to "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet". (That's a mouthful.) It doesn't do much for the HTML case, however, as described in the above link. No good workaround...

Sunday, April 12, 2009

Latest Google Toolbar Globally Hijacks Ctrl-Space, Causes Massive Eclipse Irritation

I installed the latest version of the Google Toolbar a couple of days ago. Shortly thereafter, I did a bit of work in Eclipse and noticed that Ctrl-Space no longer invoked Eclipse Content Assist, but rather brought up a "helpful" Google search window. I have since learned that Google calls this the "quick search box", and the behavior is system-wide, at least on Windows. More importantly, I discovered how to turn it off: There is an option for it in the Google Toolbar Options dialog as shown below. Certainly they could have come up with something better than Ctrl-Space, couldn't they?

Monday, March 23, 2009

Perils of (Ab)using the java.ext.dirs Property

I frequently use the java.ext.dirs system property as a shortcut for setting the classpath for a command-line Java program, i.e.:

java -Djava.ext.dirs=lib com.scottmcmaster365.MyMainProgram

No, this isn't great for production use, but for quick-and-dirty testing, it's much more convenient than creating a launcher batch file / shell script or manually typing a ginormous claspath.

Well, this practice jumped up and bit me the other day. After the requisite amount of pain and suffering, I observed that setting java.ext.dirs does not extend but rather overrides the real optional-extension location (at least with Sun's JVM), as sort-of-alluded-to here. As a consequence in this situation, Java will, at runtime, lose track of your optional packages. In my case, this manifested itself as an inability to load the JCA crypto providers (i.e. ClassNotFoundException).

If you're not using any of the standard optional packages, setting your own extensions path doesn't seem to matter. But I guess the moral of the story is that you shouldn't count on the extension mechanism to stand in for the classpath mechanism, even in all expedient situations.

Wednesday, March 11, 2009

Generating Strongly-Typed Properties Classes With openArchitectureWare, Part 4: Generating Java Classes

In the fourth part of this series, I'm finally read to generate strongly-typed wrappers for Java .properties files. If you would like to review how I got to this point, check out the earlier installments here, here, and here.

First, a little bit about my code generation philosophy. I prefer to only generate specializations -- the things that will actually vary from model to model -- when possible. Inheritance works well for this: I can put common functionality in base classes and generate subclasses. This cuts down on the amount of code that I have to work with in the template. As an added bonus, the base class can be tested (and perhaps even used) independently of the code generation process.

With that in mind, I will start by creating base classes. Recall that my PropertySet is defined as supporting two different "load types" -- resource bundle and classloader resource. That means two base classes, which I will call ResourceBundleProperties and ClassLoaderResourceProperties. Since my goal is to demonstrate openArchitectureWare, model-driven development, and code generation, and not to write the be-all-and-end-all of .properties file loaders, this code won't support everything you can imagine. Generally speaking, I'm following the tried-and-true approaches described here.

ResourceBundleProperties.java

package com.scottmcmaster365.properties;

import java.util.Properties;

/**
 * Load a properties file from a resource bundle
 * following the approach described here:
 *  http://www.javaworld.com/javaworld/javaqa/2003-08/01-qa-0808-property.html?page=2
 * @author scottmcm
 *
 */
public class ResourceBundleProperties
{
 protected java.util.Properties properties;
 
 /**
  * Load the properties from the resource bundle with the given name.
  * @param name
  */
 public ResourceBundleProperties(String name)
 {
  ClassLoader loader = Thread.currentThread ().getContextClassLoader();

  final java.util.ResourceBundle rb = java.util.ResourceBundle
    .getBundle(name, java.util.Locale.getDefault(), loader);

  properties = new java.util.Properties();
  for (java.util.Enumeration<String> keys = rb.getKeys(); keys
    .hasMoreElements();) {
   final String key = (String) keys.nextElement();
   final String value = rb.getString(key);

   properties.put(key, value);
  } 
  
  if (properties == null) {
   throw new IllegalArgumentException("Could not load " + name + " as resource bundle");
  }
 }
 
 public Properties getProperties() {
  return properties;
 }
}


ClassLoaderResourceProperties.java

package com.scottmcmaster365.properties;

import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;

/**
 * Load a properties file from a resource bundle
 * following the approach described here:
 *  http://www.javaworld.com/javaworld/javaqa/2003-08/01-qa-0808-property.html?page=2
 * @author scottmcm
 *
 */
public class ClassLoaderResourceProperties
{
 protected java.util.Properties properties;
 
 private static final String SUFFIX = ".properties";

 /**
  * Load the properties from the resource bundle with the given name.
  * @param name
  * @throws IOException 
  */
 public ClassLoaderResourceProperties(String name) throws IOException
 {
  ClassLoader loader = Thread.currentThread ().getContextClassLoader();

  InputStream in = null;
  try {
   if (!name.endsWith(SUFFIX)) {
    name = name.concat(SUFFIX);
   }

   in = loader.getResourceAsStream(name);
   if (in != null) {
    properties = new java.util.Properties();
    properties.load(in);
   }
  } finally {
   if (in != null)
    try {
     in.close();
    } catch (Throwable ignore) {
    }
  }
  
  if (properties == null) {
   throw new IllegalArgumentException("Could not load " + name + " as classloader resource");
  }
 }
 
 public Properties getProperties() {
  return properties;
 }
}


Next I create a code generation template (an .xpt file, written in the oAW Xpand template language) to create a subclass based on the information in PropertySet models.

JavaWrapperTemplate.xpt

«IMPORT metamodel»

«DEFINE javaClass FOR properties::PropertySet»
 «FILE Package.replaceAll("\\.", "/") + "/" + Name + ".java"»
  package «Package»;
  
  /**
   * Load a properties file based on the approach described here:
   *   http://www.javaworld.com/javaworld/javaqa/2003-08/01-qa-0808-property.html?page=2
   */
  public class «Name-»
  «IF LoadAs == properties::LoadType::CLASSLOADER_RESOURCE-»
   extends com.scottmcmaster365.properties.ClassLoaderResourceProperties
  «ELSE-»
   extends com.scottmcmaster365.properties.ResourceBundleProperties
  «ENDIF-»
  {
   public «Name»() 
    «IF LoadAs == properties::LoadType::CLASSLOADER_RESOURCE»
     throws java.io.IOException
    «ENDIF»
   {
    super("«ResourceName»");
   }
   
   «EXPAND property FOREACH Properties»
   «EXPAND propertyWithDefault FOREACH Properties»
  }
 «ENDFILE»
«ENDDEFINE»

«DEFINE property FOR properties::Property»
 public «Type» get«Name»() {
  «IF Type == properties::PropertyType::Integer»
   return Integer.parseInt(properties.getProperty("«Name»"));
  «ELSE»
   return properties.getProperty("«Name»");
  «ENDIF»
 }
«ENDDEFINE»

«DEFINE propertyWithDefault FOR properties::Property»
 public «Type» get«Name»(String defaultValue) {
  «IF Type == properties::PropertyType::Integer»
   return Integer.parseInt(properties.getProperty("«Name»", defaultValue));
  «ELSE»
   return properties.getProperty("«Name»", defaultValue);
  «ENDIF»
 }
«ENDDEFINE»

While not quite as simple as the template I presented last time to generate sample .properties file, this is still pretty straightforward. Notice how I determine which base class to extend using the value of LoadAs defined in the input model. The last two Xpand templates for Property generate overloaded getProperty() methods, with and without the default value.

Before I can use this template, I need to wire it into my oAW workflow. So I make the following addition to generator.oaw, right after checking the model with the .chk file:
<!--  generate code -->
<component class="org.openarchitectureware.xpand2.Generator">
 <metaModel idRef="mm"/>
 <expand
  value="template::JavaWrapperTemplate::javaClass FOR model" />
 <outlet path="${src-gen}" >
  <postprocessor class="org.openarchitectureware.xpand2.output.JavaBeautifier" />
 </outlet>
</component>


Now when I invoke generator.oaw as an oAW workflow using my sample PropertySet model, I get the following code generated into my project's src-gen directory:

MyProperties.java

package com.scottmcmaster365.propertiessample;

/**
 * Load a properties file based on the approach described here:
 *   http://www.javaworld.com/javaworld/javaqa/2003-08/01-qa-0808-property.html?page=2
 */
public class MyProperties
  extends
   com.scottmcmaster365.properties.ClassLoaderResourceProperties {
 public MyProperties()

 throws java.io.IOException

 {
  super("MyProperties.properties");
 }

 public Integer getMyIntegerProperty() {

  return Integer.parseInt(properties.getProperty("MyIntegerProperty"));

 }

 public String getMyStringProperty() {

  return properties.getProperty("MyStringProperty");

 }

 public Integer getMyIntegerProperty(String defaultValue) {

  return Integer.parseInt(properties.getProperty("MyIntegerProperty",
    defaultValue));

 }

 public String getMyStringProperty(String defaultValue) {

  return properties.getProperty("MyStringProperty", defaultValue);

 }

}

The benefit of having a strongly-typed wrapper can be seen in this unit test example:
@Test
public void testLoadMyProperties() throws IOException
{
 MyProperties properties = new MyProperties();
 assertEquals(2, properties.getMyIntegerProperty());
 assertEquals("hello world", properties.getMyStringProperty());
}


And that's just about it. If I want to use my new properties wrapper in a real project, I will need to pull in all of the files from this project and add some sort of "generate" target to my Ant build which uses the oAW Ant task to invoke generator.oaw.

I should point out that there is a notable overhead involved in spinning up the oAW code generation pipeline, and if you have to do it multiple times per build, your build process can slow down considerably. It is therefore beneficial to consolidate into as few model files (.xmi's) as possible. So a possible enhancement to what I have done here would be to metamodel a class which is a set of PropertySets and then enhance my Xpand templates to generate multiple files, so that I can cram more than one PropertySet into a model.

I hope that after reading this series, you're ready to run out and EMF, oAW, modeling, and code generation. If you do, please share your experiences with me, because I'm always looking out for new techniques and best practices.

Tuesday, March 10, 2009

Generating Strongly-Typed Properties Classes With openArchitectureWare, Part 3: Checks, Workflow, and a Template

Last time I created a sample model using the Properties metamodel that I built in the first post of this series. Now I'm going to put it all together and actually generate something with openArchitectureWare -- not Java code just yet, but rather, a sample .properties file which can be customized and loaded using the Java code that I'll create in the next entry.

Checks
openArchitectureWare provides Check constraints for validating models. For the Properties scheme that I have designed, it makes sense to require models to conform to three constraints:
  1. PropertySet must have a name. I will use this to generate the Java class name.
  2. PropertySet must have a resource name. This will tell me what file to load at runtime.
  3. Properties must have a name. This one is pretty obvious.
Here is how to encode these rules in the Check language (file Checks.chk):
import Properties;

context properties::PropertySet ERROR "PropertySet must have a name" :
Name != null && Name.length > 0;

context properties::PropertySet ERROR "PropertySet must have a resource name" :
ResourceName != null && ResourceName.length > 0;


context properties::Property ERROR "Properties must have a name" : Name != null && Name.length > 0;
The "import" line pulls in the Properties metamodel. Marking these constraints as "ERROR" stops oAW processing with the specified message when such a condition is encountered. The rest of the code is pretty intuitive. Later, when I talk about oAW Workflows, you'll see how this gets wired into the generation process.

Simple Template
It's time to write an oAW template to generate a sample .properties file. This is done with the oAW Xpand language, whose most visible feature is the cute little «chevrons» that mark off the directives. I won't attempt to detail the features of Xpand, but rather point you to the documentation. Suffice to say that it's very easy to get started. Here is a template (PropertiesFileTemplate.xpt) that generates a .properties file from a model:
«IMPORT metamodel»

«DEFINE propertiesFile FOR properties::PropertySet»
«FILE ResourceName + ".SAMPLE"»
«EXPAND property FOREACH Properties-»
«ENDFILE»
«ENDDEFINE»

«DEFINE property FOR properties::Property»
«Name» = «DefaultValue-»
«ENDDEFINE»
When I finally run this in the next section, it will generate a file in .properties format with each property in the model. The file is named based on the value of the ResourceName attribute of the PropertySet. It is suffixed with ".SAMPLE" because presumably it would be customized in a real application. In my next post, I'll use Xpand to achieve the ultimate goal of this project, namely, to generate some Java code to load and wrap these properties. First, though, I need to tie everything together so that the oAW generator can run.

Workflow
"Workflows" are essentially build scripts for openArchitectureWare-based artifacts. They generally end with an extention of ".oaw", and they can be launched directly from Eclipse in an openArchitectureWare project using a "Run As | openArchitectureWare Workflow" context menu item. There are a handful of things that you can do in a workflow, but in my experience, they tend to be pretty boilerplate. Here is one (generator.oaw) that runs Checks and invokes the generator on the PropertiesFileTemplate for our model:
<?xml version="1.0"?>
<workflow>
 <property name="model" value="PropertiesGenerator/src/SamplePropertySet.xmi" />
 <property name="src-gen" value="src-gen" />
 
 <!-- set up EMF for standalone execution -->
 <bean class="org.eclipse.mwe.emf.StandaloneSetup" >
  <platformUri value=".."/>
 </bean>

 <!-- load model and store it in slot 'model' -->
 <component class="org.eclipse.mwe.emf.Reader">
  <uri value="platform:/resource/${model}" />
  <modelSlot value="model" />
 </component>

 <!-- check model -->
 <component class="oaw.check.CheckComponent">
  <metaModel id="mm"
   class="org.eclipse.m2t.type.emf.EmfRegistryMetaModel"/>
  <checkFile value="metamodel::Checks" />
  <emfAllChildrenSlot value="model" />
 </component>

 <!--  generate sample .properties file -->
 <component class="org.openarchitectureware.xpand2.Generator">
  <metaModel idRef="mm"/>
  <expand
   value="template::PropertiesFileTemplate::propertiesFile FOR model" />
  <outlet path="." />
 </component>
</workflow>
In a more serious implementation, the execution of the .oaw file would be integrated in an Ant build using oAW's custom Ant task. For this project, I'll just use the oAW Eclipse runner.

The model to work with is declared as a property at the top of the file. The key sections are the last two -- "check model" and "generate sample .properties file". The only potentially-tricky part is getting the package references right. So here is a snapshot of my project for you to compare to:



Now I actually have a working code generation pipeline set up. To conclude in my next post, I'll write a slightly more sophisticated template to generate Java code, and maybe talk a little bit about code generation philosophy.

Sunday, March 1, 2009

Generating Strongly-Typed Properties Classes With openArchitectureWare, Part 2: Model

Previously I demonstrated how to create an Ecore metamodel describing Java .properties files, with the ultimate goal of using openArchitectureWare to build strongly-typed accessor classes for my properties. Before I start writing oAW templates, though, I want to have a sample model that I can work with which is relatively simple but still exercises most of the breadth of my metamodel.

The oAW sample project contains a model already called "Model.xmi". The "xmi" is "XML Metadata Interchange", an OMG specification concerned with interoperable models, and the default format for models in Ecore/EMF. I will delete the existing Model.xmi file and create a new model called "SamplePropertySet.xmi". One way to do this is to open the Properties.ecore metamodel, right-click on the class that I want at the "root" of my model (namely, PropertySet), and select "Create Dynamic Instance". Choosing a location in my workspace, away I go.

The .xmi file can be edited with the "Sample Reflective Ecore Model Editor", which is essentially the same as the tree-node-properties editor I used on the .ecore file, except now the entities that I edit on my model are instances of the classes and attributes defined in my metamodel. In other words, the "root" of my model is a PropertySet, and to that I can add children which are Property instances. When I select one of these and open the Eclipse Properties grid, the properties are the class attributes defined in the metamodel. If you have been following along and are still a little fuzzy on the relationship between metamodel and model, the following illustration might be helpful:



I'm going to test my openArchitectureWare code generation templates with a couple of properties -- one of type "Integer" and another of type "String". My "completed" model looks like this:



There are a number of other things that I could do with EMF at this point, such as generating a Java object model and "starter" Eclipse plugins for working with my models. But I don't need to do any of that for this project: Starting next time, everything going forward will be in the Land of openArchitectureWare.

Generating Strongly-Typed Properties Classes With openArchitectureWare, Part 1 Addendum: Graphical Ecore Editor

Last time I showed how to build an Ecore metamodel in Eclipse using the Sample Ecore Model Editor. Ed Merks from the EMF team (whose EMF book is out in a new edition, by the way) commented that I could be using the Graphical Ecore Editor, which is a part of the Ecore Tools subproject. He is absolutely correct. (Thanks Ed.) There are a couple of steps to go through to do that, so I'll talk about those here.

First, you need to install the Ecore Tools package, which you can find on the Ganymede Update Site (assuming you're using Eclipse 3.4) under "Models and Model Development". After you do that, you can select File | New | Other in Eclipse, and there will be an "Ecore Tools" folder available. Underneath that, choose "Ecore Diagram". A wizard appears, and you can select your .ecore file as the domain model.

This will add a file to your project which has the extension ".ecorediag", and when you open it up, you'll get a Rose-like boxes-lines-and-properties editor which is far more user-friendly than the sample editor. Here is what my Properties Generator metamodel looks like in diagram form after some slight rearrangement:

Saturday, February 28, 2009

Generating Strongly-Typed Properties Classes With openArchitectureWare, Part 1: Metamodel

I do a lot of model-driven software engineering and code generation. But you might not know that, because I've never written about it at all in my blog. In the next couple of posts, I'll go about changing that.

First, a bit about the application that I'm going to build. I have long believed in having wrapper classes for resources and properties. Visual Studio and .NET now do a lot of this for you automatically. In Java, we're still pretty much on our own. What I would like to do is automatically generate a Java class which exposes a strongly-typed getter for each property in a .properties file.

To accomplish this, I'm going to use one of my favorite model-driven development tools, openArchitectureWare (oAW). oAW is comprised of a pretty extensive toolchain for model transformation, validation, and code generation, and of course, it's open-source. As you will see, it integrates well with the Eclipse Modeling Framework (EMF). If you would like to follow along, install the openArchitecture Eclipse plugin. While you're at it, check out the oAW documentation, particularly the oAW-EMF tutorial (but note that we're going to take a simpler approach that what is described there).

In Eclipse, I start by creating a new "openArchitectureWare Project", which is available courtesy of the oAW plugin. I took the option to "Create a sample", because it creates a number of EMF and oAW files that I need. The sample project looks like this:

Right now, it's possible to right-click the "generator.oaw" file and select "Run As...oAW Workflow" and see some sample Java code get generated into the src-gen folder from Model.xmi.

Rather than stop right now and discuss what each of these files does, I'll proceed through the development of my application and hopefully you'll pick things up intuitively. For starters, I'm not going to use extensions, so I can go ahead and delete Extensions.ext and GeneratorExtensions.ext. This will break the generator.oaw build for now, but I'll put it back together as I proceed.

One of the most important files, and the focus of this post (finally!) is the .ecore file, which I will rename to "Properties.ecore". This is where the metamodel is defined. In this example, I want to create models for properties, so my metamodel needs to describe what a "property" is. In the Java .properties file sense, it is basically a "name", a "type", and a "value". Additionally, I want to bundle individual properties into "property sets" which are stored in .properties files. The property sets themselves need a "name" and, optionally, a Java package to live in. Finally, for illustrative purposes, I'll support loading them either as Java classloader resources or resource bundles.

When I double-click to edit the .ecore file, it opens in the "Sample Ecore Model Editor" by default. This editor is nothing to write home about, and in fact I suspect they call it "Sample" because the idea behind EMF is that tool builders will write (and sell you) their own higher-end Ecore editors, probably in the form of Rational-Rose-like graphical modeling environments. No matter -- this editor will work fine for my purposes here. The upside of the simple Ecore editor is that it is quite intuitive to use -- just right-click nodes to add children (classes, enums, attributes, references to other classes, etc.), and use the Properties view to configure them. My metamodel ends up looking like this:

The only trick I want to point out here is how to configure the "Properties" EReference on the PropertySet EClass. First, I want to have multiple Property instances in a PropertySet, which I achieve by setting the "Upper Bound" on Properties to be "-1". Second, a PropertySet logically "contains" its Properties, so I set the "Containment" property to "true". (If you don't do this last step, you will experience perhaps-confusing difficulties when you actually go to create a Property while modeling in the Eclipse EMF model editor.)

Now I need to use Eclipse to create a sample model based on this metamodel to use for testing while I develop code generation templates. I'll demonstrate that in my next post.

Saturday, February 14, 2009

Getting Integer and Long System Properties in Java

In the learn-something-new-every-day department, in a code review this week for one of the developers on my team, I noticed that he was calling Integer.getInteger(). I had never seen this before. Turns out that this method and its overloads get system properties as Integers, handling the numeric conversion for you and returning null if it fails. There is also a Long.getLong() that does the same thing for (you guessed it) Longs. So all of those times I called System.getProperty() followed by Integer.parseInt() with a catch of NumberFormatException, I was doing WAY too much work. Good to know.

Saturday, January 24, 2009

Programming Culture

I was asked recently how long I thought it should take someone to become "proficient in a new programming language", assuming said individual is already proficient in a different language. In various contexts, people will throw out numbers like six weeks, six months, a year, or more. Obviously the answer depends greatly on what the questioner means by "proficient", as well as what languages are involved. But I think it's the wrong question.

There are several good analogies between spoken languages and programming languages. I've heard it said that a good (spoken) language learner can become conversational new languages very quickly, but the hard part (and the part that takes the most time) is figuring out how to apply that language within its native culture.

I have picked up a (very) little bit of Chinese here and there, and I know that "xiao jie" translates into English as "miss" and is used as a term of address for women. But I'm afraid to use it in practice (at least until I discuss the matter further with one of my Chinese friends), because I have heard that its appropriateness might vary depending on the specific context, as well as from where in China the listener hails, and first and foremost I do not want to offend anyone.

Similarly, a competent software engineer can certainly become comfortable with the syntax and mechanics of using a programming language in a relatively short time. Learning how to apply those concepts in such a way that experienced practitioners of the language feel comfortable reading and working with the code, though, will take more time.

Back in the .com heydey, I was working on desktop applications at Microsoft and looking for an entry into the web world in my spare time. A friend of mine was working at a web startup in DC, and they were doing Perl. I won't name names to protect the innocent, but my friend was and is a True Perl Expert. So while going through the process of learning Perl using (among other things) the Larry Wall book, I would periodically send him my code, usually quite impressed with myself, and ask him how I was doing. The reply always went something like:

"Well, yes, that would work, but it looks too much like C++. A Perl programmer would never do it that way."

Perl, it turns out, has a very distinct culture. Although I had learned the Perl language, I had not learned how to apply it within the context of that culture. I wasn't correctly using the language's idioms, shortcuts, and libraries, and the code I was producing was "turning off" a more experienced Perl hand. Framing the situation in terms of Webster's definition of "culture", I was not respecting the "shared attitudes, values, goals, and practices" of the Perl programmers' community.

I know how my friend felt. Since that time, I've had the opportunity to look at a lot of Java and C# code written by unreformed C/C++ hands, and it is sometimes quite amusing. My personal favorite -- and the deadest giveaway -- is consistent use of:

String s = new String("Hello");

But there are many more subtle signs, ranging from code formatting and indentation to how one writes a simple conditional expression, and, perhaps most importantly in this age of large platforms, not knowing which libraries and APIs are available. Finally, there is the issue of context: Using a language and its libraries to write toy programs to reverse strings is one thing, but using it to put together a scalable web application is an entirely different matter.

Before a programmer can become truly "proficient" in a team environment, (s)he must internalize this knowledge. There is really no substitute for time spent reading code and observing more experienced developers. To borrow again from the world of spoken-language learning, immersion will yield the best results.

Joel Spolsky has an interesting essay on the cultural differences between Unix and Windows. If we wanted to, we could do similar analyses of Java vs. C++ vs. .NET, C# vs. VB, and others. To take one hypothetical, a C# programmer can start writing syntatically (and even stylistically) correct Java code in short order. But to ask that programmer to immediately and independently stand up a web application using Spring, Hibernate, and Struts, with an automated Maven build, unit tests, and one-touch deployment, and make it follow all of the best practices of the Java culture so that others can effectively carry the application forward -- well, that doesn't strike me as reasonable regardless of how experienced that person was in C# and .NET.

In this time where many software developers are reevaluating their careers and techology portfolios (often not by choice), I'm not saying that folks should avoid putting themselves or other developers in positions where they have to adapt to new programming languages and cultures. Quite the contrary -- throughout my career, I have observed that there is much to be gained from a "cross-polination" of ideas between different parts of the software development world. I would suggest, however, that when we do go this route, we need to set realistic expectations for people making these changes.

Sunday, January 18, 2009

.NET FileTime Struct Revisited

On my old blog, I wrote about the problems with the .NET FileTime struct and gave an approach to working around them. Since then, a couple of people have sent me Spaces messages suggesting a way to simplify at least a portion of my solution. Unfortunately, because of all of the recent changes on Spaces, and because I do not check my old space on anything like a regular basis, I have "misplaced" those messages. If anyone having problems with FileTime lands here with improvements to share, please do so and I will follow up ASAP. Thanks!

Sunday, January 11, 2009

Library Thing

Last fall I wrote about using an Amazon Widget to track my library of technical books. Well, I just got turned on to Library Thing, which seems to be a FAR better approach to recording and sharing my interest in books. I won't go through all of Library Thing's features here, but in a nutshell it provides a means to "socially catalog" your books, in the same sense that del.icio.us provides for bookmarking. My first pass at entering my library is here. Perhaps the best part is that after I entered these books, I was able to get book recommendations based on the catalogs of others with similar interests, and I can attest that they are amazingly good. In fact, of the top seven recommendations that it gave me, two were books that I already owned but had forgotten to enter. Finally, with the Library Thing blog widget, you can now see random books from my library in the sidebar of my blog.