Typing Heroism

I'm now two weeks into the post-qwerty period of my life. To celebrate, I subjected my laptop to the switch:



The paper clip method worked very nicely and never made me feel like I might break anything.

I feel a slight desire to proselytize. I'm not going to knock on doors with a book of mormon, but I would like to be scientific about gathering evidence to make my case.

Before the switch, I was pretty fast on qwerty:

typingtest.com - default settings, Zebras
Accuracy: 94%
Gross Speed: 58 WPM

Irrational side of...:
Accuracy: 96%
Gross Speed: 65 WPM

Then I changed keys, and my last blog post was 407 words in about an hour:

week 0:
6.8 wpm

Only 10 times slower... but I would forget the end of my sentences by the time I got there.

week 2
Accuracy: 100%
Gross Speed: 27 WPM

So now I'm up to half my original speed. This doesn't exactly make the case, but I'm getting there.

I also made a quick Flash version of the typing lesson I've been working through, and would like to make a real game out of it. It's at TypingHero.com.

Switching to Dvorak, or, the slowest typing ever

4:45 PM

It all started when I spilled some Guinness in my Apple keyboard a couple months ago. I tried to clean it up quickly, but after it dried, a couple keys didn't work. I ran it through the dishwasher, I heard that can work on NPR, but although it came out clean as new, the bad keys were still on vacation.

After moving around the corner in my building I pulled the keyboard out again and decided to take another shot at repairs. I discovered that the keys pop out really easily, there's a simple center post and two snappy clips. It was so much fun to pop out the keys that even though only Caps lock, Tab, and tilde are broken, I ended up with this:



So what would you do in this situation? Once all the keys are in a pile, it's a perfect time to make some changes instead of putting them back in those lame places they started in. The name for those lame places is the QWERTY keybard layout. (Ironic how long it took me to type QWERTY...)

wikipedia:

[O]nce an operator had learned to type at speed, the bars attached to letters that lay close together on the keyboard became entangled with one another, forcing the typist to manually unstick the typebars, and also frequently blotting the document. A business associate of Sholes, James Densmore, suggested splitting up keys for letters commonly used together to speed up typing by preventing common pairs of typebars from striking the platen at the same time and sticking together. The effect this rearrangement of letters had on maximum typing speed is a disputed issue. Some sources assert that the QWERTY layout was designed to slow down typing speed to further reduce jamming.

So while QWERTY wasnt designed to be the slowest layout, it does try to help the typewriter deal with fast typers, and as a result, it is a slow and antiquated system that has become a anachronistic standard at best. Why do people still use it? Plainly, it is already on the keyboard, and people want to do what's familiar. Maybe they don't want to question common wisdom and norms. And sure, once you learn it, you can be pretty fast.

Programmers do like to question norms of course. And sometimes do wildly impractical projects, like using the fastest layout, Dvorak:



Once I had the keys in place, it took 10 seconds to switch keyboard layouts in OS X. Windows makes it easy too... you can probably find the tempting Dvorak option on your machine right now. Don't you want to click it, and ditch that lame layout from the 1800's? Look at the metrics, and imagine yourself Instantly typing 40% faster!

So that last claim is a bold lie. I've now spent almost an hour typing this, which is completely infuriating, so I'm going to get dinner. Check back in a week or two to see how long I can make myself do this.

(did I mention I've tried twice before?)

5:45 PM

IntelliJ IDEA, week 1

I switched from eclipse to IDEA about a week ago, mostly due to prompting from jkuhnert. I feel like making a Borat reference: "Verry niiiice."

If I start getting preachy about IDEA, I'll have to back it up with real reasons why it is good, so here are my favorites. Mind you, I'm just learning things as I stumble upon them, and I'm no expert yet - but maybe these are some things you can look forward to in that crucial week 1... that time when you throw away most new tools you try.

First, the obvious things - it's streamlined for common developer tasks, doesn't try to compile your whole project all the time, and comes with most of the stuff you need, instead of giving you the pleasure of choosing the right plugins.

Keyboard shortcuts. I've had "learn more keyboard shortcuts" on my list for a while, because when it comes to raw productivity, you want to be able to do things in the computer at a similar speed to what happens in your brain. The mouse is fine up to a certain speed, but it hits a wall. Learning keyboard shortcuts, on the other hand, is an investment like making yourself touch-type, you can keep getting faster. IDEA really makes it possible to do away with the mouse. And, they are great about putting keymap hints on the UI, as well as providing a nice cheatsheet

The changelist feature. We are always somewhere between an svn update and an svn commit, and every change we make ends up in a list when commit time rolls around. Most of the time, I don't have a nice clean changeset to commit, because I've made some other changes that belong with each other, or I don't want to promote to version control. IDEA shows you that list of changes all the time, instead of at commit time, and lets you push those changes into different changelists as you go. It's really consistent, too, with the diff viewer open you can flip back and forth through your changes and move them to changelists there too. Howard Lewis Ship blogged about this one. Although duck_typer has a good point - you can just have multiple working copies of the project, and use one for each set of changes. That way if you touch the same file in two changesets, you're still ok.

The UI Layout. I actually have problems reading the text in eclipse, which makes me feel old. You can force some text size changes, but things like the navigator panel and tab titles were always annoying. For some reason, the swing UI in IDEA has nicer fonts, a more comfortable layout, and seems more responsive. (yeah, most things are more responsive than eclipse)

Anyway, those probably won't be my favorite features in a few months, but they're making me happy for now.

Android!

For those who don't know (somehow), Google has acquired and released their platform for "iPhone killer" phones. It's called android. Seems like this is a good chance to practice building phone apps, especially since it doesn't involve learning Objective C.

I think it could be a fun way to spend the day-after-Thanksgiving. Also, please enjoy my requisite self-referential pic of the new phone, like I did for the iPhone.

IMAP, how I missed you!

Dear GMail,

I'm leaving you because you didn't fulfill me the way a real mail service can. It was great that you were free, but free POP3 doesn't fill that spot in my heart the way a paid IMAP can. Back in college, I promised to be forever faithful to IMAP, but you were sexy, and I needed a mail agent so badly, it was easy to fall for you.

This morning, I knew I made the right choice. I flipped through my mail on my phone on the way to work, and deleted some things, and when I got to my computer at my desk, the mail was deleted. My drafts and outbox are on the server too, so I can start an email on my phone and finish it later.

I know, you expect me to use your interface. I'm supposed to use my browser when we hook up. Even with beer goggles, I'm sorry to say, you weren't hot, so I was using native mail apps whenever we talked. I can't even tell you how frustrating it could be when my three mailboxes that you touched your POP3 would get their own separate copies of mail. I think my other friends were upset about our relationship, too, because their messages would get lost in one of the inboxes that I couldn't keep up with.

I know, you wish I could see through the protocol to who you really are... but I can't. I'm so sorry, and by the way, please forward my mail to my new IMAP account. That way I don't have to tell anyone we broke up.
-Me

Handy tip for layouts in Freemarker

If you're writing a Java webapp and using Freemarker as your templating engine, hooray! Skip to the next section. Maybe this tip will come in handy.

If you're not using Freemarker, you should really consider it. JSP's are ridiculous - why are you compiling all your markup into Java code? Do you like it when you make a small typo in your JSP page, and get an error from the JSP engine referencing the Java code it created to render your page?? Gross, let's go read through the generated code to infer what the typo was! Do you like how the JSP pre-compilation step is annoying to set up, but if you don't, you have the dreaded first request to worry about? JSP's might be a little faster than interpreting markup on each request, but with smart caching, and the ease of adding more web server boxes, that's a dumb reason. And wow, let's give people the ability to write arbitrary controller code in the view, and then trust them to do things the right way instead.

Oh, and if you like Velocity, that's cool I guess. Freemarker is nice because it's a real stickler for correctness - a null value isn't just rendered as a blank string, because it's not (insert scary life-support app example). The model is the model, you can render it, and that's all. Just what you should be doing.

The layout tip

If you've played with (or gotten paid to work in) Ruby on Rails, you're familiar with the "layout" concept. It's just a simple template for a page, and instead of using "includes" where the template has to know what's going to be inserted, it just "yields" to the page content. It's the same philosophy as Struts Tiles - the dependencies between your markup files should go from the bottom up - the content needs a place to be inserted, and the template doesn't care what it is.

This layout method gives you a similarly easy and readable result in Freemarker.

First, let's create a page, with some wishful thinking for how we think the layouts should work:

<@layout.basic>

<a href="/customer/accounts">Browse accounts</a>

</@layout.basic>

That's saying, I'd like to use a layout named "basic" and in the place that layout puts the page content, I'd like to have my link to Browse Accounts show up there. The page is as short as can be, and readable.

Now, let's make it work. It seems I'm calling a macro "basic" and I expect it to be in the "layout" namespace. I haven't used an import macro on my page, so the code for the layouts will be auto_imported into my pages, using the "auto_import" setting in Freemarker's Configuration object. I set it to "/layouts/layouts.ftl as layout" so that the template /layouts/layouts.ftl gets imported to all my pages with the layout namespace. (It has to be that way because Freemarker can only associate one template with each namespace.) If you're using Spring to configure Freemarker, you'd create a props element containing an "auto_import" prop and pass it to the freemarkerSettings property of the FreeMarkerConfigurer bean.

Now I create the layouts.ftl template. This could contain all my layouts, but it would look strange for several of them to live in one file, so instead my layouts.ftl looks like this:

<#include "admin.ftl" />
<#include "basic.ftl" />

Clearly this is where you add additional layouts as you need them.

Finally, let's look at the contents of basic.ftl:

<#macro basic>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
  <head>
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
  </head>
  <body>
    <div id="header">
      <p>
        Cows make moo sounds
      </p>
    </div>
    <hr/>
    <div id="content">
      <#nested>
    </div>
  </body>
</html>
</#macro>

This creates a macro named "basic" and uses the nested macro to include whatever appeared in the body of our caller's <@layout.basic> tag.

Neato!

Work smarter, not harder... and faster!

Maybe it's obvious. When you try to do things really fast, it can actually be slower.

I always thought "A stitch in time saves nine" meant that you should stitch as fast as possible. I think I just assumed that, which I need to be careful of, because when you ASSUME, you make an ASS out of U and ME.

It's certainly true with programming. When it comes down to it, if you look at a finished application, it's conceivable you could just type the code for it in an afternoon, especially if you have good tools that let you express your intent quickly. Of course, you don't know exactly what the code will look like until you start to create it, but let's ignore that for a sec.

So... taking my time and thinking for a second to save time. I was too frantic for the last couple years, and wrote a lot of Log object instantiations.

public static final Log log = LogFactory.getLog(NiceObject.class);

I wrote that a lot, and got a Das Keyboard so I could type it even faster. I knew that Eclipse could do this for me, but there's that nagging startup cost, where the first time I set it up, it would take longer, and I might go looking around for more of the same (or blog about it), making it much longer than typing it out.

Well, lo and behold, I am happy I took the 30 seconds to:

Correction!

Well, I learned that what I said yesterday isn't quite true. It IS true that the geocoder.us site does a bad job geocoding on this street. You can see their map here:
http://geocoder.us/demo.cgi?address=4050+BEN+LOMOND+DR+PALO+ALTO%2C+CA+94306

BUT, if you zoom in, you see the shape of the street is right, they just have the pin in the wrong place. The maps come from the TIGER/Line data as well as the geocoding. The problem is that the Perl module, Geo::Coder::US, doesn't use one of the data files in the TIGER set - the RT2 "type 2" data. This file is Complete Chain Shape Coordinates which means it has midpoints for some street segments. I found the two midpoints for the street I showed yesterday, and put some pushpins at those spots - and behold, the curvy street problem would go away!

Now, too bad I can't find any libraries (in any language!) that do the same thing as the Perl library, but use the RT2 data...

Why TIGER/Line can be a bad source of geolocations

I came up with this neat map in Google Earth, showing the difference between Google's coordinates for some addresses in Palo Alto, CA, and a line connecting those to the TIGER/Line-backed geocoder.us coordinate, colored blue-to-red based on how far off it is. The TIGER/Line data only has endpoints for a range of addresses, and the geocoder.us Perl module tries to do a linear interpolation to get addresses in the range. Obviously this doesn't work well for non-linear streets like this one...

FeedBurner, finally!

I have sort of known in the back of my head that I ought to have my feed managed better. It would be nice to see who is subscribed, both because I don't really want to blog to hear myself talk, and also because you get an incentive to write more interesting and useful stuff if there's an audience.

So I hooked up feedburner, and was pleasantly surprised by how useful it is. Not only do I get my basic readership metrics, it also manages the trick of getting existing subscribers under the metrics (by telling blogger to redirect requests for the feed URL) and lets me fold in daily bookmark digests and photos.

Hopefully my 2 readers (?) will also enjoy my random API and documentation bookmarks (?) and at the same time, my attempts to do film photography (?).

Now I am wondering how to advertise my Twitter and Skype and Facebook and so on. I guess it's good to make them widgets on the right side of the blog, and just assume that one URL will give away my complete identity. Maybe hCard is good for this? I do feel like my "Card", whether it's vCard or whatever, should have my contact info including many URL's, and should be easy to broadcast my card to *everyone*, not just online friends. Then the offline people like grandma get my new phone number, and online people get the skype ID, and all is well.

That would be another reason to try out one of the email->snail mail gateways - I can subscribe my grandmother to my vCard feed and she'll get it via postal mail... slick. Except then she'll call me wondering why I sent her a letter, just because I added a web 2.0 service to my card... maybe I should think about that some more.

liveblogging from barcampdc

I saw the technorati page over justin's shoulder so I should probably contribute. The conference is cool, it's a little presentation-heavy rather than interactive, I guess that's just because it's hard to get people comfortable with other participants when there are 9 sessions in 8 hours.

I was thinking about how attendees at this sort of conference are into scripting, web 2.0, design, user experience, "lightweight", fun stuff. There isn't much community forming around enterprise, Java stuff anymore. The NoVA JUG has been trying to recover from near-death, as far as I can tell from the mailing list. Maybe dynamic, smart people just want to spend their free time on the fun stuff that's hard to get paid for.

Building codes

When I was at my last government contract, we were developing a Single Sign-on solution that dragged on forever due to Oracle bugs, misunderstanding of the requirements, politics, and the other usual suspects. In the meantime, across the street and plainly visible from my seat, a building went up, a nice 8-story office building with a parking garage. They literally broke ground, dug the foundation, erected cranes, put up the building, added HVAC, walls, windows, and it was basically done, while we still hadn't implemented a working single sign-on solution across all the applications that were supposed to have it. We were just typing, and these guys had brought in truckloads of materials and assembled this giant structure.

This has always struck a nerve for me, especially because I think erecting a building is the single best analogy for software development. I've heard others, like from my crazy first boss who always said that if we were missing a certain feature, it was like a car with three doors. But a car isn't a good analogy for development - they are mass-produced, never modified, consumer products with a pretty small set of complexity - combustion, electrical, mechanical, suspension, seat cushions.

Buildings, on the other hand, share a TON of interesting features with software. They are large projects, similar to each other in form but each unique in their implementation. There are a number of skilled workers, with different areas of expertise that must participate. There is a complex schedule that must account for the dependencies between pouring the foundation and inspecting the concrete and so on.

Pre-fab is another nice part of the analogy. I don't know much about it, but I think there are a lot of standards in buildings that make it possible to get lots of your materials in a partially constructed state. Door frames are a similar size, and you can just buy the door already hinged into a frame, nice and square and level. I see trucks with rafters for houses, already assembled and ready to "raise the roof". Same goes in software, where standards like those from the JCP let us plug in pre-assembled components rather than building them from scratch.

Looking further, there is an agency problem. The people who will ultimately inhabit and use the building are rarely stakeholders during the construction. The builders do the construction, but don't have to live in the structure or deal with the problems that the maintenance company uncovers. In software, end users are similarly more like tenants than they are landlords.

Yet, in real estate, there is much less sense of risk or mystery about the quality of the work, whether the building will stay standing for 5 years, whether it's safe and structurally sound, if it was built by qualified craftsmen. When you see a new building go up, you're pretty confident that it will get finished and be a successful project, as long as the funding stays intact and the contractors aren't too disreputable. But in software, while companies focus a lot on a structured development process and testing, they don't have a grasp on what the code ends up looking like, whether the system is structurally sound and safe from fires (= production emergencies of course).

I want to explore why this is. I think I'll look at it over a series of posts and maybe develop this into an article, because I think there's something very interesting that software development could gain from civil engineering practices.

To start answering questions, I'm thinking of how buildings work. First, there are house inspectors. I can't just look around a house they might buy, and figure out if things are structurally sound, yet I expect I could hire someone with this skill to give an honest, impartial appraisal. I'm sure this is the case for new, commercial construction as well. Yet I've never heard of a third-party auditing/appraisal service for software.

Next, the inspector has a set of building codes, enforced by local law, that guide him to check many important features, and provide an objective baseline of criteria that the building should satisfy. The most popular set of codes is the International Building Code - I think I'll poke around in there and see how they manage to capture the complexity of what they assess. Just as with software, I'm sure there are a lot of grey areas and places where there's some discretion the architect can take, yet there's a right and wrong way to assemble the house. We do have some such guidelines in software, but I think this could be improved based on how the civil engineers do it.

Finally, there's the qualifications of the craftsmen. I like posts like this from Ken Auer on apprenticeship in software that talk about how software engineers learn their craft. You really don't learn it in school, or from a book. (Maybe Cisco or MS engineers can do book learning and take a certification test, but I don't know any good software engineers who do this.) Sure, programming is not all science, there is art to it as well. But there's also art in architecture, and maybe even in quality welding, for all I know, and those guys don't just learn their craft by copy/pasting some code and trial-and-error.

So, if I keep this line of thinking running through my head, I think I'll write about:
- How building code works, how it is applied to real world buildings, and whether the software analogy works
- How building inspectors can apply the building code, and whether there are such people for software
- The way craftsmen like electricians, welders, as well as artists like architects and designers, learn their craft and conform to the code

How do buildings go up faster and with less risk than software??

I'm interested in comments!

Eclipse Europa and Modeling

I needed to whip up some class and sequence diagrams today, but had some time to spare first. So I thought I'd reevaluate how easily I could get a modeling tool working that's code-aware.

First, I saw that sometime over the last week, Eclipse released Europa. I'm still a little confused how that relates to Eclipse 3.3 - I think 3.3 is the new version, which is released, however they encourage people to get Europa instead, which is a bundled release for 3.3 along the lines of Callisto.

That was only the beginning of how fragmented the development efforts around Eclipse seem to be. I started poking around in the ever-disappointing GEF/EMF/MDT/UML tooling. It seems like they've abandoned/renamed/re-organized these projects more frequently than they've gotten meaningful releases out the door, and although there's a project called UML2 Tools that's supposed to provide class and sequence diagramming, there's zero documentation on how one would use it in Eclipse. I get the impression that despite the large number of eclipse tools and projects that make up the modeling stack, none of them is meant to be used by "users", they're all for "toolsmiths" (http://wiki.eclipse.org/GMF_Tutorial)

And this toolstack always caused the dreaded Eclipse plugin spaghetti mess for me in the past, so I'm still not really impressed with it. Maybe I'm completely misusing it, but I've put in a good faith effort to read some docs and learn what to do, and it can't be that hard. So, time to move up the stack to the vendors who build on top of the eclipse tooling.

One tool I've used before, Omondo, seems to have gone through some nasty personnel issues, because now there's a competing plugin, eUML2 that apparently came from Omondo's CTO and a mass defection of engineers. So I gave eUML2 a try.

It's nice to see it builds on the Europa/Eclipse toolstack concisely. It helped me through the reverse engineering, although I got XDoclet-style annotations in my source files rather than Java5 annotations. It's terribly slow even on my MacBook Pro, but it does eventually pull in the classes I want, has a very nice presentation (including full package name, yellow warning icons, etc), and manages to discover some of the connections between the classes. The sequence diagram editor had some problems with the view falling a step behind the true model state, which I had to work around. Also, it didn't leave enough margin for some text, although it did so consistently. Overall, it was usable, and I got my diagrams done... I think it would drive me nuts for a real design phase though.

Also used the new Spring IDE 2.0 which was just released - I didn't hit any new features but I do enjoy the bean dependency graph, that makes for nice quick documentation too.

blogging from the iPhone

just a quick post to announce (brag) that the iPhone is nice

Apple Proselytism

I try not to be a total fanboy when I can help it. It's really easy to do this with Apple computers - once you've switched, you are consumed with a desire to open they eyes of your friends who are still in the PC blindness stage.

I want to share my favorite description of the virtues of Macs, something short enough and indicative enough to give it all away.

"When you're using a PC, and an error occurs, sometimes a dialog tells you to contact your systems administrator. This is because they expect you to HAVE A SYSTEMS ADMINISTRATOR!!! You're at home, you should never be told this. Macs don't expect that you have one."

Choices, choices

I've been thinking about the choices programmers have to make eventually. I guess this assumes that you're somewhat talented, ambitious, and thoughtful, because I'm sure a lot of coders can just sit happily at a desk and not ask themselves this. Also, it's just one way to divvy up the options, there are probably better ways.

I don't feel like writing pros and cons, so I'll just give cons and you can use your imagination.

Product company or Consulting?

Product company

You will have to work on the same thing for a long time. At first, when the product is under development, there are a lot of cool things to try and big decisions to make. But then it gets boring, you have production support to do, and some of those decisions turn out to have been wrong and can't be easily corrected. The bottom line for the business at this point depends more on sales of the existing products, so the emphasis on cool technologies is diminished.

Because you're supporting existing code, it's hard to find a chance to learn new stuff. Maybe you can deceive your manager into using the stuff you want to learn about, under the guise that the system needs to be re-written and this technology will save a lot of money or fix a lot of problems.

Consulting

You'll have to work with "non-ideal" client's employees. If the client hired a lot of really smart developers, they wouldn't need the consultants. Or maybe the really smart developers know to stay away from working there, so they need to pay the people who are willing to work anywhere as long as it's only for a year or less at a time. This makes it harder to be challenged by your peers to improve - if you're spending most of your time working with people who are not smart, you tend to go down to their level instead.

You're pretty much guaranteed to work on junky code. Again, if they had to hire consultants, they need your help to solve the challenging problems. It's probably not an interesting new business idea being implemented, more likely it's a self-created problem with their monster spaghetti codebase and vendor-provided toolset.

You have to give up some of your ideals about the quality tools/platforms you prefer. If your company gets a .NET contract, and your client insists you use Windows, then guess what? You're going to use a closed platform with second-rate versions of your favorite tools (if there is an 'N*' of your tool) and you're not going to use your Mac even though it's your more productive environment without a pile of bugs. And if the client is using the Oracle J2EE app server, then you're going to be decompiling it. This is a tough one, since a good developer stays agnostic about the tools and platform - but sometimes you have that ideal for a reason.

Many people don't want you there. Sometimes this includes your client-side supervisor, who was told to bring in consultants by their boss but thinks they could handle it fine on their own. If you produce a report that supports their pre-conceived ideas, then maybe they'll be excited about having you there, but more likely you'll be creating work for them and telling them things they don't want to hear. And of course the client's employees are mistrustful and think you'll make them look bad. And they probably value job security more than you do.

Keep coding or move to management?

Keep coding

You may get stuck as a "line programmer". This is like modern factory work. Your job is outsourcable or offshorable because there's a heavy reliance on "requirements" as an excuse to do waterfall, so the programmer doesn't have an interactive role and anyone can do it. This happens because many people think programming is typing and will never understand the art of it.

There are few places where you can move up the pay grades as a programmer. The Java Posse did a listener feedback question on this once, and said if your company doesn't promote really good developers as highly as managers, then you should move to another company. Problem is, Sun, Apple, and Google are mostly doing this work in the Bay area so that leaves a lot of people out. I'm still hoping there are more places like this.

Also, you may not get a chance to be a leader or promote your ideas. Developers don't have much decision-making power.

Management

You will probably become a bullshitter. Since you are disconnected from coding, you won't know exactly how the software behaves in a corner case, or how difficult a feature would be. So you make stuff up and make educated guesses that become random guesses over time.

You'll have to give up the dream of being the next Gavin King or Craig McClanahan or other coder hero. Instead, you will be driven to improve non-programming skills like corporate finance and negotiating.

The business environment is cutthroat, which doesn't match well with hacker ethics.

Worst, you won't do what you love, you will write proposals and performance reviews and project plans and push tickets and schedule releases instead.

Use left-side or right-side of your brain?

Right-brained - the Artist

It's hard to find a paying gig where you can invent the next great thing (or be a part of it), so you have to do one-off contracts and either manage a lot of business stuff yourself, or be a starving artist type. Until you get your job at Apple, where I think most of the programmers are the artist type.

It's not clear how to pursue an artist/hacker ethic. You can work on your brilliant idea at home, but where are the interviews for this sort of work? Perhaps there are not many openings and a lot of applicants, like aspiring to get in the NBA.

You have to gain true respect of your peers, for example, become open source committer on a well-known project. The artist rejects unearned corporate rank and title, especially when bestowed by someone who doesn't recognize quality.

Left-brained - the Analyst

You won't "save the world" - you will create predictable products that mimic others.

There may be pressure to create average-quality products, since there is no business need to exceed customer expectations, especially if it makes development more expensive.

JavaOne wrap-up

It was a long travel day yesterday to get back to DC. Here's the rest of the

Things I learned:

Glassfish 3 has a really neat-sounding architecture. The app server core is like a 20K jar and launches in half a second. It then has a maven-based dependency system that causes it to load OSGi modules that add things like JEE container, servlet container, HTTP server, PHP support, JRuby on Rails (directly from the app directory - no WAR building), Phobos, etc. So you can add a number of containers to your one running Glassfish instance and then deploy apps into it. (I think Cargo is another way to do this.) It also means it's a pretty extensible app server. There's some nice classloader magic, with a public and private classloader for each module, to make all of this possible and avoid typical J2EE classloading nightmares. Also, their build scripts create configuration files out of annotations, as opposed to introspecting them at runtime.

Rod Johnson's talk was packed. I wonder if Sun intentionally gave him a smaller room than his popularity demands because they're not sure they like him undermining their EE specs. He is more like a corporate CEO than a coder - he's very driven and seems to want market share more than hacker credentials. Interface 21 got $10M in funding at the beginning of the week so maybe this gave him big britches. Spring is adding annotation support - he did mention Guice in passing. He's also certain that OSGi is the next big thing, Interface21 is working closely with them. There's a Spring-OSGi module with the implementation. Sounds like an important thing to go learn!

Tor Norbye of the Java Posse kicked some butt! Not only did he give a talk on his Ruby contributions to NetBeans, he also was up on stage in front of the entire show at the last general session ("Toy Show") and demonstrated his speed in Netbeans and the number of keyboard shortcuts he must know. I wonder, it looked like they were all presenting on Solaris boxes, and he codes on a MacBook Pro, so wouldn't the modifier keys on the keyboard be hard to find? Maybe he practiced... I did download NetBeans 6 and opened my RoR project in it - there are some nice things. I used the local history, completion, and navigation features, and they worked pretty well. The real issue with IDE support for Ruby is the debugger - in C, there are a couple debuggers, but there's no equivalent of JPDA that lets you remote-debug a running Ruby interpreter. For JRuby, someone is working on a debugger, but the Sun guys are pushing for contribution to the Debug Commons project which will help all the IDE's have a consistent way to invoke debug operations. They're looking for help - Gabe?

Hibernate Search looks like a nice union of Hibernate meta-data with the Apache Lucene search engine. You really can't do good search with Hibernate criteria - the LIKE operator in SQL is way too slow. I ran into this with a search application at my gov't client that tried to create a massive SQL query for every user search, which was a really bad approach. Hibernate search allows you to add some annotations to your JPA class to indicate what fields should be full-text searchable, then it handles building the Lucene index, and even keeps the index up-to-date if you change the entity. There's a JMS-based solution for clustered front-end machines to share a master Lucene index. And, extra nice, the Lucene search is wrapped in a standard Hibernate Query object so your DAO code doesn't have to know it's not using SQL for some queries. Nice! I did notice Emmanuel called something a "workaround" - this made me wonder about some of the Hibernate code and whether they've allowed that mentality to get into the codebase - but this looks like a really nice way to support user search in Java.

Selenium is a great way to automate functional tests of a web app. Unlike HTTPUnit and other Request-Response level testing tools, Selenium runs a true client within the browser. This allows it to work with Ajax apps, for instance. There's a Firefox plugin that records your interactions with the app and turns it into a JUnit test case - also very handy. Selenium gets its name from the element, because it's used as a remedy for Mercury poisoning. Man, Mercury has been a terrible vendor in both of my last two clients.

Chris Richardson (author of POJOs in Action) had a nice presentation on testing, most of which I already knew. But he had a nice way of expressing why good guys don't win when it comes to unit testing and TDD - the "paradox of excellence", which is where you are rewarded for fixing the nasty bug right before production (or in production, shudder), while you are not rewarded if things quietly work and you leave every day at 5.

Feeds and REST are the community response to the enterprising of web services with XML-RPC and SOAP. There was a great graphic on one of the slides from the Rails guy, David Heinemeier Hansson:


So true.. I went to a talk on WS-Policy and pretty much wanted to shoot myself after the barrage of consultant-speak and enterprise BS. So - feeds - Rome is the Java library of choice because it handles the mess of RSS and Atom formats (RSS never had a spec!), but it's JDOM based. Watch for it to merge with the Atom-only but STAX-based feed library (A-something...)

There's even an alternative to WSDL for the lightweight REST crowd - WADL. However, there are a lot of security issues lingering in this whole mash-up environment, basically, you should only mashup to sites you really trust like Google. Assuming the stuff about Chinese censorship hasn't put you on a crusade...

Domain-specific Languages are really great for solving some problems, and I started to think of them as an alternate solution when you consider a rule engine. Rules are good for implementing a lot of business requirements that would involve big chunks of repetitive, procedural Java code. But, a lot of times, those requirements aren't globally true facts about the business, they're very contextual and too simple, and there are too many of them, and a rule engine isn't a good fit. Perhaps you want to code the solution, just not in Java. You want a language that expresses the business concepts - aha! Now, the talk from another Interface21 guy went into implementation details for writing your DSL in JRuby, where it is easy, then easily integrating this with your enterpris-y Java app. Managers don't have to freak out since it's just another JAR dependency, but it's a much better way to create and use a DSL than any solution in Java (I don't actually know of any). I think this is a great addition to the programmer tool-belt. Also, this guy actually demanded a raise from Rod during the presentation, I guess the $10M has gotten to their heads... not sure I want to work there anymore.

My overall observations from the conference:

• JRuby is going to make a major impact on Java development, both in the web and enterprise (I don't know about micro... but I think there was a presentation on that). Java makes some things hard to do, and getting new major releases into common adoption takes a couple years (still not on Java 5 at a lot of big companies!). So waiting 4 years for closures isn't going to compete with a single jar you can drop into your app to start including Ruby classes. I'm getting The Ruby Way, a recommendation from the JRuby DSL presenter.


• It's important to work with smart people. Since they share a lot of common motivations, if there are a lot of smart people at a company, all your other criteria are probably met as well. I'm starting to think there are two distinct camps of brilliance, though - the artist type and the businessman type. Consultants seem to be the latter, and innovative ideas come from the former. Maybe you can distinguish them based on who has a Mac.


• All of the good software companies are in the Bay area. I hope this isn't entirely true.


• I want to go to JavaOne next year.

Lessons from JavaOne

Here are some things I have learned so far:

• If you're a Java geek, then being next to Rod Johnson, meeting Howard Lewis Ship, seeing the Java Posse interview the Cenqua team, are probably like most normal people seeing Julianne Moore at the grocery store. (I also ran into an architect from my last client and a manager from the client before that!)
  


• With static analysis (ie. FindBugs), you find errors that are not caught by the compiler. This is a lot like a document where you make a typo that is a correctly spelled word, so it doesn't get a red underline. It's nice to have a grammar checker for those situations. That's one way to think of static analysis. It was funny how Bill Pugh ripped on Josh Bloch's JVM code for having obvious typos. Apparently there isn't much underway to do this sort of analysis on scripting languages, but he thinks it's even more important there since the syntax is more relaxed.
  


• WS-Policy is completely un-fun and sounds like consultant-y BS


• Romain Guy is a true programmer rockstar because he has to beg for breakfast donations at the beginning of his talk due to being up till 4:30 am, trashed.


• Non-RESTful web services are tunneling through HTTP. I hadn't thought of it that way before. RESTful services, on the other hand, take advantage of all the infrastructure set up around HTTP since things like caches work at the protocol level.


• The proper way to distribute business rules to your running Drools (or other engine) instances is by publishing them to a content management system. JBoss Rules will soon be integrated with Apache Jackrabbit for this purpose.


• Closures are needed in Java, anonymous inner classes don't cut it. Many APIs and language constructs would be less awkward and verbose if we had them. Neal Gafter is very opinionated.


• A workflow engine can be useful for business intelligence if the history of workflows is data mined.

At JavaOne!

I'm in SF at JavaOne this week. Have to head off to an Open Source SOA talk, but here's a quick post...

Maybe it's because I selected RESTful web services, Grails/Trails/Sails, and Closures, but I keep seeing people trying to catch up with RoR. They admit Java makes the easy problems hard (Neal Grafter's words) and want to develop more APIs and language changes that would give Java some of the same capabilities. I'd really like to hear more about JRuby and see if we can leave Java alone and admit that adding APIs isn't the same as using a different language.

Went to see the Java Posse live. They had giant foam hats, and talked about the keynotes which I missed. (I flew in yesterday morning, which meant being up 22 hours, and I couldn't stay up to get beers after the BOF with them :( ) It was great to see some of the industry rockstars yesterday. Hope today is as interesting.

XML: configuration or code?

Seems like there's a backlash against XML going on. Coding in XML is a code smell, and everyone is rethinking the overuse of XML in Spring configuration, for instance. Google Guice is the annotation-based IoC container, and now I see Spring 2.1 is doing the same.

I think there are two ways to look at the XML problem. One way is from the configuration management perspective - if you have a long process to build and deploy your software, you know how nice it can be to have a change be to configuration and not code. And maybe you'd deploy your software to many customers who configure it differently, if you're writing Tomcat for instance. There are surely some situations where the configuration of your app isn't simply key/value pairs, and XML is a nice way to structure your config files.

But of course this is abused, and Spring contexts are a good example. You don't expect that the customer is going to change the files, they're very long, and it's likely that a change in the app is going to require compiled changes as well as the XML changes. And they're really part of the code, wiring up objects in a specific way, such that changes to the context are likely to result in the app not working.

Annotations are the answer to this, it seems, since they put the configuration right there in the code. It's a more concise way to express the intent, the compiler catches your typos, and should make for better tool support. But, I have one problem with this - thinking about the Dozer bean mapping framework as an example. Here, we have two JavaBeans and the XML configuration describes how to map properties from one to the other. It seems anachronistic to give either Javabean knowledge of the other.

Maybe the better way to think about it is that XML should hold data. Facts, unchanging in time, with no logic, no tie to the classes themselves and certainly no relation to a certain runtime context. Sometimes the data is obvious, like the input to a DBUnit test. It could be some wire-protocol data, like a web service request used in SoapUI. Maybe sometimes it's configuration, like the servlet.xml file. Maybe it's even the dozer mapping file, describing how two JavaBeans' properties are related. Anything else, and yeah, XML is not a good place for it.

Use a $%*& SocketAppender

Pretty much everyone uses Log4J, hopefully through a facade layer like Commons-logging or JDK1.4 Logging. However, almost no one I know uses the SocketAppender.

Everyone has to establish a balance between using the debugger and writing log() statements. It's annoying to leave behind logging that was relevant only in the context of fixing a bug, but also bad to have no data on production issues because your debugger was your only source of diagnostics while you coded.

So, assuming you have some log statements, you ought to be able to watch them on one screen (you have several, right?) while you navigate through your app on the other. And, when there's a stack trace, the log entry should still only occupy one line on your log viewer, and you can summon a nicely formatted stack trace for that line when you like.

Too many developers are ignoring logs (how many times do I have to answer your question with "did you check the log?") and about as many are eye-parsing through log files. Use Chainsaw or Lumbermill already!

It's pretty easy to do:
<appender name="lumbermill" class="org.apache.log4j.net.SocketAppender">
<param name="RemoteHost" value="localhost">
<param name="Port" value="4445">
</appender>

Also, why is no one maintaining the log viewers? I tried to take over Lumbermill, which hasn't been active for 1.5 years but I thought I saw Robert Bushman claims he's still working on it.

Workarounds

This morning we are fighting with a bug in the Microsoft MSI installer, that deletes part of the bundled JRE from our app after an upgrade, as described Here. There is a workaround listed on the Microsoft knowledgebase site for this bug.

A workaround is a "temporary" solution to a bug, that is created in lieu of fixing the bug and re-distributing the software. This happens when a manager sees the cost of fixing the problem as higher than the cost of creating the workaround. The problem, as all software engineers know, is that there is an under-appreciated hidden cost - the quotes around the word "temporary". It's not temporary, it will always be there and will always suck, because once the software works, it's too risky to change it.

My last government client was a great case study for how repeated use of workarounds can literally box you into a corner. We used the full Oracle J2EE stack, which is pretty buggy, especially if you're using the off-in-the-weeds use cases that the Oracle QA department didn't get around to. For every bug, another workaround from the on-site consultant. Management didn't see a problem with this, because a workaround appears to offer the same value as a fix, except for that pesky "temporary" bit - but hopefully that will be someone else's problem. In the end, workarounds on top of workarounds left the system fragile and unstable.

In contrast, look at the Mac OSX Save As... dialog. Apple recently posted on their propaganda blog about the keyboard shortcuts you can use in the dialog. It occurred to me that this tip applies to all applications, which makes the user trust that if they learn this behaviour, they'll be able to apply it whenever they encounter this dialog. That consistency is an attribute of quality software, and it's possible because Mac software (generally) uses the system-provided functionality to do lots of stuff.

If this were a Microsoft OS, when a problem was found in the Save As... dialog, rather than fix it, there would be a workaround. Each programmer can just make their own dialog box, that looks the same, but has the extra flag or modified data structure or whatever avoids the bug. And now, each of those developers can introduce differences, like being lazy and leaving out the keyboard shortcuts. Now the user can't trust that anything is consistent, so they avoid learning faster ways to use the "computer" as a whole.

To be fair, it's not realistic to expect workarounds to disappear. As a lead, sometimes you find yourself in a situation where a proper fix pushes out the schedule, and you weigh the costs and benefits, including how bad the workaround will be if it lives forever, and what implications it could have on the design. And you feel dirty.

Some people feel so dirty that they won't go along with it. There's a spectrum of developer reactions, from indignation that leads to an indefinite delay of the release, to an irresponsible lack of concern. But even if they're forced to give up their ideals and do some messy hack, ultimately, developers know that quality software has zero workarounds, period.

SO, I think perhaps the best metric for how bad your software is: how many workarounds it contains. Repeatedly leaving bugs in the software is a bad practice that not many managers really understand. There's a good reason for programmers to get out the resume paper when they see a culture of accepting workarounds on their project.

Darwinian Evolution of Software?

In high school math class, we learned about gradient descent. The idea is to find the low point on a graph, like finding the lowest elevation in a geographical area. Although it sounds pretty straightforward, there is a problem doing this: we can tell if something is a local minimum, but not if it is globally the lowest. The algorithm for gradient descent can be summed up "go downhill", which will eventually find you to a low point. It's possible that right next door to your low point, however, is a much lower point, but you would have to go uphill to get there so you're never going to.

Lots of things behave like this, which I've always found interesting. Rivers lie where they do because they seek out the lowest point. And evolution acts this way too, because the best it can do is to improve on what's currently available. Each generation can have mutations based on their parents, and if it's an adaptive mutation, the new trait lives on. But, if the improvement is dependent on something else happening first, and that prerequisite is detrimental, then the improvement isn't possible. That's like going uphill to look for a new minimum.


My favorite example of this is that the back our eyes have a hole where the optic nerves bundle together and pass through the eyeball to get to our brain. This is needed because the nerves come out of the front of the receptor cells, instead of the back. If the nerves came out the back, we wouldn't need the hole, but we happened to evolve the receptor cells and optic nerves in this order, where by the time the eye was finished, the pipes were layed wrong. If only the intelligent designer could refactor, right?

From http://www.2think.org/eye.shtml

Ok, now I'll make my point. In software, we should be intelligent designers, and we should be able to refactor. If we made an eye with a hole in it, we could probably fix that in release 2.0. And yet, in a larger and slower scale, software really does "evolve," and critical mistakes aren't corrected if it would mean the technology is temporarily crippled. Instead, we build on them and move on to the next layer of abstraction with those funky things still frozen in that layer.

Around 2002, as the web was turning into a place for applications to live instead of just reference material, a need arose to have appealing and highly functional user-interfaces in a web browser. We wouldn't have chosen to turn a web browser into a rich-client platform if we had a big meeting about it, but they were already deployed on everyone's desktop. It was clearly descending the gradient to just put an application on that platform, since we weren't sure our users would put up with downloading and installing something without their employer making it mandatory. Also, we need it to be cross-platform since we don't want to make any assumptions about what hardware our users have. Java tried to get onto the desktop, which would have made more sense, but it didn't catch on.

So then Web 2.0 came along, and improved the browser's ability to present a decent UI. We deploy rich-client applications in a browser using the XmlHttpRequest javascript feature. Javascript has to be the most abused programming language ever, and at first I was aghast that AJAX would push it to do so much. It felt buggy and unnatural, and again I think this was an evolutionary process - we're writing web applications, we have developers who know CSS and DHTML, lets put them to work.

Bruce Eckel wrote about this in his article on moving to Flex. He gives a nice history of the presentation layer, finally ending up on Adobe Flex. I'm sure many of us are a little annoyed by the frequency of the Bruce Eckel "thinking in Flex" ads that have been all over the place lately - it reminds me of how I wanted to get away from T-Mobile when they had Jamie Lee Curtis everywhere. But Flex does make evolutionary sense - now that we've evolved these capabilities to write web applications, let's improve our toolset and move to ActionScript and have some better graphics and animation capabilites. We don't have to deploy much to clients because again they mostly have flash plugins in their browsers already.

So now here comes Apollo:

Jesse and Gabe were experimenting with it, and from what I could tell, this is a neat way to take the current evolution of Web 2.0 presentation technology and bring it back to the desktop again. This is where I think of the hole in the eye - it's great to have a true desktop app that can interact with peripherals and be responsive and work offline, but some of the technology stack comes from the web. CSS wasn't intended to layout a rich-client desktop app, was it?