Environment specific settings for Log4J 1

Posted by wsargent Tue, 04 Sep 2007 05:59:00 GMT

Finally scratched a long standing itch today and wrote something to break up Log4J configuration and enable parameter substitution. You can download it here: layered-configurator.zip

LayeredConfigurator will go through a properties files with references to DOM or Property configurator references (files or URLs), and will call the appropriate configurator in the order they are defined.

This doesn’t mean much in theory, so let’s explain why this package exists, and give an example.

Log4J lacks a way to spread configuration over several files. You have one configuration file, either log4j.properties or log4j.xml, and that’s it.

In projects where you have several environments to keep track of, you may want to have some loggers set to DEBUG in the development environment, but set to WARN on the production environment. However, you usually want to keep all the appenders and logging infrastructure the same.

Because Log4J defines appenders and loggers in the same file, you either have to define several almost identical files with the appropriate changes for the environment, or you have to have an Ant script that goes through and replaces tokens for the appropriate environment.

The file syntax for the configurators is simple. Here’s an example log4j-config.properties file:

# The complicated appenders are rendered in XML.
base-appenders.xml

# The logging levels can be defined in properties, and can use system properties
# as parameters. (in this case we assume -Dmy.environment=dev is defined)
${my.environment}-loggers.properties

The class is called LayeredConfigurator because it’s meant to work in layers. The configurators are not reset, so all the settings from the previous configurator will still apply unless you override them.

When you define -Dmy.environment=dev, then your development settings will be loaded. When you define -Dmy.environment=prod, then your production settings will be loaded. Either way, your binary distribution is exactly the same, with only the environment specific properties

To enable this class, you must start the JVM with the following system properties:

 -Dlog4j.configuratorClass=com.tersesystems.log4j.LayeredConfigurator
 -Dlog4j.configuration=log4j-config.properties

And that’s it. Pretty simple solution for something that’s bugged me for at least three years now.

Recovering from Burnout 2

Posted by wsargent Thu, 30 Aug 2007 18:49:00 GMT

Right now, I know three people who are burnt out. Almost everyone I’ve worked with has worked to the point of burnout, and I’ve done it myself, both at home and at work.

Burnout is not unusual. It’s a physical exhaustion brought on by protracted, unrelieved stress. If the human body is under stress for long periods of time, every day, then eventually the body will break down, and take the mind with it. The precise mechanism is not known; it’s assumed to be a breakdown in the endocrine system that manages stress.

What distinguishes burnout from just plain stress is the inability to care; not because you are bored or apathetic, but because caring requires an insurmountable effort. People who are burnt out are disengaged. Going through the motions. Blunted. And unable to relax or breathe; any stress or unexpected activity will set them twitching. Because it’s brought on by prolonged stress over a period of weeks or months, it’s not an exhaustion that is cured by a couple of nights of good sleep. Because it’s physical, it’s not something that focus or willpower will fix. Trying to work through burnout will only make it worse. And denying burnout exists because you don’t have room in it for your life… well. I’ll tell you about a man I know who refused to admit to physical limitations. He worked until he collapsed on the floor of the server room and had to be taken to a hospital bed.

Of course, if you’re burnt out already, you are not in a position to fully appreciate it. In fact, one of the ironies in burnout is that the more burnt you are, the less able you are to effectively manage yourself and your stress levels.

The interesting thing about burnout is that it’s not brought on by prolonged periods of work. Work in itself is fine – it’s stress which is the problem. And what causes stress is not work.

I remember a study that measured the amount of stress that an executive and a secretary went through in a day by measuring their cortisol levels. What they found was that even though the executive was making decisions and working moment to moment more than the secretary was, the secretary was under more stress. Why? Because although the executive was doing more, he had more control over his decisions and his environment. The secretary was interrupted on a regular basis, was multitasking far more than the executive, and had less control over what was happening at work. The same results have been found in lab rats: what gives rats the most stress is not receiving electric shocks, but the uncertainty of possibly being given an electric shock. What causes stress is lack of control, and more basically a lack of respect.

That’s why Christina Maslach defines burnout as:

“The index of the dislocation between what people are and what they have to do. It represents an erosion in values, dignity, spirit, and will—an erosion of the human soul. It is a malady that spreads gradually and continuously over time, putting people into a downward spiral from which it’s hard to recover.”

In some ways, burnout is the end result of a reaction to stress. When people used to doing a good job feel out of control or lacking in respect, then they’ll work harder or try to take on their environment. They might feel in control or respected while they are working long hours or dealing with a stressful situation, but that will only last until they use up their reserves.

So what do you do about burnout?

1) Just about every article I’ve read says that you should consider quitting your job. I understand if this isn’t an option.

2) Learn how to say no at work. Let go of things that aren’t your responsibility.

3) Go home. It’s a marathon, not a sprint. You will only be able to do so much work in a day, no matter how much work you put in.

4) Relax. Accept that there will be time when you get home and don’t feel like doing anything. Lie there and practice blinking.

5) Meditate. Meditation’s a little strange at first, but studies prove it has a positive impact on stress and mood. Here’s how it works: you sit down, crosslegged, on a cushion. You set a timer for ten minutes. You count your breath on every exhale, from one to four. When you pass four, you count from one again. And you don’t try to think about anything else. You may notice thoughts passing through your brain, but you focus on counting breaths. For ten minutes.

There’s more to it than that, of course. But that’s really it – the idea is to be able to focus on one thing without getting distracted. (Or thinking of a pink elephant. Oops.)

5) Delegate. Whil Hentzen recommends getting an administrative assistant. He goes into it in some detail, but the basic idea is that there are any number of small, non-technical tasks in the course of the day that could be passed off to someone specifically for doing those jobs. I’ve often thought this would be a great idea; my parents work in film and video work, and it’s common practice to have someone who can fill in when needed.

6) Vacation. I’ve heard various people refer to this as “unplugging.” I’m not sure I buy this route; it seems to be a short term solution that doesn’t really address the problem.

The best thing you can do is be aware that it is real, it does exist, and that, left untreated, it can cause problems that are not easily solved. The New York Post has an excellent article on burnout, although it tends to meander towards the end.There are a couple of articles that really nail the experience of burnout:It’s Not Just You: Exploring the causes of worker burnout (which has a link to What Makes the Job Tough? The Influence of Organizational Respect on Burnout in Human Services, a PDF that is the basis of the article) and Job Burnout. I highly recommend you read them.

Without Wings 1

Posted by wsargent Mon, 20 Aug 2007 22:35:00 GMT

I’ve been going to AYE for years now. The last time I went, Charles Adams asked me “Will, every time you come here, you seem so frustrated.” I don’t disagree. For me, AYE is frustrating for many different reasons. But part of it is just that AYE is by its nature, ahead of the curve.

People in AYE take for granted concepts and methodologies which have not filtered out into the wider world. And much of the discussion is not about getting the framework up. It’s about incremental improvements on what is already known.

Over several years, I’ve made progressively closer steps to agile. I’ve written unit tests, mock objects and unit testing frameworks. I’ve written integration tests and built integration test frameworks, and can talk about data driven frameworks. I’ve created repeatable, automated build and deployment processes and have continuous integration with built in code analysis and code coverage.

But I’ve never worked in an agile project using iterations, or a company comfortable with agile processes. The closest I’ve ever come has been the promise of a discussion where agile would be discussed as a serious option. Someday.

To me, AYE, or BayXP, or any number of blogs, sound like discussions on the best way to fly using the wings on your back. If you have wings, flying is easy. If you don’t have wings… how do you get them?

I’m not trying to be funny here. I think that agile processes are an honest attempt to redefine the practice of software engineering. But there is a divide between the have and have-nots. And trying to learn what I can in a field in which I can’t practice it is… well… it’s…

It’s frustrating.

Getting work out of Programmers, Part 2

Posted by wsargent Mon, 20 Aug 2007 04:07:00 GMT

So here’s how I think you get the most work out of programmers. This is a follow on from part 1.

Morale. Programmers have good morale when they are treated well, and they are given a problem that they know they can solve. Thank programmers whenever they do something. Let them know you not only know how much they work, but that you care. Keep track of morale through weekly one on ones with each and every programmer. Keep track of commitments you make to your programmers (including the verbal ones) and follow through on them. Make it clear that you are looking out for their interests. (Creating a Software Engineering Culture, Chapter 3.)

Money and stock options only have a limited effect on morale, and may have a negative effect (Agile Development, p63). Most often, a gift of money or stock options is a substitute. (Rapid Development, p262). Morale events can be fun, but don’t actually raise morale – they just allow for a different kind of interaction with people. (How to avoid Lame Morale Events).

And there are all kinds of things that can hurt morale. I think the major one is the Broken Window Theory (Pragmatic Programmer, p4). Under the Broken Window Theory, any neglect or rot in a system that is not directly addressed and countered is a drag on morale. People wonder why it is that they have to write good code and do things right, when they’re not allowed to fix the crappy code. Management assertions that “we don’t have time right now” or “we’ll do it later” start to sound empty and hollow as project after project goes by, and the crappy code festers and rots as hack after hack is piled on top of it.

The Psychology of Computer Programming, Chapter 10 deals specifically with Morale and Motivation. Rapid Development, Chapter 11 goes into typical developer motivations. They are both very much worth reading.

Sleep. You can’t make people sleep, and you can’t do much about sleeping arrangements. But you can tell them that you want them to get 8 hours of sleep a night, and you can bring up lack of sleep as an issue. Anyone who looks sleep deprived needs help; either they have been trying to sneak in work late at night, or they’re suffering in other ways. Give them all the help you can. The number of work hours will be an issue there. And if someone loads up on caffeine and junk food late at night… well, I’d point out that there may be a connection (How to Sleep Better).

Exhausted employees are easy to spot. They’re the people who frighten small children and spouses. They are not fit for work. They barely even know they are at work. They should be sent home until they know what they’re doing. Just that simple act of humanity will raise morale.

Focus. The best way to ensure focus is to ensure transparency and feedback. If programmers have a public, physical way to see what needs to be done at a granular level, whether in a todo list on a whiteboard or a series of 3x5 cards on the wall, they can see at a glance what they will be working on not just today, but next week as well. (Agile Development, p97) This works very well for helping out programmers who have a large list, or determining the priorities – you can only have one task on the top of the list, so what you are supposed to do is never in doubt. This is a technique that is used in several agile methodologies, and is known as an information radiator. (Crystal Clear, p32)

Don’t confuse your programmers, or worse, try to multitask them. If you give them two jobs to do at the same time with the same priority, you’re putting them in a situation where they cannot win; no matter what they do, they’ll be working on the wrong thing. And if they try to do them in parallel, they’ll do both jobs more slowly than they would if they did them sequentially. (Joel on Software) (The Multitasking Myth) (Quality Software Management)

Keep the number of goals small in a project. Pick one objective and make it clear that it’s the most important one. (Rapid Development, p257)

But the best thing you can do for programmers is to let them work. Don’t interrupt them. Don’t spring meetings or interviews on them with no warning. Don’t change what they’re working on. Don’t spring last minute high priority projects on them. Don’t file marketing requests to change the font size as priority one critical bugs. Don’t come up and ask when a bug is going to be fixed. Don’t ask them for status updates every five minutes. I’ve seen constant change requests happen at company after company and I can tell you from experience what happens… the programmers roll their eyes, and they stop taking priority changes seriously. Because they know that in an hour’s time, it won’t be a priority any more. (Rapid Development, p259)

If you let your programmers work uninterrupted, something wonderful will happen. There’s a psychological state called flow that has been documented by Mihaly Csikszentmihalyi. In this state, programmers are able to be “in the zone” and become focused to a great extent. Programmers in flow can produce far more code than they would be able to ordinarily. There is a catch though; it takes the average person at least 15 minutes of uninterrupted work to enter flow. If they are interrupted, or even expect to be interrupted, then they’re less lightly to enter flow. (Rapid Development, p506) Some teams implement a policy called “focus time” (Crystal Clear, p33) or the “cone of silence” (Alistair Cockburn) where meetings and interruptions are banned for a portion of the day. Other teams use a red bandanna (Peopleware) or a sign to indicate that they are not to be interrupted, or other methods.

Background. The best way to have programmers with the best background in the problem domain is to cultivate them. I’ve been surprised in the past how little programmers know what the business does. I’ve often thought it would be a useful exercise to have new programmers spend some time with each member of the business team to understand their concerns and priorities. Failing that, it can be a good idea to have documentation (business process management or six sigma documentation) that can bring new programmers up to speed on the organization as a whole. Of course, this only works so far in that it doesn’t track the history of the organization. Legacy code is nettlesome to new programmers, because typically they reflect legacy business processes and legacy business decisions. But ultimately, it comes with time.

Business specific domains, by nature, share little common ground with each other. There are only a couple of books I can recommend here. Domain Driven Design is the single best book I have read about how to effectively model and discuss domains. It is a similar book to Design Patterns, as it not only talks about implementation and common patterns, but it talks about domains as a common language. And Working Effectively with Legacy Code does an excellent job of pointing out useful ways to desnarl and refactor code that is no longer up to snuff.

Experience. Experience comes with work. It doesn’t always come with time. To quote Weinburg, experience is the best teacher, but it doesn’t necessarily teach anything. Experience can be passed on by proxy, through education and mentoring: some of the best experiences I’ve learned from have been the ones other people have had. If you want books that give experience, then Code Complete 2 and Refactoring are the best bets. If you want to read about experience, then Software Craftsmanship and Software Creativity are the best books. But if you want to make gathering experience, then make it available to your programmers. Set aside an education budget. Encourage your employees to attend conferences and seminars. Join a software engineering book club and have programmers pick out reference books for an in-house library. Have regular brown bag sessions and encourage your team to pass techniques around the company. Do this, and you will not only raise the general experience level of your team, but you will raise morale as well. (Rapid Development p257) (Software Craftsmanship, Chapter 19) (Building a Software Engineering Culture, Chapter 4).

Talent. You can hire talented programmers, but I think that’s as much as can be done. I think that talent is not a static quality. I believe talent is a series of mental habits, and that new habits can be learned, just as old habits can be put aside. I believe that some useful habits are solid grasp of systems theory, along with an ability to ask the “right” question. But that’s another essay. And there is another problem: talented people get bored doing things that don’t stretch their capabilities. If you have work that doesn’t require PhDs, you may be better off not hiring them.

Hours Worked. According to decades of economists and management experts: 40 hours. Contrary to popular belief, the standard work week was not invented by the government or the unions. It was pioneered by Henry Ford in 1926. More than 40 hours a week, and the factories didn’t produce as much money; the temporary increase in productivity was more than offset by industrial accidents and mistakes, and after two weeks there was less productivity. (Why Crunch Mode Doesn’t Work) (Can People Really Program 80 Hours a Week?) (When Should You Start Project Overtime?)

This is counter-intuitive, so I’ll say it again: studies prove overtime provides a temporary benefit for a maximum of two weeks, and is worse than useless thereafter. James Shore provides a recent example.

I believe this, not just from the studies, but from my own experience with extended overtime. Programmers will not only do less work, they’ll make mistakes in the work that they do. Then they’ll get irritable and suffer from low morale. Then they’ll burn out completely. I believe (but do not have the studies to prove) that after even after normal work hours are restored, there is a convalescent effect; people will produce less work following the overtime, producing the same amount of work overall. So after the project goes live, they’ll need a long convalescent period before they’re up to snuff, or even worse, they’ll look up from their monitors, take a good hard look at the results of their labor and their (usually meager) rewards, and quit, producing a huge opportunity cost for the company in terms of hiring, maintenance, and reputation. (It’s Not Just Abusive, It’s Stupid)

For every complex problem, there is a solution which is simple, obvious, and wrong. Extended overtime is that solution.

Fine; hours worked have no effect. What about directly applied pressure? What happens if we keep the hours, and if we tell the programmers to work harder and produce more work during those 8 hours?

Surprisingly, nothing. Programmers produce work using their brains; the amount of thoughts a programmer can have is fairly constant. Tom DeMarco and Tim Lister researched this and formulated Lister’s Law: People under time pressure don’t think faster. (Slack, p50) They might be more stressed, but that doesn’t help people think faster; stress impedes complex thought, and pushes the brain to a “flight or flight” response. If you’re stressing your programmers, they’ll be at their desks more. But they’re not going to write any more code.

This advice is all simple and straightforward. I don’t see anything in here that comes as a shock, and even my mother said “Well, that’s obvious, isn’t it? It’s like being a farmer and taking care of your cows. If you want your cows producing the most milk, you make sure they’re treated like cows should be treated.”

So treat your programmers well. Keep track of morale through weekly one on ones. Make it clear you care about the health and welfare of your programmers. Make sure programmers know what they should be working on at all times. Don’t change out work that the programmers are doing or abuse the bug tracking system. Keep interruptions to a minimum to allow for flow. Establish a training and education budget, and establish mentoring and brown bag sessions to transfer experience. Keep to 40 hour work weeks, and forgo direct pressure.

Do all of these things, and you’ll get more work out of your programmers. And you’ll probably have programmers beating down your door to work for you.

EDIT: Also see this LinkedIn question that provides some useful advice. Larger monitors have been mentioned in several studies, but I don’t have the references to hand.

Getting work out of Programmers, Part 1

Posted by wsargent Fri, 17 Aug 2007 02:20:00 GMT

I was recently asked the question: what is the best way to get the most work out of my programmers?

I’ve thought about this for a while. In some ways, it’s the story of my career – every manager I’ve ever had has wrestled with this question. I’ve worked with enough teams and individuals that I think I have a good understanding of what programmers can and can’t do, and how work gets done. And how work doesn’t get done, for that matter. I believe I have an answer, although certainly not the answer.

The first thing to do is to look at the assumptions behind the way the question is phrased. The assumption here is that there are veins of untapped work, hidden inside of programmers somewhere. And the manager’s job is to extract it. I think that this is the wrong place to start from. I believe that most people would rather do a good job than a bad one, and will do the best work they can do given what they have to work with. (Software Creativity 2.0, p22.) The best way to phrase this question would be: “What can I do to help an individual programmer be most productive?”

So, let’s construct a hypothetical employee. What is known to create productivity? Let’s start from the simplest level and work up.

Morale. An employee who feels safe and secure in his position and his ability to raise issues will do more work. You can argue whether this is the cause or the effect, but it’s well recognized that morale is extraordinarily important. Someone who believes in the team, the company and the work he is doing is likely to do more of it. I believe motivation and morale are equivalent, but morale feels like a better word to me.

Sleep. An employee who has slept 8 hours a day will be more productive than one who has slept 5 hours. I have yet to hear anyone argue with this in principle. Studies have been done that compare the effects of sleep deprivation to alcohol intoxication. The outcome: a programmer awake for 21 hours is effectively too drunk to drive. Think about that. There is a second stage of incapacitation, which is exhaustion AKA burnout. Exhausted employees are, by definition, not productive.

Focus. An employee who knows what he is supposed to be working on will be more productive than one who is not sure which programming task takes priority. An employee who knows that he can complete the work he is doing without having his priorities changed will be more productive than one who does not know what’s coming next. Programmers dislike surprises as much as anyone else. But this also covers the programmer’s inherent self-discipline. Given a gap in work, will the programmer ask what he should be working on, or will he create his own scripting language?

Background. An employee who knows the domain, the history and the subtle interplay of forces at work in the codebase will be more productive than a programmer who has no previous background in the work assigned. This factor is often overlooked, but domain expertise is an incredibly valuable asset for a programmer, and anecdotally speaking I’d say it makes an order of magnitude difference in time and quality.

Experience. An experienced programmer – a veteran – is more productive than an inexperienced one. An experienced programmer will understand that this project needs two weeks just to be deployed, that there’s a particular tool that’s perfectly suited to refactoring the database, and when it’s worth building extra flexibility into an area of the system because the business team will almost certainly want customized logic in the next release.

Talent. Some people are just good programmers. It’s not simply intelligence, or interest; I’ve known very smart people who don’t make good programmers, and some of the most interested programmers often did the worst work. A talented programmer may do work that may be simply beyond other programmers, no matter how much experience or background they have.

I don’t think anyone will argue that the above factors matter. If any one of these goes to zero, your programmers will be effectively useless. There’s also another factor, which is more complex.

Hours Worked. A programmer who works 40 hours a week will do more work than one who works 20 hours a week. And a programmer who works no hours a week, will do no work. But I think everyone will agree that a programmer who works 168 hours in a week will not produce 168 hours of work. In fact, I feel fairly safe in saying that a programmer who works 168 hours will produce less usable work than the programmer who works 40 hours, because the programmer’s ability to work will be degraded by lack of sleep long before they stop working.

Obviously, these factors are interconnected. Someone with no experience will typically lack background. Someone with no sleep will have low morale very shortly, and will even lose talent. So let’s see what we can do to increase these variables.

Custom Argument Matchers in EasyMock 1

Posted by wsargent Sun, 05 Aug 2007 22:15:00 GMT

This post goes further into EasyMock, a tool for reducing dependencies in unit tests.

Say that you’re dealing with some complex object that has to be generated by the method under test, and you want to have a unit test verify that the object is created with all the configured parts necessary. You can’t simply use equality, and there may be parts of the object that you specifically do not want to test.

So you create a custom argument matcher and pass it back through the system. This is a pain the first time, but afterwards you can reuse the argument matcher in many different tests, so it works out.

So, here’s an example. We’re going to pass in a Car, and we expect to have a new object instantiated with some of those settings.

public class CarServiceImpl implements CarService
{

CarDao carDao;
public void createCar(Car pCar)
{
     Car car = new Car();
     car.setColor(pCar.getColor());
     car.setModel(pCar.getModel());
     car.setYear(pCar.getYear());
     carDao.addCar(car);
 }

}

We can mock out the carDao with EasyMock:

public class CarServiceImplTest {
    CarDao carDao = createMock(CarDao.class);
    CarServiceImpl carService;

    @Before
    public void setUp() throws Exception {

        carService = new CarServiceImpl();
        carService.setCarDao(carDao);
    }

    @Test
    public void testCreateCar() {

        Car car = new Car();
        car.setColor("pink");
        car.setModel("lamborghini");
        car.setYear(2000);

        // expect this
        carDao.addCar(car);

        replay(carDao);
        carService.createCar(car);
        verify(carDao);
    }
}

Except that… oops. The carDao isn’t going to be called with that object. The object is instantiated inside the method, so you’ll end up with this:

java.lang.AssertionError:
  Unexpected method call addCar(com.tersesystems.easymock.Car@67ac19):
    addCar(com.tersesystems.easymock.Car@53c015): expected: 1, actual: 0
<typo:code>
</p>
<p>There are two ways we can deal with this problem.  We can either expect something vaguer:</p>
<p>
<typo:code>
carDao.addCar((Car) anyObject()); // vaguest
carDao.addCar(isA(Car.class)); // a little less vague

However, you cannot say:

carDao.addCar(and(isA(Car.class), eq(car.name, "blue"), eq(car.model, "ford"), eq(car.year, 2000));

(At least not without using a custom plugin called EasyMock PropertyUtils. But let’s not get into that right now.)

If you want to make sure that the object being passed into a mock has a precise state, then you have to be explicit about it. This means creating a custom argument matcher.

A custom argument matcher is a class that implements IArgumentMatcher. It provides EasyMock with a way to know that we want an object compared using specific criteria rather than instance equality or equals().

So we want a CarMatcher. The CarMatcher class looks like this:

public class CarMatcher implements IArgumentMatcher {

    private Car expected;

    CarMatcher(Car pCar)
    {
        expected = pCar;
    }

    public static final Car eqCar(Car pCar)
    {
        EasyMock.reportMatcher(new CarMatcher(pCar));
        return null;
    }

    public void appendTo(StringBuffer b) {
        b.append("eqCar(").append(expected).append(")");
    }

    public boolean matches(Object arg0) {
        if (! (arg0 instanceof Car))
        {
            return false;
        }

        Car actual = (Car) arg0;

        String model = expected.getModel();
        if (model == null && actual.getModel() != null)
        {
            return false;
        } else if (! model.equals(actual.getModel()))
        {
            return false;
        }

        String color = expected.getColor();
        if (color == null && actual.getColor() != null)
        {
            return false;
        } else if (! color.equals(actual.getColor()))
        {
            return false;
        }

        Number year = expected.getYear();
        if (year == null && actual.getYear() != null)
        {
            return false;
        } else if (! year.equals(actual.getYear()))
        {
            return false;
        }

        return true;
    }
}

And then we can rewrite the unit test to take advantage of the CarMatcher:

carDao.addCar(CarMatcher.eqCar(car));

So the next question: is this worth it? I’d say yes, because you can reuse the CarMatcher across many different unit tests. You’ll run into code like this in many different situations, such as dealing with DOM4J or XOM elements, where you want to check that they are being created correctly inside of a method. These kinds of cases are perfectly suited for unit tests, because there are not that many dependencies, and they’re small enough and fiddly enough that they’ll be missed by larger integration tests. So it’s good to have on hand.

EasyMock Examples 7

Posted by wsargent Mon, 25 Jun 2007 05:41:00 GMT

I’ve played with EasyMock for a while now, and I think I get it. I tend to see more from examples, so this is my interpretation of the EasyMock documentation.

Say that you have a class:

public class CommentService
{
   private CommentDao commentDao;
   private CommentDao getCommentDao() { return commentDao; }
   private void setCommentDao(CommentDao commentDao) { this.commentDao = commentDao; }

/**

* Adds a comment to the database.
*/

public void addComment(String name, String message) {

    Comment comment = new Comment();
    comment.setName(name);
    comment.setMessage(message);
    CommentDao dao = getCommentDao();
    dao.save(comment);

} }

There are several things that you want to happen in the unit test for the code. You want to make sure that dao.save is called. You want to check that the comment exists, and that it has the name and message that you passed in.

With all of that, the unit test looks like this:

public class CommentServiceTest
{

private CommentService commentService;
private CommentDao mockDao;

@Before
 public void setUp()
 {
     commentService = new CommentService();
     mockDao = EasyMock.createMock(CommentDao.class);
     commentService.setCommentDao(mockDao);
 }

@Test
public void testAddComment()
{   
      // We expect this method to be called when we call commentService.addComment()
      // and we don't care about the object that gets passed in...
      mockDao.save(EasyMock.anyObject());

      // Okay, we're done setting the mock up.
      EasyMock.replay(mockDao);

      // Run the method.
      commentService.addComment(name, message);

      // Check that the mock's happy with what happened between it and the service.
      EasyMock.verify(mockDao);
}

}

So that’s the simple case. What about situations where you’re relying on a context object being passed in, and you get your services from there?

You have two layers of lookup:

@Test
public void testApplicationContextLookup()
{

ApplicationContext mockContext = EasyMock.createMock(ApplicationContext.class);
ExampleService mockService = EasyMock.createMock(ExampleService.class);

// Here we DO care about what gets passed back:
EasyMock.expect(mockContext.getBean("exampleBean")).andReturn(mockService);
mockService.doStuff();

EasyMock.replay(mockContext);
EasyMock.replay(mockService);

// etc...

}

This works for interfaces. But what if you want to mock out classes?

Well, then you need the EasyMock Class Extension library. This is basically the same interface as the other library, the difference being that you need CGLIB to fill in a mock proxy class for you.

@Test
public void testHashtable() {

Hashtable hashTable = createMock(Hashtable.class);

Object key = anyObject();
Object value = anyObject();
expect(hashTable.put(key, value)).andReturn(value);

replay(hashTable);
callTheHashTable(hashTable);

verify(hashTable);

}

Okay, so we can do classes. What about other stuff? What if we only want to mock out a particular method in a class? Well, we can do that as well.

Hashtable hashTable = createMock(Hashtable.class, new Method[] { Hashtable.class.getMethod("put", Object.class, Object.class) } );

What if we don’t know what kind of object we want to return, or need to do some logic to construct a big hairy object? Use IAnswer:

expect(service.getSystemTime()).andAnswer(new IAnswer() {

  public Object answer() throws Throwable
     long setTime = 0; // return a custom time back based on some algorithm...
     return new Long(setTime);
  }

});

What if we want to verify that mock1 is called before mock2? We have to break down using a IMocksControl object.

@Test
public void testMockOrder() {

IMocksControl control = createStrictControl();

List mockList = control.createMock(List.class);
Map mockMap = control.createMock(Map.class);

// Set up the expected calls...

replay(mockList, mockMap);

// Make sure "list stuff" is done before "map stuff"
fixture.doStuff(mockList, mockMap);
verify(mockList, mockMap);

}

This does not cover everything, of course. In particular, it doesn’t cover some of the stranger edge cases (static methods, constructors, finalizers) that code may be nestled in. Also, private and final methods / classes are not covered at all by EasyMock.

Configuring Hibernate properties in Spring 2

Posted by wsargent Sat, 17 Mar 2007 02:28:00 GMT

Okay, so here’s the deal.

I want to have Hibernate work against Oracle and HSQL. I want it to be configured through Spring, and I want it to not be the schema owner in Oracle.

Setting a different schema owner is easy in Hibernate. You specify this in a spring.properties file, using PropertyPlaceholderConfigurer:

hibernate.default_schema=db

and then you have a properly parameterized Hibernate through Spring:

<bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">

&lt;property name="configLocation"&gt;
   &lt;value&gt;classpath:hibernate.cfg.xml&lt;/value&gt;
&lt;/property&gt;
&lt;property name="dataSource"&gt;
  &lt;ref bean="dataSource" /&gt;
&lt;/property&gt;  
&lt;!-- Overrides what is defined in hibernate.cfg.xml --&gt;
&lt;property name="hibernateProperties"&gt;
  &lt;props&gt;
    &lt;!-- properties chopped for space --&gt;
    &lt;prop key="hibernate.default_schema"&gt;${hibernate.default_schema}&lt;/prop&gt;
  &lt;/props&gt;
&lt;/property&gt;

</bean>

This works fine with Oracle. When you try a default schema with HSQL, it complains, because it doesn’t like the DB.table syntax.

However, setting hibernate.default_schema to an empty string doesn’t work. Even if you somehow hacked PropertyPlaceholderConfigurer to have ${null} as a value (which it won’t do, you can only do it through XML), it wouldn’t work, because a Properties object can’t have a null value.

The way to do this is to use a PropertiesFactoryBean, and define any “conditional” properties through the locations. Then you can define a property as hibernate.config.file=hibernate-oracle and have it pick up hibernate-oracle.properties with the hibernate.default_schema=${hibernate.default_schema} defined in there.

<bean id="hibernateProperties" class="org.springframework.beans.factory.config.PropertiesFactoryBean">

&lt;property name="properties"&gt;
  &lt;props&gt; 
      &lt;!-- normal properties --&gt;
  &lt;/props&gt;
&lt;/property&gt;
&lt;!-- hibernate.config should be defined somewhere in the spring.properties layers --&gt;
&lt;property name="locations"&gt;
  &lt;list&gt;
      &lt;value&gt;classpath:${hibernate.config.file}.properties&lt;/value&gt;
  &lt;/list&gt;
&lt;/property&gt;

</bean>

<bean id=”sessionFactory” class=”org.springframework.orm.hibernate3.LocalSessionFactoryBean”>

&lt;!-- normal stuff --&gt;
&lt;property name="hibernateProperties"&gt;
  &lt;ref local="hibernateProperties" /&gt;
&lt;/property&gt;

</bean>

Using DBUnit Effectively on Oracle 10g 5

Posted by wsargent Wed, 14 Feb 2007 02:51:00 GMT

Integration tests are the rule when it comes to testing. It is far more common to run a script to drop and recreate the database, set up some initial data and then manually work through the test case than it is to run a unit test.

So I’ve been playing with DBUnit, trying to get it to work with both Oracle 10g and HSQL. This is what I’ve found out about DBUnit 2.2, especially working with Oracle 10g.

Create a utility class to manage DBUnit operations

Many of the examples given in the DBUnit assume that you’ll be extending the DBTestCase class. You don’t need to do this, and I think it’s a move that will limit your options. Consider that there are only so many things that you can do with DBUnit, and look at writing a utility class that will manage the low level interaction for you, such as this:

public void insertDataSet(String schema, IDataSet dataSet) throws DatabaseUnitException, SQLException, IOException
{

LOGGER.info("inserting dataset: " + dataSet);

DataSource dataSource = getDataSource();
// Using spring-mock here as well, although you can do this directly
Connection connection = DataSourceUtils.getConnection(dataSource);
IDatabaseConnection dbconn = new DatabaseConnection(connection, schema);
try
{
    DatabaseConfig config = dbconn.getConfig();

    // Define the schema using a streaming configuration.
    config.setProperty(DatabaseConfig.PROPERTY_RESULTSET_TABLE_FACTORY, new ForwardOnlyResultSetTableFactory());

    DatabaseOperation.INSERT.execute(dbconn, dataSet);
} finally
{
    dbconn.close();
}

}

or use the IDatabaseTester interface.

Consider using P6Spy

In as much as DBUnit works, it runs JDBC statements. If you’re having problems with DBUnit, P6Spy will show you what JDBC statements are actually being run by DBUnit. This has helped me more than a few times when I’ve forgotten a step.

Purging Recycle Bin

DBUnit will not pick up on “purged” tables in Oracle, and will give you errors like this:

org.dbunit.database.AmbiguousTableNameException: BIN$KWdKkk3jYEvgQAB/AQAa8A==$0

A bug has been filed here. There are two solutions to this problem that I’ve seen. One is to run “PURGE RECYCLEBIN” before you try any database operation in Oracle. The other is to modify and recompile the source, following the instructions in the bug.

Truncating tables

Oracle does not allow you to truncate tables that have a foreign key constraint. Ever. Even if there is no data in either of the tables. You will get this error:

"ORA-02266: unique/primary keys in table referenced by enabled foreign key"

And then you’ll wonder which constraint on which table was responsible, because Oracle doesn’t bother to tell you. This query will show you the offending constraints:
select constraint_name from user_cons_columns where table_name = 'example_table';

And then you can disable a constraint with this command:
alter table foo disable constraint constraint_name;

If you use the DELETE_ALL command, then it’ll be slower, but it’ll work out of the box. Sometimes, that’s enough. Alternately, you may want to consider disabling foreign key constraints for the duration of the tests. If you know everything is valid, and you’re setting up the database just so you can get to a known state, then that should be good enough.

Always Specify the Schema

Oracle likes to have the schema defined. If you don’t have the schema defined, then DBUnit will get very confused. You can specify explicit schema names using a property, but then you can’t share XML data sets very easily.

Be Careful with CLOBs and LONGs

DBUnit takes some shortcuts when working with Oracle CLOBs. It assumes that you’re not using a DB Proxy, and it assumes it knows what Oracle class is valid. This assumption will fail if you are using a database proxy such as C3P0. Filed bug 1637073, but I do not know of a non-intrusive workaround at this point.

DBUnit also does not interact well with Oracle LONG objects. The bug is here and there is a workaround in the bug comments.

Use the right tool for the job

As much as DBUnit is useful for importing data, it’s still written for ease of use and generality than it is raw speed. You can use ForwardOnlyResultSetTableFactory to speed it up, but ultimately using the 10g data pump facilities is going to be faster than using DBUnit by an order of magnitude. If the data pump facilities don’t do it, you can use external tables and transportable tablespaces to set up a large database even faster.

Snippets

Posted by wsargent Wed, 31 Jan 2007 17:36:00 GMT

Me: “I think my best quality is my humility. Sometimes I’m so humble, I can’t stand it.”
Jessica: “Yes dear. Neither can I.”