Terse Systems

Why Amazon S3 backup doesn't work

Sat, 12 Jul 2008 14:00:00 -0500

I've been going through an inordinate number of machines lately. I'm down one Seasonic power supply, two motherboards (one Megabyte, one MSI) and the Thinkpad came back from lenovo with the original problem solved, but with half the screen on the fritz.

This isn't new or unusual; most of the machines I've put together have only lasted a couple of years or so. Consumer hardware just isn't built for reliability; hard drives being the worst offenders. As a result, most of the information I really care about is based off-site. The two biggest exceptions are the iTunes library, and my financial data.

The standard solution is to have a backup hard drive. The problem here is that backup hard drives also go bad. I've run into this a couple of times with the Linkstation and the Western Digital External. There's not much you can to do test this except restore from backup every once in a while and see if it actually worked (which is about as fun as it sounds).

Another possible solution is to use offsite backup such as Mozy or Amazon S3. I read Zawodny's experiment with it and started using s3sync. But.

1) Uploading 30GB of music took a week.
2) Most, if not all, of that data was corrupted to the point of unintelligibility.

I only found this out when I thought I was missing some data, of course. Downloading 30GB of data is also a pain (and frankly, I didn't expect so many errors -- every single track sounded like it was playing underwater at half speed.

So. It turns out the best solution is to make sure you have an inordinate number of machines, and have them synced off the master. I may not be able to rely on one machine, or on a backup hard drive, but I can have music (or any encrypted data) put onto available laptops -- and I know that data is good, because I play it from there. This is a looser implementation of jwz's backup strategy, but it's good enough for me.

Thinking about this a bit more, it's surprising that backup technology is as limited as it is. Creating an encrypted bittorrent and sharing it amongst a pool would be an excellent way of ensuring redundancy and error correction (is this what Tor does?), but you may not even need to go that far; every time you do a backup, encrypt it, chop it up into yenc blocks, and dump it onto Usenet. A thousand servers will pass it around and make your data retrievable for all time.

Topspin

Fri, 20 Jun 2008 20:15:00 -0500

The new company blog is up, and I can finally talk about some of the stuff I've been doing for months. Stealth mode, you can keep.

It is out of character for me to gush. But seriously, Topspin is the first company I've ever been in where I look forward to the meetings. Things just keep getting better every sprint. Our customers are literally rock stars. I've been a huge NIN fan since I was 18, and just getting the opportunity to help out was beyond awesome. It's in Potero Hill right next to the Whole Foods, I bike to work every day, and every day there's some DJing tunes through the Airport Express.

Oh, and the Billboard cover was nice.

Evernote + Camera import = love

Sun, 18 May 2008 11:17:00 -0500

Okay, I understand why Evernote is so cool now.

Take your digital camera and take photos of all your handwritten notes. Import them to the PC. Rotate them so that they're the right way up. Drag them into Evernote. Wait for it to finish indexing. Presto. You now have all your notes in digital format no matter which computer you're on, and you have them indexed and searchable.

Good Karma

Fri, 25 Apr 2008 20:18:00 -0500

Today, my brother and I found a man lying in the street on Van Ness and Market. His eyes were open but he wasn't moving, and he was clearly unconscious. I checked he had a pulse and was breathing, phoned 911, and tried to figure out what the hell happened.

Pupils weren't dilated. He had blood in his mouth, and it looked like he'd bitten his tongue. The back of his head was bloody. His heart rate was elevated, and his breathing was ragged -- it sounded like his tongue was rattling in the back of his throat every time he breathed. It looked like he'd just been walking down the street, bit his tongue really hard and then fell straight back onto the pavement.

He started to move around a little bit after four minutes, and after five could say 'police' and 'wanna get up'. I didn't think it was a good idea for him to move until the paramedics came by, but I wasn't going to hold him down either. It turned out he couldn't really get up by himself, so we supported him until the paramedics came by.

I'm very glad he didn't stop breathing. I was not looking forward to giving him the breath of life.

An hour later, we successfully contested a parking ticket before a hearing officer.

Advanced Mocking

Sat, 15 Mar 2008 19:10:00 -0500

I know enough about EasyMock now to feel very comfortable with it. I've even got an acronym for writing unit tests with EasyMock -- SEREVA.

This breaks down to the following stages:

Setup the object under test with any required mocks.
Expect some methods on the mock to be called.
Replay the mock once all the expectations have been made.
Execute the object's method when under test.
Verify the mock has had all its expectations met.
Assert the output of the method.

This basic framework takes care of most of the grunt unit tests.

But there's more to mocking than that. There's one particular gotcha I know about with EasyMock, and then there's the horrible, nasty, no-good edge cases that you run into. Let's start off.

Custom Argument Matching with Multiple Parameters

Let's start off with the gotcha first. Say you've got a method like the following:

List findCarsByYearAndColor(Integer year, String color);

Say that you want to match year, but not color. You can't say:

expect(carService.findCarsByYearAndColor(year, anyObject()).andReturn(null);

If you want to apply a custom argument matcher to either argument, you must apply it to both. So instead, you need to do this:

expect(carService.findCarsByYearAndColor(same(year), anyObject()).andReturn(null);

This is why same() exists.

Static and Final Methods

EasyMock does not allow you to mock static or final methods. EasyMock ClassExtension does not allow you to mock private methods or static classes. You can argue that such things should not exist, but we live an imperfect world where people don't always think of our needs.

This means that dealing with final classes like java.net.URL or java.io.File are fiddly. But not impossible. There's more than one way to skin this particular cat.

The by hand way is to use reflection: "final" is not final anymore. Heinz demonstrates how you can change final fields through several different versions of the JVM. This is a somewhat disturbing read, as it plays merry hell with my conception of what should and shouldn't be possible in Java. But it's there if you need it.

The snazzy way is to use JMockit, which is a small toolkit specifically designed to subvert the JVM. jMockit depends on the java.lang.instrument library, which is only available in JDK 1.5, but it provides functionality you can't find anywhere else.

The alternative method is to say "if your code is hard to test, then your code is badly written." While there may be a correlation, I think that what is 'badly written code' can only really be accurately be judged by a human being.

Private, Default or Protected Methods

jMockit is all over this.

Native methods

Don't know.

Static or Final Classes

This is handled as above by reflection, or by jMockit.

Inner Classes

Don't know. I suspect that anonymous inner classes are very tough. I believe that static inner classes should be accessible through jMockit, but haven't checked.

Constructors

EasyMock 2.3 allows you to define a partial mock that will call a specified constructor. That's good if you have something that needs to be passed into the object under test. So you can do:

public static abstract class A {
    public String s;

    protected A(String s) {
        this.s = s;
    }

    protected abstract int someMethod();
}

public void testFoo()
{
    Constructor cstr = A.class.getDeclaredConstructor(String.class);
    ConstructorArgs constructorArgs = new ConstructorArgs(cstr, "test");
    A a = createMock(A.class, constructorArgs, new Method[0]);
      
    foo.setA(a);
    expect(a.someMethod()).andReturn(0);
    replay(a);
    foo.execute();
}

If the object under testing needs to have a constructor mocked out, i.e.

public class Foo
{
    public Foo() { new File("c:\\boot.ini").delete() }

    public void methodToTest() { }
}

Then jMockit will replace non-default constructors. I don't know of anything that will replace a default constructor.

Enums

As far as I can tell, there is no facility in EasyMock, EasyMock ClassExtension to mock out enum logic, as it is defined as final by the JVM. jMockit should be used here. And that's it. jMockit looks like it's branched out into integration tests as well, notably Hibernate. I know that Unitils is a competitor here, but haven't played with either of them that much.

Toodledo

Mon, 11 Feb 2008 00:40:00 -0600

I've stopped using Ecco Pro and Shadowplan, and I'm now using Toodledo.

I looked some other online systems, but Toodledo was the only one that had both state (well, folders) AND context. Since Toodledo is a web based task system was that I'd be unable to add to it when I wasn't in front of a computer. However, with the integrations to Twitter and Jott, I can SMS or even phone up and leave messages.

Once I found out there was an API as well, it was really only a matter of time before I'd find myself writing an adapter for it. And a command line client, in case you want to access it from a shell.

So here's my first Ruby gem.

http://toodledo.rubyforge.org

If you're on a mac, you should be able to type 'sudo gem install toodledo' and then 'toodledo setup' and that should walk you through most of it.

Mocking ActiveRecord Associations

Fri, 09 Nov 2007 23:29:00 -0600

Flexmock does mocking of ActiveRecord objects if you ask it nicely, but it doesn't say much about associations.

It turns out that ActiveRecord does the same thing that Hibernate does and uses an internal proxy. So if you want a post containing a single image as an association, do this:

# Create a single image.
image = flexmock(:model, Image) { |mock|
  mock.should_receive(:url).and_return('/post/10/blah.jpg')
  mock.should_receive(:name).and_return('Blah Description')      
}

# It's an association proxy pretending to be an array.
images = flexmock(ActiveRecord::Associations::HasManyAssociation) { |mock|
  mock.should_receive(:find_by_name).and_return(image)
}

# The post itself.
post = flexmock(:model, Post) { |mock|
  mock.should_receive(:object_id).and_return('10')
  mock.should_receive(:images).and_return(images)
}

This is a Law of Demeter violation, so the better solution may be to have a get_images method.

More info here:

Misunderstanding the Law of Demeter
Demeter's Revenger
Loose Coupling takes Tight Logic

Rails Unit Test Confusion

Fri, 09 Nov 2007 20:28:00 -0600

I think I understand why the unit tests in Rails were confusing me. I had assuming that unit test fixtures ran YAML into some kind of in-memory activerecord object. But in reality, fixtures will put data into a test database and then unit tests will interact with it.

As far as I understand it:

1) An integration test is one where the test code goes outside the realm of the class and has a dependency on an external entity.
2) Databases are external entities.
3) Fixtures will write to the test database.

Therefore:

Any test containing a fixture is an integration test, even if it's in test/unit and extends Test::Unit::TestCase.

Furthermore:

Unit tests are tests that can be run in isolation, in any order, without any external dependencies.

Whew. Had to get that out of my system.

Not that I don't understand the rationale; or that using fixtures could be extremely handy. But a spade is a spade.

As usual, as soon as I realized the underlying issue, there was no shortage of help, Jay Fields being the most helpful.

Starting from Scratch

Mon, 05 Nov 2007 00:35:00 -0600

A little over a week ago, I made the appropriate sacrifices, made a backup to an external machine, setup a new computer (after figuring out why it wouldn't POST with a USB keyboard plugged into it) and then transferred the hard drives of my old computer into the new one.

Long story short, the hard drive with all my important data died, and the backup contains some hairball that causes the backup server to freeze when the data is transferred. Oh, and my laptop died as well. The same day.

Ever get the feeling the ground has just crumbled under your feet?

I lost some stuff. Most annoyingly, I lost all my software registration keys. This is good in a way -- it means I can try mint.com without having to worry about my old data. And I've found that I tend to use online services such as Google Docs and Google Bookmarks over any local solution, so in some cases there is no need to backup at all.

I spent most of the beginning of the week reinstalling my toolkit: Perl, Java, Ruby, RSIGuard, jEdit, etc. And making sure this can never happen again. I have everything local (and this server) backed up onto Amazon S3, using s3sync. I also have a weekly backup of the database onto S3, and I've already restored it to check it's still good.

Of course, it's going to take me three full days to upload my MP3s onto S3, but I think it's going to be worth it. And eventually, iTunes will get smart enough that I'll never need to download them again.

TMI

Thu, 25 Oct 2007 23:29:00 -0500

A problem I have learning Ruby on Rails is that there is such an active community base that I can't simply pick up what I need to know without getting sucked in.

For example. I want more integration tests. So I need fixtures to plop data into the test database. Fixtures are written in YAML.

Well, all my data's in the development database. I suppose I can hunt through Google. Ah. I can export it to YAML.

And there's a plugin that will allow YAML to be created with fewer foreign keys. Okay. I can use that then.

And the fixtures aren't run in dependent order. Okay.

Oh, there's another way to do do that. Okay. I'll use that instead.

And Rails doesn't tear down fixture data. Okay...

By the time I've read through everything in Google, I've forgotten what the heck I was trying to do in the first place. It doesn't help that some of the blog posts date back to the point where it's part of the core Rails package now -- I have no ability to distinguish between old and new.

So. Start at the beginning again. I want more integration tests. Step 1: Ignore Google.