Navigation
Monday
Apr022012

Asynchronous, lazy initialization with synchronous accessor

I’ve come to love Grand Central Dispatch and blocks for making it so easy to add asynchronous tasks to your application. Without the overhead of thread class instantiation or defining callback methods you can send a task in the background and keep your main thread unblocked.

However, sometimes you need a mix of synchronous and asynchronous tasks or more specifically you want to start something asynchronously initially and to block and wait for its completion elsewhere in your code. One example of this could be a unit test of an asynchronous algorithm where you need synchronous access to the results for validation.

Another example is a current project of mine which involves plotting of medical data that is parsed from CSV files. There are four CSV files and each takes about a second to parse. It’s not long but when you try and do it on demand when a plot is about to be displayed on screen, you find that blocking your main thread for a second can be very annoying. The obvious solution is to do the parsing on a background queue but that immediately raises the question: How do you then handle the plotting? Do you show an empty plot which populates later, when the data is available? That doesn’t look good. Another alternative would be to make the whole display plot action asynchronous. But then you’ve decoupled user interaction (user taps a button to bring up a plot) and GUI action (plot actually displays) and will probably find that users tap multiple times until the plot shows up.

Ideally then, the data would be loaded early on and the actual plotting would be synchronous. In my application, the data is loaded and parsed asynchronously in the initializer of a singleton which is used throughout the application for global data. Therefore, as soon as my global is being accessed for the first time the data gets loaded in the background. I can then afford to use blocking access to the data, because there is no (or very little) chance for the user to activate the GUI to display the plot before the data has been parsed. And even if they do, the processing is far along and the delay minimal.

So in summary, the requirements for my use case are:

  • Several initialization tasks need processing
  • Processing can happen in parallel
  • Processing must not block the main thread
  • Access to the results should block while processing is in progress

Here is how it’s implemented:

First off, we have an initializer that does our parsing, slowInitForKey in the example code. The idea here is that initialization work is based on a key (e.g. a filename) and returns a single result object that can be stored in a results dictionary.

Next we define a singleton Globals which is instantiated early on in our code, for example in viewDidLoad and has the following init method:

- (id)init {
  self = [super init];
  if (self) {
    valuesSerialQueue = dispatch_queue_create("valuesSerialQueue", NULL);
    self.values = [NSMutableDictionary dictionary];

    [[NSArray arrayWithObjects:@"A", @"B", @"C", @"D", @"E", @"F", nil]
     enumerateObjectsUsingBlock:^(id obj, NSUInteger idx, BOOL *stop) {
      dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
        NSString *value = [self slowInitForKey:obj];
        dispatch_async(valuesSerialQueue, ^{
          [self.values setObject:value forKey:obj];
        });
      });
    }];
  }
  return self;
}

What does this do?

  • First, we set up a serial queue and a dictionary for the results. We use a serial queue to make sure that only one thread will access the values dictionary at a time. Think of it as a locking mechanism in GCD terms.
  • Next, we iterate over the initialization keys (A-F in this example – these would be filenames in the CSV parsing example). Each key we send to a concurrent dispatch queue for parallel processing of slowInitForKey:. After processing is finished, the result is written to the dictionary via an async dispatch to our serial queue valuesSerialQueue. Again, this ensures that no two threads access the values dictionary at the same time.

Now that initialization is on its way, all that’s left is the synchronous access to the results. This is pretty simple:

- (NSString *)valueForKey:(NSString *)key {
  __block NSString *result = nil;
  do {
    // keep polling until there’s a value
    dispatch_sync(valuesSerialQueue, ^{
      result = [self.values objectForKey:key];
    });
  } while (result == nil);
  return result;
}

All we do is simply poll the results dictionary via the serial queue until there is a value. Of course you need to make sure that your initializer will always set a result – otherwise you would block forever. A safer way would be to set a time limit on how long you block before you eventually break from this method and return nil.

If you are worried that there may be a lot of polling going on until there is a result you could add a little delay after each unsuccessful poll. It’s probably irrelevant though, because polling only happens until initialization is finished.

An example project is available on github.

 

Tuesday
Feb072012

iOS User Accounts

Wouldn’t it be convenient if you could pick up any iPhone or iPad and have it personalized with your settings quickly? This is something that occurred to me last week when my girl friend had left her iPhone at home and wanted to continue reading her book in iBooks. I had my iPad with me but of course it is tied to my iTunes account, not hers, and it’s way too much hassle to reconfigure it just for a brief reading session.

But it made me wonder what that feature could look like on iOS and what it would take to make it happen. Basically, you’d want an extension of something that’s already possible on OSX: signing in with an Apple-ID. Once you’re authenticated with your Apple-ID, your content and settings are only a few steps away: iCloud, if you’re using it, has got it and in theory, that’s all you need to restore your device.

I’ve upgraded quite a few devices in the past and so far backup and restore has worked really well. Now imagine there were an (optional) login screen on iOS devices where you could log in to your iCloud account and immediately you’d get your home screen, with your content and settings trickling in in the background - just like it’s happening now when you restore through iTunes or from iCloud. With future devices having more storage space, the OS could cache multiple user accounts so that on subsequent logins your data would only need an update rather than a completely fresh pull. Also, you can imagine some things like big apps being referenced from multiple accounts and therefore needing to be stored only once on a device and not per account.

If that use-case still sounds esoteric to you, because your iPad is yours alone, think about places where iPads could be shared by larger audiences: Schools, universities, sales people, etc. For example, if a school wanted to start using iPads in one course only, say their biology class, they’d only need to get enough iPads for their maximum class size, not for the total number of students attending that class. (Caveat: no iPad based learning at home unless students log in using their private iPad.) Or there could be iPads per course that wouldn’t need to be moved: Your course material appears at your desk wherever you are - you don’t actually carry it there anymore. It would certainly help reduce the risk of iPads being dropped between classes or on the bus.

Technically, I would assume something like that being investigated or even in place already at Apple. It’s probably just a matter of broadband connections catching up to make this a smooth experience. One that Apple would be willing to ship and tout as a new feature.

Monday
Jan022012

autotm 0.94 supports local backups

As introduced in a previous blog post, autotm is an OSX system daemon that automatically switches Time Machine targets depending on their availability. The initial version of autotm only supported network based targets but I’ve recently updated the script to also allow locally connected disks (e.g. USB). This update requires some minor changes to your autotm.conf file: The server section is now called “destinations” and each destination has a “type”, which can be remote or local. For example:

destinations:
 - type: remote
   hostname: myhomeserver.local
   username: jdoe
   password: s3cr3t
 - type: remote
   hostname: myofficeserver.local
   username: john_doe
   password: pa55
 - type: local
   volume: /Volumes/Time Machine

To learn more about autotm, have a look at the Readme on Github. Please file any problems you encounter on the issue tracker at github.

Thanks to Andy and Daniel for their help in testing this release!

Monday
Dec192011

CouchDb Migrations

A few weeks ago I attended CouchConf in Berlin and during the sessions (and in between) one topic was raised several times: How to migrate data between “schemas” or document versions. I described how we are migrating documents and I want to take a moment to explain the process in more detail. It might sound trivial but there was interest in the description during the conference, so I’m hoping it may prove helpful for others nonetheless.

Since CouchDb is inherently unstructured, there’s no global schema that you manage to control your data’s structure. That’s often a good thing, because it gives you flexibility, but it can also cause problems, for example when you want to access documents without handling against all sorts of different “versions” of your document you might have.

For example, say you have started out with an initial player document (we’re sticking with the RPG theme set in the Couchbase examples ;)):

{
  'version' : 1,
  'name:' : 'Player A',
  'xp': 1234
}

but you find during testing that you need to know a player’s level. You’ve decided that the level should always be xp/100 + 1 but you don’t want to recompute this all the time in code but rather store it in the document directly. For various other reasons you’ve also decided against creating a view and therefore you want to migrate all your documents to this format:

{
  'version' : 2,
  'name:' : 'Player A',
  'xp' : 1234,
  'level' : 13
}

Note that the initial document already included a version attribute that we’re using to keep track of our migrations but even if this weren’t the case from the start, it’s easy to simply treat documents without a version attribute as “version 0” so to speak and handle them similarly to the rest of this example.

So how do we migrate from version 1 to version 2 then?

The idea is to create a view that shows all old revision documents and process them until the view has no more items. The view would be defined with the following (trivial) map function:

function(doc) {
  if (doc.version && doc.version == 1) {
    emit(doc._id);
  }
}

Now it’s simply a matter of processing all items in this view, for example with the following python-couchdb method that takes a database object as a parameter:

def migrate_v1_v2(db):
  v1 = db.view('_design/migration/_view/v1')
  for row in v1.rows:
    doc = db[row.key]
    if doc['version'] == 1:
      doc['version'] = 2
      # we want to add the level stat, which is simply xp/100, starting from 1
      doc['level'] = doc['xp']/100 + 1
      db[doc.id] = doc

and where “v1” is the name of the view we defined above.

The complete example in the form of a unit test is available on github. The only dependency is python-couchdb. It should be trivial to translate this pattern to other client libraries. It might also be useful to extend this concept to a migration framework á la Ruby on Rails.

Monday
Dec122011

Using S/MIME on iOS Devices

The following article explains how to set up your iPhone or iPad to send and receive encrypted emails via S/MIME. Prerequisite is an S/MIME certificate from a certificate authority. Some CAs provide them free for personal use. The procedure is not very complicated even though the description may look lengthy due to the many screenshots. The biggest hurdle is to pick the correct file format when exporting your S/MIME key on your Mac. (A description on how to export the correct certificate on Windows will follow.)

Set-up for Receiving Encrypted Emails

1. Export your private key in a format that you can import on your iOS devices.

To do this, open “Keychain Access” and find your certificate. Select it and choose “File” / “Export Items”, as shown below.

01 export key

2. Next, save the certificate in p12 format.

In the process of saving the certificate, as detailed below, you will be asked to provide a password to encrypt your key. This will allow you to send it via email without fear of it being intercepted and used by someone else. Depending on your keychain settings you will also be asked to provide your administrator password to read the privatekey for exporting.

02 save p12

3. Now drag this exported file to your Mail.app icon to send it to yourself.

(Make sure you don’t encrypt it ;)

03 send key

4. Turn to your iOS device to import the certificate.

Open the email you just sent to yourself and tap on the attachment to import your certificate.

04 import on ios 05 unsigned certificate 06 enter password 

5. Enable S/MIME in advanced mail settings and choose your certificate.

On your iOS device go to “Settings” / “Mail, Contacts, Calendars” / “<Your Account>” / “Advanced” (at the very bottom of your account settings) and activate S/MIME. Important: Make sure you leave the account settings by tapping “done” in the top right of the tool bar. Changes don’t appear to be applied until you do so.

07 enable smime 07b confirm settings

You can also enable signing and encrypting of messages here but more on that in a moment. What we’ve achieved so far is being able to read messages that have been signed with our public key. Unfortunately, sending encrypted messages involves a few more steps and has a few caveats.

Set-up for Sending Encrypted Emails

In order to send an encrypted message, you need to do the following.

1. Import the recipient’s public key.

This happens automatically in Mail.app on OSX but requires some manual interaction on iOS. You may have noticed when looking at signed messages (like the one you sent yourself earlier) that there’s a new little star icon in the blue email address bubble after S/MIME has been activated. This is the UI indicator for signed messages. And the address bubble is also a button that you can tap to bring up address - and certificate - information.

08 address bubble star

Tapping this button will bring up the address info view:

09 address info

Tap install to register this public key, which will allow you to send encrypted emails to the key’s owner. You will need to repeat this procedure once for every recipient.

2. Send email.

There’s not really a step two other than making sure you’re sending to the recipient’s correct email address and from your correct account so that the available keys match up with the email addresses used in the process. You can tell that your message is being encrypted by the “Encrypted” string in the title bar of your message:

10 encrypted message

Caveats

What’s a bit unfortunate is that there’s no easy way to selectively send encrypted emails. The encryption setting is global for the account under “Settings”, meaning that you have to go there and enable/disable encryption for all messages from that account. It would be nice if that were the default only, with an option to override it in the message composition view.

It would also be nice if public key importing were automatic, like it is on the Mac.

But all in all, it’s nice to be able to read encrypted emails on iOS devices now.