Core Data Migration Woes with Binary Data and External Storage == Data Loss

I’m currently on a flight to San Francisco to attend Apple’s WWDC 2012 Conference – with 10 hours of time to kill I thought I’d spend a few hours writing a post about a CoreData bug I discovered recently. I’m also planning on showing the bug to Apple engineers next week so with a bit of luck this issue may get fixed in iOS 6. Fingers crossed.

Apple’s Binary Data attribute type

In iOS 5 Apple introduced a great new Core Data attribute type, Binary Data. This was a great new attribute – prior to iOS 5 it was typical to manage and store images and other binary files to disk outside of Core Data and store references to the files from Core Data entities – not ideal.

Use Case Scenario

A lot of the apps I build make extensive use of caching images so that users of my apps can continue to use my apps even when they’re offline. A good example of this is my latest iPad App, Portfolio Pro. Portfolio Pro is an app for Photographers and Designers to import their photos and videos into the app and then be able to present those binary files to clients in a coffee shop for example. The images need to be cached by the app for offline use.

The Problem

I’ve been using Apple’s Binary Data attribute type to store both the large photographs imported by users of my app and thumbnails for the photos. I’ve updated the app a few times without experiencing any problems accessing the previously cached Core Data binary attributes. Until that is, I tried to attempt a very simple automated Core Data model migration in a recent update. A strange bug occurred. Cached Core Data binary data started disappearing!

What was odd was that all of the cached thumbnails remained after an automated migration but the larger binary data attributes containing the original large sized photographs (2048 pixels long side) became nullified.

I was using exactly the same approach to store the thumbnail binary data and the large photo binary data. So why was one migrating and the other not? In both cases in addition to setting the Core Data attribute type to Binary Data I’d also selected Allows External Storage option. It seemed like the sensible choice, as I’d expect an optimized database framework to read/write large binary files to disk.

Allows External Storage

While investigating the migration issue I was experiencing in Portfolio Pro I discovered that Core Data writes large binary data to disk but not smaller binary data. Exactly what size it uses as the cut-off limit I don’t know but with large photo data I found that Core Data had created external binary files on disk in a subfolder of my app’s documents folder named “_EXTERNAL_DATA” within another folder named “.[MyProjectName]_SUPPORT”. So if your Xcode target is named MyCoolApp then the path to Core Data’s external storage will be [DocumentsFolder]/.MyCoolApp_SUPPORT/_EXTERNAL_DATA

Goodbye External Storage

Once I’d discovered this hidden storage folder I then found the cause of my disappearing photos. When Core Data performs an automated model migration it deletes/resets the _EXTERNAL_DATA folder – goodbye photos! All of my entities were being migrated just fine but the large, externally stored binary attributes were just disappearing because the storage directory was getting nuked in the migration process.

So that’s the bug with migrating a CoreData model that uses Binary Data attributes with Allows External Storage switched on.

See for yourself

I’ve put together an example project to demonstrate the bug. Download the source files here:

Download Source Files

Begin by opening up the Xcode project within the Before Migration folder. Run the project in the iOS Simulator. You should get similar results to figure 1.

Figure 1

Figure 1: Before CoreData Migration

The 2 image views are being populated by a single Core Data entity. The entity contains 2 identical Binary Data attribute types. 1 named smallImage and one named largeImage. Both have been set up to Allow External Storage. In the view I’m rendering the smallImage binary data into the first image view and the largeImage binary data into the second image view. Both image views have content mode set to aspect fit.

Close the Before Migration version of the project. Now open After Migration (Bug) version of the same project. The only difference between this version of the project and the first is that I’ve migrated the CoreData model by adding a model version and adding a new unused string attribute named newAttribute to the Core Data Entity.

The project is already set-up correctly for automatic Core Data migration. Build and run. This should replace your previous version of the example app and perform the automatic model migration. But what happens to the binary data attributes of our entity? Here’s what happens, the small image remains and the large image is deleted (see figure 2)!

Figure 2: After CoreData Migration

Figure 2: After CoreData Migration

Ok, point proven. Delete the demo app from your iOS Simulator. Run the original Before Migration version once more – you should now have both images displayed as before. Now for the solution…

The Solution

Now build and run the third version of the example app within the After Migration (Solution) folder. You should still have 2 images displayed after migration – hurray!

The solution I’ve come up with is when your code initializes a persistent store coordinator for your Core Data model run a few checks before attempting automatic migration. Check whether the new model is compatible with the current stored model. If it’s not then you know that Core Data is about to migrate your old model to your new version and in doing so will wipe the external storage folder. Before it does so simply move the external storage folder to a temporary location. Once the migration has completed replace new empty external storage folder generated by Core Data. Here’s the code that you’ll also find in the Model class of the example project within the After Migration (Solution) folder:

- (NSPersistentStoreCoordinator *)persistentStoreCoordinator

{

if (_persistentStoreCoordinator != nil) {

return _persistentStoreCoordinator;

}

NSURL *storeURL = [[self applicationDocumentsDirectory] URLByAppendingPathComponent:@"CoreDataBinaryBug.sqlite"];

NSError *error = nil;

NSDictionary *sourceMetadata = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType:NSSQLiteStoreType

URL:storeURL

error:&error];

//Check if the new model is compatible with any previously stored model

BOOL isCompatibile = [self.managedObjectModel isConfiguration:nil compatibleWithStoreMetadata:sourceMetadata];

BOOL needsMigration = !isCompatibile;

NSFileManager *fileManager = [NSFileManager defaultManager];

//Prepare a temporary path to move CoreData's external data storage folder to if automatic model migration is required

NSString *documentsPath = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) lastObject];

NSString *tmpPathToExternalStorage = [documentsPath stringByAppendingPathComponent:@"tmpPathToReplacementData"];

NSString *pathToExternalStorage = [documentsPath stringByAppendingPathComponent:@".CoreDataBinaryBug_SUPPORT/_EXTERNAL_DATA"];

if (needsMigration) {

if ([fileManager fileExistsAtPath:pathToExternalStorage]) {

//Move Apple's CoreData external storage folder before it's nuked by the migration bug

[fileManager moveItemAtPath:pathToExternalStorage toPath:tmpPathToExternalStorage error:nil];

}

}

NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:

[NSNumber numberWithBool:YES],NSMigratePersistentStoresAutomaticallyOption,

[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];

_persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];

if (![_persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeURL options:options error:&error]) {

 

NSLog(@"Unresolved error %@, %@", error, [error userInfo]);

abort();

}

else {

if (needsMigration) {

//Apple's automatic migration is now complete. Replace the default external storage folder with the version pre upgrade

[[NSFileManager defaultManager] removeItemAtPath:pathToExternalStorage error:nil];

[[NSFileManager defaultManager] moveItemAtPath:tmpPathToExternalStorage toPath:pathToExternalStorage error:nil];

}

}

return _persistentStoreCoordinator;

}

Once more, here are the source files for the example project:

Download Source Files

Interested in iOS Development and Core Data? Follow me on Twitter for more tidbits :-)

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon

Speaker at the next Brighton iPhone Creators Meetup

As per the title, I’m going to be speaking about iPad Development at the next Brighton iPhone Creators Meetup. Here are a few details about the event:

Presenting: iPad Development: New Challenges and Opportunities for iPhone Devs

When: Tuesday 21st @ 7pm

Where: The Skiff, 6 Gloucester Street, Brighton, BN1 4EW

I’ve been primarily asked to talk about iPad development so that’s going to be the main feature in my 20/30 min presentation. How iPad development creates new challenges (User Experience, Development for multiple iDevices etc) and opportunities for iPhone Devs (increased sales, AppStore rankings).

I’m also planning to share some of my experiences of developing for the AppStore such as tips on how to improve your app reviews and ratings and how to avoid some of the mistakes I’ve already made :-)

That’s it for now… Gotta get on with my latest app idea :-)

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon

Portfolio to Go for Flickr

Portfolio to Go for Flickr has just been approved for sale by Apple and is now availabile for iPad on the App Store.

A recent winner of Apple’s ‘iPad App of the Week’ award, Portfolio To Go features an elegant and intuitive photo-wall giving users instant access to all of their Flickr photosets: scroll each gallery horizontally to browse thumbs, scroll vertically to traverse through all synced galleries.
The main purpose of Portfolio To Go is to enable photographers to present their portfolio offline to clients – perfect for use with iPad wi-fi: just sync and go. Photographers create and edit photosets on Flickr.com and Portfolio To Go acts as a presentation tool, keeping in sync with all your Flickr changes.

Portfolio To Go supports all iPad rotations so can be flipped vertically for portraits and horizontally for landscapes.

Another unique selling point of this app is the ‘Send Portfolio to a Client’ feature. The Photographer picks and chooses which of his galleries to share and can then send an auto-generated email to clients and friends containing a free download link to ‘Portfolio To Go Player’ iPad App. By clicking the emailed link clients launch the photographer’s portfolio on their own iPad(s).

[flickr album=72157624609630073 num=30 size=Small]

Check out this great video review of Portfolio To Go on iPad. Mark Wallace from Adorama TV takes you through most of the key features of the app :-)

Key Features

# Photo Wall:
View all your Flickr photo galleries at once – scroll each gallery horizontally to browse thumbs, scroll vertically to traverse through all synced galleries. Click on any thumbnail to jump into the main image view.

# Main image view displays gallery thumbnails to enable intuitive gallery navigation. Just click the main image to go full-screen.

# Flickr Authorization: access to all of your private, friends and family and public photos

# Multiple cached portfolios: add unlimited Flickr portfolios via your contacts or Flickr IDs to P2G and switch between them.

# Auto-cache photos: all photos get cached in the background while you browse (you can turn this feature off in settings)

# Send Portfolio to a Client: Pick and choose which galleries to include and then send your portfolio straight to clients and friends by email from the app. Clients will download the free Portfolio to Go Player app to their iPad and launch your portfolio from the emailed link.

# Share your photos and Flickr links via Facebook, Twitter or Email

# Save favourite Photos to your iPad Photo Library

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon

Portfolio To Go goes Pro!

Portfolio to Go for Flickr has just been approved for sale by Apple and is now availabile for iPad on the App Store.

Following the success of Portfolio To Go Lite – a top 10 free iPad App for Photographers – this pro version adds many new features including:

  • Photo Wall:
    View all your Flickr photo galleries at once – scroll each gallery horizontally to browse thumbs, scroll vertically to traverse through all synced galleries. Click on any thumbnail to jump into the main image view.
  • Main image view displays gallery thumbnails to enable improved gallery navigation. Just click the main image to go full-screen.
  • Pick and choose which galleries to display and which to hide
  • Share your photos and Flickr links in app via Facebook, Twitter or Email

Apple has chosen to feature Portfolio to Go for Flickr in the new and noteworthy section of the iPad Photography category – they did the same for the Lite version :-)

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon