Core Data Migration Woes with Binary Data and External Storage == Data Loss

I’m currently on a flight to San Francisco to attend Apple’s WWDC 2012 Conference – with 10 hours of time to kill I thought I’d spend a few hours writing a post about a CoreData bug I discovered recently. I’m also planning on showing the bug to Apple engineers next week so with a bit of luck this issue may get fixed in iOS 6. Fingers crossed.

Apple’s Binary Data attribute type

In iOS 5 Apple introduced a great new Core Data attribute type, Binary Data. This was a great new attribute – prior to iOS 5 it was typical to manage and store images and other binary files to disk outside of Core Data and store references to the files from Core Data entities – not ideal.

Use Case Scenario

A lot of the apps I build make extensive use of caching images so that users of my apps can continue to use my apps even when they’re offline. A good example of this is my latest iPad App, Portfolio Pro. Portfolio Pro is an app for Photographers and Designers to import their photos and videos into the app and then be able to present those binary files to clients in a coffee shop for example. The images need to be cached by the app for offline use.

The Problem

I’ve been using Apple’s Binary Data attribute type to store both the large photographs imported by users of my app and thumbnails for the photos. I’ve updated the app a few times without experiencing any problems accessing the previously cached Core Data binary attributes. Until that is, I tried to attempt a very simple automated Core Data model migration in a recent update. A strange bug occurred. Cached Core Data binary data started disappearing!

What was odd was that all of the cached thumbnails remained after an automated migration but the larger binary data attributes containing the original large sized photographs (2048 pixels long side) became nullified.

I was using exactly the same approach to store the thumbnail binary data and the large photo binary data. So why was one migrating and the other not? In both cases in addition to setting the Core Data attribute type to Binary Data I’d also selected Allows External Storage option. It seemed like the sensible choice, as I’d expect an optimized database framework to read/write large binary files to disk.

Allows External Storage

While investigating the migration issue I was experiencing in Portfolio Pro I discovered that Core Data writes large binary data to disk but not smaller binary data. Exactly what size it uses as the cut-off limit I don’t know but with large photo data I found that Core Data had created external binary files on disk in a subfolder of my app’s documents folder named “_EXTERNAL_DATA” within another folder named “.[MyProjectName]_SUPPORT”. So if your Xcode target is named MyCoolApp then the path to Core Data’s external storage will be [DocumentsFolder]/.MyCoolApp_SUPPORT/_EXTERNAL_DATA

Goodbye External Storage

Once I’d discovered this hidden storage folder I then found the cause of my disappearing photos. When Core Data performs an automated model migration it deletes/resets the _EXTERNAL_DATA folder – goodbye photos! All of my entities were being migrated just fine but the large, externally stored binary attributes were just disappearing because the storage directory was getting nuked in the migration process.

So that’s the bug with migrating a CoreData model that uses Binary Data attributes with Allows External Storage switched on.

See for yourself

I’ve put together an example project to demonstrate the bug. Download the source files here:

Download Source Files

Begin by opening up the Xcode project within the Before Migration folder. Run the project in the iOS Simulator. You should get similar results to figure 1.

Figure 1

Figure 1: Before CoreData Migration

The 2 image views are being populated by a single Core Data entity. The entity contains 2 identical Binary Data attribute types. 1 named smallImage and one named largeImage. Both have been set up to Allow External Storage. In the view I’m rendering the smallImage binary data into the first image view and the largeImage binary data into the second image view. Both image views have content mode set to aspect fit.

Close the Before Migration version of the project. Now open After Migration (Bug) version of the same project. The only difference between this version of the project and the first is that I’ve migrated the CoreData model by adding a model version and adding a new unused string attribute named newAttribute to the Core Data Entity.

The project is already set-up correctly for automatic Core Data migration. Build and run. This should replace your previous version of the example app and perform the automatic model migration. But what happens to the binary data attributes of our entity? Here’s what happens, the small image remains and the large image is deleted (see figure 2)!

Figure 2: After CoreData Migration

Figure 2: After CoreData Migration

Ok, point proven. Delete the demo app from your iOS Simulator. Run the original Before Migration version once more – you should now have both images displayed as before. Now for the solution…

The Solution

Now build and run the third version of the example app within the After Migration (Solution) folder. You should still have 2 images displayed after migration – hurray!

The solution I’ve come up with is when your code initializes a persistent store coordinator for your Core Data model run a few checks before attempting automatic migration. Check whether the new model is compatible with the current stored model. If it’s not then you know that Core Data is about to migrate your old model to your new version and in doing so will wipe the external storage folder. Before it does so simply move the external storage folder to a temporary location. Once the migration has completed replace new empty external storage folder generated by Core Data. Here’s the code that you’ll also find in the Model class of the example project within the After Migration (Solution) folder:

- (NSPersistentStoreCoordinator *)persistentStoreCoordinator

{

if (_persistentStoreCoordinator != nil) {

return _persistentStoreCoordinator;

}

NSURL *storeURL = [[self applicationDocumentsDirectory] URLByAppendingPathComponent:@"CoreDataBinaryBug.sqlite"];

NSError *error = nil;

NSDictionary *sourceMetadata = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType:NSSQLiteStoreType

URL:storeURL

error:&error];

//Check if the new model is compatible with any previously stored model

BOOL isCompatibile = [self.managedObjectModel isConfiguration:nil compatibleWithStoreMetadata:sourceMetadata];

BOOL needsMigration = !isCompatibile;

NSFileManager *fileManager = [NSFileManager defaultManager];

//Prepare a temporary path to move CoreData's external data storage folder to if automatic model migration is required

NSString *documentsPath = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) lastObject];

NSString *tmpPathToExternalStorage = [documentsPath stringByAppendingPathComponent:@"tmpPathToReplacementData"];

NSString *pathToExternalStorage = [documentsPath stringByAppendingPathComponent:@".CoreDataBinaryBug_SUPPORT/_EXTERNAL_DATA"];

if (needsMigration) {

if ([fileManager fileExistsAtPath:pathToExternalStorage]) {

//Move Apple's CoreData external storage folder before it's nuked by the migration bug

[fileManager moveItemAtPath:pathToExternalStorage toPath:tmpPathToExternalStorage error:nil];

}

}

NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:

[NSNumber numberWithBool:YES],NSMigratePersistentStoresAutomaticallyOption,

[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];

_persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];

if (![_persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeURL options:options error:&error]) {

 

NSLog(@"Unresolved error %@, %@", error, [error userInfo]);

abort();

}

else {

if (needsMigration) {

//Apple's automatic migration is now complete. Replace the default external storage folder with the version pre upgrade

[[NSFileManager defaultManager] removeItemAtPath:pathToExternalStorage error:nil];

[[NSFileManager defaultManager] moveItemAtPath:tmpPathToExternalStorage toPath:pathToExternalStorage error:nil];

}

}

return _persistentStoreCoordinator;

}

Once more, here are the source files for the example project:

Download Source Files

Interested in iOS Development and Core Data? Follow me on Twitter for more tidbits :-)

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon

Speaker at the next Brighton iPhone Creators Meetup

As per the title, I’m going to be speaking about iPad Development at the next Brighton iPhone Creators Meetup. Here are a few details about the event:

Presenting: iPad Development: New Challenges and Opportunities for iPhone Devs

When: Tuesday 21st @ 7pm

Where: The Skiff, 6 Gloucester Street, Brighton, BN1 4EW

I’ve been primarily asked to talk about iPad development so that’s going to be the main feature in my 20/30 min presentation. How iPad development creates new challenges (User Experience, Development for multiple iDevices etc) and opportunities for iPhone Devs (increased sales, AppStore rankings).

I’m also planning to share some of my experiences of developing for the AppStore such as tips on how to improve your app reviews and ratings and how to avoid some of the mistakes I’ve already made :-)

That’s it for now… Gotta get on with my latest app idea :-)

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon

Portfolio To Go goes Mini!

Portfolio to Go Mini has just been approved for sale by Apple and is now availabile on iPhone and iPod from the App Store!.

Following the success of the iPad version and winning Apple’s prestigious App of the Week award, Portfolio To Go goes mini and arrives on your iPhone and iPod touch!

Portfolio to Go syncs your Flickr photosets to local, cached galleries on your iPhone or iPod Touch. Once all galleries and images are cached you can view your portfolio without an internet connection. Perfect for use with wi-fi: sync and go.

Portfolio To Go is elegant and intuitive, giving users instant access to all of their Flickr photosets. The unique interface design presents thumbnail images horizontally and vertically on a photo-wall capable of displaying thousands of images simultaneously that scroll at the touch of a finger.

Portfolio To Go supports all device rotations so can be flipped vertically for portraits and horizontally for landscapes.

Another unique selling point of this app is the ‘Send Portfolio to a Client’ feature. The Photographer picks and chooses which of his galleries to share and can then send an auto-generated email to clients and friends containing a free download link to ‘Portfolio To Go Player’ iPad/iPhone App. By clicking the emailed link clients launch the photographer’s portfolio on their own iPad/iPhone/iPod Touch.

Top

Screen Grabs

[flickr album=72157624869114732 num=30 size=Medium]

Top

Key Features

# Photo Wall:
View all your Flickr photo galleries at once – scroll each gallery horizontally to browse thumbs, scroll vertically to traverse through all synced galleries. Click on any thumbnail to jump into the main image view.

# Main image view displays gallery thumbnails to enable intuitive gallery navigation. Just click the main image to go full-screen.

# Flickr Authorization: access to all of your private, friends and family and public photos

# Multiple cached portfolios: add unlimited Flickr portfolios via your contacts or Flickr IDs to P2G and switch between them.

# Auto-cache photos: all photos get cached in the background while you browse (you can turn this feature off in settings)

# Send Portfolio to a Client: Pick and choose which galleries to include and then send your portfolio straight to clients and friends by email from the app. Clients will download the free Portfolio to Go Player app to their iPad/iPhone/iPod Touch and launch your portfolio from the emailed link.

# Share your photos and Flickr links via Facebook, Twitter or Email

# Save favourite Photos to your iPhone/iPod Photo Library

Top


Portfolio To Go Player

You can try out the Portfolio To Go Player experience for free (just like your clients and friends) by following 2 simple steps on your iPad, iPhone or iPod Touch:
1) Click and download this free iPad app: Portfolio To Go Player
2) Click and launch this link (it will open in the downloaded app): Nicole Carman’s Portfolio

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon

W.A.Ellis Property: iPhone App

Built for Property specialist digital agency World Archipelago, I was supplied with the customized design and commissioned to build a reskinnable iPhone property app that can be added to WAI’s range of online solutions for their extensive range of Real Estate Agencies.

Here’s a live version for Estate Agency Halls:

The app includes the following features:

  • Auto-paginated property search
  • User-location finding for local property searches
  • Auto-saves recent search requests
  • Saved property list between sessions
  • Property location display on a map
  • In app email property to a friend
  • Property details image gallery
  • Cached thumbnails

With additional functionality such as previous/next property buttons directly from within each property page this app is a great solution to most estate agency requirements. A robust, optimised and elegant-looking iPhone property app.

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon

Introducing Loop&Learn!

After 5 months ‘ad hoc’ development and a short but anxious 6  day wait in the Apple submission queue we have finally launched our ‘Social Learning’ App for iPhone, iPad and iPod Touch: Loop&Learn.

Loop&Learn is more than just ‘my best ever iPhone App’, it is my best ever app… desktop, web, mobile or otherwise. It is both technically and content-wise the coolest software I’ve been lucky enough to develop.

Learn Chinese Phrasesloop-study-birds
Whether you want to learn Chinese or are an avid bird watcher,  there’s a loop for that :-)

Here’s my business partner, Neil Gerstenberg, presenting our app on YouTube:

As Neil describes in the video, Loop&Learn is a tool to help you learn ANYTHING! Right on your iPhone you can create multimedia ‘Learning Loops’ consisting of recorded audio phrases, images and text. On iPhone and iPad the audio recording works directly from the device microphone. On iPod Touch you’ll need a compatible microphone earphones for authoring but all of the study and learning loop playback is compatible without any extra hardware.

Create

We’ve designed the authoring process to be as intuitive as possible with features such as:

  • body text is auto split into loop phrases based on line/paragraph breaks – so you can easily import large learning texts into the app
  • auto-record in the phrase editing screen detects silences and enables you to split up your multiple audio phrases without continuously clicking a record button
  • the full current paragraph text is instantly displayed when recording each phrase
  • while editing body text the app enables users to play the existing audio phrase associated with the current text cursor position

Loop&Learn AuthoringBody Text Editor
New/Edit Loop Home Screen and Body Text Editor

Loop&Learn AuthoringLoop&Learn Authoring
Phrase Record and Edit

Learn

Likewise, in study mode here are just a few handy learning features offered by our app:

  • multi-gesture control: double tap to start/stop playback, swipe left and right for next/prev phrases, swipe down and up for next/prev looping groups
  • single or multi-select the loop blocks at the top of the screen to split your learning into looping bite-size chunks
  • 2 learning modes: ‘Repeat with me’ and ‘Listen’n’Repeat’ – the latter plays the full loop group and then repeats silently giving you the chance to repeat aloud

Share

Saving the best for last, my favourite feature of Loop&Learn is the new collaborative learning concept it introduces: ‘social learning’.

Users of our app can share the learning loops they create via our ‘Loop Library’. All that’s required is an account with Twitter.  Once a user shares one or more of their learning loops they retain ownership of the loop and can change and update it at any time via the same share and update process.

Loop Libraryshare-on-twitter

Think of the amazing possibilities that social learning offers. Language learning for example: User A is French and wants to learn English, User B is English and wants to learn French. If User A and User B create and share language loops in their native tongues then they both will benefit from the shared content!

In anticipation of launch, Neil, myself and friends have created some example loops that demonstrate how versatile the app is. Take a look at the Featured Loops page on loopandlearn.com. You’ll find learning loops for:

Just a few of our examples to kick off the app. As more people purchase Loop&Learn we expect the content to grow and it will always remain free to download user generated loops so the loop library should get bigger and better :-)

Thanks to the power of Twitter when users share new learning loops Loop&Learn sends a tweet with a link to the newly created loop preview page on our site. Like this one for example: http://www.loopandlearn.com/9ddbb9

what’s really cool is that on the top right of that preview page is a download link which when clicked on by Loop&Learn users via Safari or most iPhone Twitter clients Loop&Learn will automatically:

  1. launch on the  iPhone
  2. download and install the loop being previewed

The Website

www.loopandlearn.com stays up-to-date automatically with the latest learning loops uploaded by Loop&Learn users. We have also launched the site with a ‘Hot Loops’ / Featured Loops section.

Happy Looping!

It's only fair to share...Tweet about this on TwitterShare on FacebookShare on Google+Share on LinkedInShare on StumbleUpon