I’m currently on a flight to San Francisco to attend Apple’s WWDC 2012 Conference – with 10 hours of time to kill I thought I’d spend a few hours writing a post about a CoreData bug I discovered recently. I’m also planning on showing the bug to Apple engineers next week so with a bit of luck this issue may get fixed in iOS 6. Fingers crossed.

Apple’s Binary Data attribute type

In iOS 5 Apple introduced a great new Core Data attribute type, Binary Data. This was a great new attribute – prior to iOS 5 it was typical to manage and store images and other binary files to disk outside of Core Data and store references to the files from Core Data entities – not ideal.

Use Case Scenario

A lot of the apps I build make extensive use of caching images so that users of my apps can continue to use my apps even when they’re offline. A good example of this is my latest iPad App, Portfolio Pro. Portfolio Pro is an app for Photographers and Designers to import their photos and videos into the app and then be able to present those binary files to clients in a coffee shop for example. The images need to be cached by the app for offline use.

The Problem

I’ve been using Apple’s Binary Data attribute type to store both the large photographs imported by users of my app and thumbnails for the photos. I’ve updated the app a few times without experiencing any problems accessing the previously cached Core Data binary attributes. Until that is, I tried to attempt a very simple automated Core Data model migration in a recent update. A strange bug occurred. Cached Core Data binary data started disappearing!

What was odd was that all of the cached thumbnails remained after an automated migration but the larger binary data attributes containing the original large sized photographs (2048 pixels long side) became nullified.

I was using exactly the same approach to store the thumbnail binary data and the large photo binary data. So why was one migrating and the other not? In both cases in addition to setting the Core Data attribute type to Binary Data I’d also selected Allows External Storage option. It seemed like the sensible choice, as I’d expect an optimized database framework to read/write large binary files to disk.

Allows External Storage

While investigating the migration issue I was experiencing in Portfolio Pro I discovered that Core Data writes large binary data to disk but not smaller binary data. Exactly what size it uses as the cut-off limit I don’t know but with large photo data I found that Core Data had created external binary files on disk in a subfolder of my app’s documents folder named “_EXTERNAL_DATA” within another folder named “.[MyProjectName]_SUPPORT”. So if your Xcode target is named MyCoolApp then the path to Core Data’s external storage will be [DocumentsFolder]/.MyCoolApp_SUPPORT/_EXTERNAL_DATA

Goodbye External Storage

Once I’d discovered this hidden storage folder I then found the cause of my disappearing photos. When Core Data performs an automated model migration it deletes/resets the _EXTERNAL_DATA folder – goodbye photos! All of my entities were being migrated just fine but the large, externally stored binary attributes were just disappearing because the storage directory was getting nuked in the migration process.

So that’s the bug with migrating a CoreData model that uses Binary Data attributes with Allows External Storage switched on.

See for yourself

I’ve put together an example project to demonstrate the bug. Download the source files here:

Download Source Files

Begin by opening up the Xcode project within the Before Migration folder. Run the project in the iOS Simulator. You should get similar results to figure 1.

Figure 1

Figure 1: Before CoreData Migration

The 2 image views are being populated by a single Core Data entity. The entity contains 2 identical Binary Data attribute types. 1 named smallImage and one named largeImage. Both have been set up to Allow External Storage. In the view I’m rendering the smallImage binary data into the first image view and the largeImage binary data into the second image view. Both image views have content mode set to aspect fit.

Close the Before Migration version of the project. Now open After Migration (Bug) version of the same project. The only difference between this version of the project and the first is that I’ve migrated the CoreData model by adding a model version and adding a new unused string attribute named newAttribute to the Core Data Entity.

The project is already set-up correctly for automatic Core Data migration. Build and run. This should replace your previous version of the example app and perform the automatic model migration. But what happens to the binary data attributes of our entity? Here’s what happens, the small image remains and the large image is deleted (see figure 2)!

Figure 2: After CoreData Migration

Figure 2: After CoreData Migration

Ok, point proven. Delete the demo app from your iOS Simulator. Run the original Before Migration version once more – you should now have both images displayed as before. Now for the solution…

The Solution

Now build and run the third version of the example app within the After Migration (Solution) folder. You should still have 2 images displayed after migration – hurray!

The solution I’ve come up with is when your code initializes a persistent store coordinator for your Core Data model run a few checks before attempting automatic migration. Check whether the new model is compatible with the current stored model. If it’s not then you know that Core Data is about to migrate your old model to your new version and in doing so will wipe the external storage folder. Before it does so simply move the external storage folder to a temporary location. Once the migration has completed replace new empty external storage folder generated by Core Data. Here’s the code that you’ll also find in the Model class of the example project within the After Migration (Solution) folder:

- (NSPersistentStoreCoordinator *)persistentStoreCoordinator

{

if (_persistentStoreCoordinator != nil) {

return _persistentStoreCoordinator;

}

NSURL *storeURL = [[self applicationDocumentsDirectory] URLByAppendingPathComponent:@"CoreDataBinaryBug.sqlite"];

NSError *error = nil;

NSDictionary *sourceMetadata = [NSPersistentStoreCoordinator metadataForPersistentStoreOfType:NSSQLiteStoreType

URL:storeURL

error:&error];

//Check if the new model is compatible with any previously stored model

BOOL isCompatibile = [self.managedObjectModel isConfiguration:nil compatibleWithStoreMetadata:sourceMetadata];

BOOL needsMigration = !isCompatibile;

NSFileManager *fileManager = [NSFileManager defaultManager];

//Prepare a temporary path to move CoreData's external data storage folder to if automatic model migration is required

NSString *documentsPath = [NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES) lastObject];

NSString *tmpPathToExternalStorage = [documentsPath stringByAppendingPathComponent:@"tmpPathToReplacementData"];

NSString *pathToExternalStorage = [documentsPath stringByAppendingPathComponent:@".CoreDataBinaryBug_SUPPORT/_EXTERNAL_DATA"];

if (needsMigration) {

if ([fileManager fileExistsAtPath:pathToExternalStorage]) {

//Move Apple's CoreData external storage folder before it's nuked by the migration bug

[fileManager moveItemAtPath:pathToExternalStorage toPath:tmpPathToExternalStorage error:nil];

}

}

NSDictionary *options = [NSDictionary dictionaryWithObjectsAndKeys:

[NSNumber numberWithBool:YES],NSMigratePersistentStoresAutomaticallyOption,

[NSNumber numberWithBool:YES], NSInferMappingModelAutomaticallyOption, nil];

_persistentStoreCoordinator = [[NSPersistentStoreCoordinator alloc] initWithManagedObjectModel:[self managedObjectModel]];

if (![_persistentStoreCoordinator addPersistentStoreWithType:NSSQLiteStoreType configuration:nil URL:storeURL options:options error:&error]) {

 

NSLog(@"Unresolved error %@, %@", error, [error userInfo]);

abort();

}

else {

if (needsMigration) {

//Apple's automatic migration is now complete. Replace the default external storage folder with the version pre upgrade

[[NSFileManager defaultManager] removeItemAtPath:pathToExternalStorage error:nil];

[[NSFileManager defaultManager] moveItemAtPath:tmpPathToExternalStorage toPath:pathToExternalStorage error:nil];

}

}

return _persistentStoreCoordinator;

}

Once more, here are the source files for the example project:

Download Source Files

Interested in iOS Development and Core Data? Follow me on Twitter for more tidbits :-)



10 Comments

  1. Kevin

    Awesome, thank you for this! We’re doing something similar (storing both thumbnails and large images in core data for offline use) and I just noticed the large images were gone after doing an auto-migration. Glad I was able to catch this and implement a solution prior to launch.

  2. Tyler Loch

    You’ve changed an Extraordinarily Bad Thing into a mere Minor Inconvenience. Thank you so much!

  3. Nick Kuh

    Thanks for the comments guys – so pleased that this post has helped you out!

  4. Nick Kuh

    I was just beginning to log this as a new bug on Apple’s radar bug reporting system but I can’t reproduce the issue in the beta version of iOS 6 so I guess they caught it since iOS 5.

  5. Andrew

    A question, I had problem keeping images on core data, due to the caching thing, when I need to loop on all my objects (object Person, with a big image), in iPad 1 crash due to low memory, with memory analyze I can see a peak of memory of 80-100Mb. Should be coredata that cache all the binary image.
    I don’t see any leak (and however I’m using ARC, so only retain cycle can create problems but there aren’t).
    How I can store images, and ask coredata to not load they when I loop objects? (I need to loop objects to check other things, but I don’t need to have images at that time). I tried with reset but don’t seem to help.

  6. Scott Gardner

    Hi Nick!

    Thanks for posting this article, and relieved to know that this is fixed (i.e., if non-reproducible == fixed!) in iOS 6.

    Scott

  7. Adam Chin

    Hey Nick,

    I stumbled on your blog posting, and timing couldn’t be better.
    I hadn’t had the chance to check it out in iOS6 yet, but will. Based on your findings above we may be on the happy path.

    Thanks much!
    Adam

  8. Kesava

    Hi,

    Did you happen to check if this bug is fixed in ios6? Or do we still need this work around?
    Any help will be appreciated.

  9. Diego

    Hi this bug seems to have disappeared in iOS 6, although this is still very useful as we need to backup the sqlite underlying Core Data on our remote server.
    Backing up only the sqlite file does not save the externally stored pictures, but give this trick we can now zip the external picture folder and store it remotely.

    We still have not tried though if a restore would maintain references to the images. Will try now and update you.

    Thanks!


Leave a reply