iOS Core Data Performance issues

[Soulstorm]

I am creating an application that fetches data from a server (like an online address book) and uses it on the iPhone. Communication and fetching data work fine.

However, problems come into play when using Core Data to store the fetched info. (NOTE: I use SQLite as type of storage) Here is my set up and classes:

-- DSAccount. This is a subclass for NSObject. Holds fetched info from accounts fetched from the server (accountID, telephone, mail, etc). I use this class to display info inside a table view.
-- CachedAccount. This is a subclass of NSManagedObject, for Core Data. Holds exactly the same information as the DSAccount, only that I use that for storage.

Way of storing data:

-- Download the info from the server and make DSAccounts for displaying the information in the table view.
-- Then, for each DSAccount, I first check if an account with the same account ID already exists in the database (using predicates) and if there is not, I insert a Managed Object inside the core data database and populate its properties from the properties of the DSAccount. Then, I save.

However, I am experiencing performance issues on my device (even on my computer!) while saving. Consider that I fetching approx 9000 accounts from the online database, that I must save into my local core data database at once.

I feel that the way I have set up my data save store has problems. Can anyone maybe recommend me a better approach?

 

[RonC]

What performance problem are you having?

I am creating an application that fetches data from a server (like an online address book) and uses it on the iPhone. Communication and fetching data work fine.

However, problems come into play when using Core Data to store the fetched info. (NOTE: I use SQLite as type of storage) Here is my set up and classes:

-- DSAccount. This is a subclass for NSObject. Holds fetched info from accounts fetched from the server (accountID, telephone, mail, etc). I use this class to display info inside a table view.
-- CachedAccount. This is a subclass of NSManagedObject, for Core Data. Holds exactly the same information as the DSAccount, only that I use that for storage.

Way of storing data:

-- Download the info from the server and make DSAccounts for displaying the information in the table view.
-- Then, for each DSAccount, I first check if an account with the same account ID already exists in the database (using predicates) and if there is not, I insert a Managed Object inside the core data database and populate its properties from the properties of the DSAccount. Then, I save.

However, I am experiencing performance issues on my device (even on my computer!) while saving. Consider that I fetching approx 9000 accounts from the online database, that I must save into my local core data database at once.

I feel that the way I have set up my data save store has problems. Can anyone maybe recommend me a better approach?

Does the database only consist of 1 record type? Not that that's a problem, but that's what I gather from your description. 9000 records isn't really that many in the big scheme of things, but it may be too much for SQLite (I'd be surprised if it is, but I don't have a lot of experience with that database).

I just took a peek at my CoreData model and see that I checked the box next to Indexed for my orderNumber field - make sure you've done this (or the equivalent of this) for your accountID field. That searches in the database to go faster given that what you're searching for is the accountID.

You told us everything EXCEPT the problem? I presume when you say "performance problem," you mean "slow," but you can also mean "unresponsive;" let me explain the difference I mean by those.

"Slow" means it takes a long time to retrieving information from the database and it impacts the way your application operates, while "unresponsive" means that while you're initializing or populating the database your app just sits there, apparently doing nothing.

Slow I can't help you with - you've got a relatively large database, but I'm certainly in no better position to make CoreData any faster than Apple is. Unresponsive, though, I think I can help with. The trick to unresponsive is to get stuff out of the way of handling the UI to make your app look MORE responsive. Move stuff to later, move stuff to threads that get run while the user is thinking, ....

Whatever you're doing, don't load the ENTIRE database into a dictionary of shadow objects. You only seem to care about existence, so let's solve the existence problem.

My first thoughts without understanding WHAT you've got in/not in your database and the structure of your app: At startup, pre-populate an in-memory INDEX of the keys fields in the database, in a sorted/searchable data structure (An array or dictionary is probably the wrong thing for this, use a NSMutableSet - it should be good at searching). Download the data and compare against the index - if the key is there, skip it; if not, update the index and pass the data to another thread add the data to the database.

How do you present the data to the user? Is it in a table (like the Contacts app)? Use a FetchedResultsController and follow the tutorial & sample code that shows how to get an indexed table. It's pretty responsive though it doesn't have the download problem that you have.

If you want to then present one of the database entries, query for that one directly and maybe add an accessor to CachedAccount that returns the corresponding DSAccount. If you modify the DSAccount and want it reflected in the database, then you need to link these together at this time, so the DSAccount change can be put directly into the database.

Finally, perhaps the download can BEGIN with a MD5 checksum that can be easily compared with the last MD5 checksum - if they're different, then the data is different; if they're the same, it's highly likely they're the same (1 in 2^123 or so likelihood of collision, according to Wikipedia).

 

[Soulstorm]

That was indeed food for thought.

First of all, I should have explained that by saying "performance issues" I mean "too slow to be an acceptable delay for the user. Running this code takes 5 minutes (!) to complete, just to save all the entries in the database.

The database contains more than one entries. I have accounts, messages, etc.

From your posts, what I have understood is this:
-- Use NSFetchedResultsController.
-- Use indexes for the Managed Objects that this database will hold
-- Get rid of the DSAccount and use Managed Objects directly
-- Prepopulate an NSMutableIndexSet with the IDs of the messages alone, and sort it, so that I can search it faster.

Have I understood that correctly?

 

[RonC]

That was indeed food for thought.

First of all, I should have explained that by saying "performance issues" I mean "too slow to be an acceptable delay for the user. Running this code takes 5 minutes (!) to complete, just to save all the entries in the database.

The database contains more than one entries. I have accounts, messages, etc.

From your posts, what I have understood is this:
-- Use NSFetchedResultsController.
-- Use indexes for the Managed Objects that this database will hold
-- Get rid of the DSAccount and use Managed Objects directly
-- Prepopulate an NSMutableIndexSet with the IDs of the messages alone, and sort it, so that I can search it faster.

Have I understood that correctly?

Pretty much except for the last one - NSMutableIndexSet only holds unsigned integers, and it's always sorted. No reason to sort separately, and I was more thinking that your indexing key was a string, in which case NSMutableSet is more appropriate.

Getting that result quickly (when the database is fully populated) might not be too bad. This code (copied from the Core Data Snippets) will return the result of the query in Dictionary form. (I'm not sure where it's hiding in the array returned by the query - my current app is broken and I'm taking a quick break to let my head settle down)

NSManagedObjectContext *context = <#Get the context#>;
 
NSEntityDescription *entity = [NSEntityDescription  entityForName:@"<#Entity name#>" inManagedObjectContext:context];
 
NSFetchRequest *request = [[NSFetchRequest alloc] init];
[request setEntity:entity];
[request setResultType:NSDictionaryResultType];
[request setReturnsDistinctResults:YES];
[request setPropertiesToFetch :[NSArray arrayWithObject:@"<#Attribute name#>"]];
 
// Execute the fetch.
NSError *error;
id requestedValue = nil;
NSArray *objects = [managedObjectContext executeFetchRequest:request error:&error];
if (objects == nil) {
    // Handle the error.
}

The key part about the set is to use it to avoid having to search the entire CoreData database with a query to determine entry existence. At the beginning, when the database is lightly populated, the search is fast; as the database grows, the search becomes harder and harder UNLESS you index the key.

(When I say the key is indexed, I mean it's a checkbox on the attribute you search on in the CoreData model - if that was not clear above I apologize)

 

[Soulstorm]

Pretty much except for the last one - NSMutableIndexSet only holds unsigned integers, and it's always sorted. No reason to sort separately, and I was more thinking that your indexing key was a string, in which case NSMutableSet is more appropriate.

Getting that result quickly (when the database is fully populated) might not be too bad. This code (copied from the Core Data Snippets) will return the result of the query in Dictionary form. (I'm not sure where it's hiding in the array returned by the query - my current app is broken and I'm taking a quick break to let my head settle down)

NSManagedObjectContext *context = <#Get the context#>;
 
NSEntityDescription *entity = [NSEntityDescription  entityForName:@"<#Entity name#>" inManagedObjectContext:context];
 
NSFetchRequest *request = [[NSFetchRequest alloc] init];
[request setEntity:entity];
[request setResultType:NSDictionaryResultType];
[request setReturnsDistinctResults:YES];
[request setPropertiesToFetch :[NSArray arrayWithObject:@"<#Attribute name#>"]];
 
// Execute the fetch.
NSError *error;
id requestedValue = nil;
NSArray *objects = [managedObjectContext executeFetchRequest:request error:&error];
if (objects == nil) {
    // Handle the error.
}

The key part about the set is to use it to avoid having to search the entire CoreData database with a query to determine entry existence. At the beginning, when the database is lightly populated, the search is fast; as the database grows, the search becomes harder and harder UNLESS you index the key.

(When I say the key is indexed, I mean it's a checkbox on the attribute you search on in the CoreData model - if that was not clear above I apologize)

Now I get it. So before starting to make changes at the database, I should extract from the database the existing account IDs and place them into an index set for easy searching. Before inserting the object, I will search for the existence of the account ID inside the index set (which is faster than searching through the database), and if it doesn't exist, I will insert the new account in the database, and the new message id inside the index set, for later searching.

Do you think creating a second Managed Object Context will help?

The only thing that troubles me is that I will also have to devise a mechanism to delete objects from the database that don't exist in the downloaded context.

Getting rid of the proxy object DSAccount is something that troubles me, though. I am afraid to do it, because of the problems I may encounter because Core Data requires you to instantiate an object and insert it into the database before you can use it.