SQLite storage

The strategy for storing documents is now in it’s own class, and determined by the docbase config file. With this I added a class to store documents in a SQLite database. Originally I wanted to use Tokyo Cabinet, however after further looking at the LGBL issues, I realized it’s not really feasible for an iPhone application. So I went with SQLite instead, basically using it as a simple key-value storage mechanism.

There are now 3 different local storage mechanisms: my initial hash file storage, SQLite storage, and a simple single file storage. Really, if ‘present me’ could go back and talk to ‘past me’, I probably wouldn’t have written the hash file storage. I wanted something that was easy to view in a text editor for development/testing purposes. It’s not bad, but a single file is much better and SQLite is a better form for release.

In a premature optimization (since I don’t think I really need it yet), I also added support for document indexing. It’s only implemented in SQLite storage and probably will never be in either of the other storage formats. I mostly did it out of curiosity to see how I can detect the appropriate fields in an NSPredicate and use them if possible in a query. It’s still very basic support but it works. I added a few class methods to NSPredicate simply to assist in building predicates that utilize the indexes.

Syncing

A fourth storage mechanism was added as well: remote storage. Along with it there is a sync server that will work with it. All this is very preliminary but it will sync documents between a client and server. The server is currently Mac only since it uses NSSocketPort, but I think it’ll be an easy change to use CFSocket instead to get it working on iPhoneOS. The sync client is currently pretty draconian about any change conflicts. Right now the later change always wins. If the change dates happen to be equal, the server version is used. Eventually a delegate will be used to allow the app to decide what to do about conflicts. The server uses Bonjour to advertise itself, and there’s no form of authentication built in, so running it allows anyone on the local network to access the served docbase.

Other Stuff

There’s probably a bunch of other small changes that went in that I’m not thinking of right now. One thing I did want to highlight is class extensions. I’m trying to be somewhat careful about what classes I add categories to. Right now 3 framework classes are extended. NSMutableDictionary has methods to make it conform to the BRDocument protocol, as well as a convenience constructor to build a proper document dictionary (adding the document id, document type, and modification date automatically). NSDate has methods to format dates in a docbase-friendly format. Mostly this is for SQLite, since it seems to like UTC times without the addition of a timezone offset. And, as I mentioned earlier, NSPredicate has some methods to build predicates that are more likely to hit indexes.

So that’s it. Like I mentioned, it’s no longer a simple 1 or 2 files, but I’m pretty happy with it so far. I was especially excited to have it running on the iPhone and syncing with documents on the Mac.

So as I was figuring out how I want to build my website, I looked at least briefly at a few LAMP-based solutions (A quick disclaimer: I’m not a ‘web developer’, so go easy on me if I say something erroneous or that you violently disagree with). I went from heavier weight frameworks (Joomla, ExpressionEngine) to considering just coding PHP pages myself when I found CodeIgniter which seemed just right (yes, the Goldilocks metaphor went through my head at the time I was checking it out).

What’s Great

Like a lot of frameworks today, CodeIgniter borrows heavily from Rails. It follows a similar MVC pattern, and uses a somewhat similar directory structure. I had actually never done any coding in PHP before this, and it helped a lot that it followed a familiar pattern, to reduce the amount I was learning at once.

Deployment is also simple. Other than needing to mess around with htaccess a bit to get the routing working like I wanted and setting up the database, it was just a matter of copying files to the server. Most of the site-specific configuration is located a few files, so just excluding those means updating the site is pretty easy.

Probably the thing I liked most about it is something that not everyone will like. I like to write code rather than click buttons to add new pages or content. I don’t really consider myself a control freak, but it drives me nuts that I can’t see or at least don’t have a good picture in my head on what’s going on behind the scenes to make a web page appear. Perhaps I’m being unfair when I compare it to CMS frameworks like Joomla and ExpressionEngine, but these are the things I checked out when I went searching, so at very least they’re comparable as ‘frameworks to build a website with’.

What’s Less Than Great

The mapping from a URL to a controller method has one limitation that bothered me. You can configure it to pass parameters in the path, or in query string, but not both. For a full featured REST API this would really bother me, but it’s just a minor annoyance otherwise.

The framework’s support for models is kind of weak. Okay, it’s really weak. It’s really nothing more than ‘put your database code here’. No relationships, migrations, or any of the good stuff you may want. Once again though, I was developing simple company website with very few database needs, so it wasn’t a big deal.

The End Result

I was surprised by how both easy CodeIgniter and PHP was to pick up. I didn’t mind at all the temporary switch from Cocoa development, and won’t mind too much when I have to go and update it.

• Manton Reece: I don’t use Core Data either
• Gus Mueller doesn’t like it’s smell

There are others as well. I was dying to post about this but was even more determined to get my first release out the door. Now that that’s done, I figured I’d share my thoughts. I think Gus comes closest to my own feelings when he say’s he doesn’t like it’s smell. Although maybe he didn’t need to be beaten on the head as much as I did to come to this realization (someone not liking EJB is just proof that they’re not totally nuts). The problem I see with Core Data isn’t with Core Data in particular, but with object-relational mapping as a whole.

I have used a lot of object-relational mapping tools: Hibernate, NHibernate, ActiveRecord (from Rails), DataMapper (another Ruby one), and of course, Core Data itself (I’ve never used EOF however). I’ve also created custom object-relational mappers (several times in differing languages) on top of very large data models. After all this my conclusion is that it’s best to avoid having to do this mapping.

Here’s my theory on how all this started. In the beginning, there were databases, and we put data in them, and life was good. Well, really it wasn’t, because then came along object-oriented programming, and we realized that life would be better if we used this and instead of data, we have objects. Of course we still needed a place to save the objects, so we figured out many different ways to shove our objects into a database. Fast-forward many years and now it’s basically standard practice: make your objects, convert them into data for databases, then get them back out converting them back into objects.

Really, all of various tools do an incredible job of managing this assuming you let them own the data model completely (I won’t talk about using them on legacy systems since it’s a nightmare and doesn’t really relate to Core Data) and it gets you a bunch of things:

• In general switching to different SQL databases isn’t too bad
• You have some powerful searching capabilities at your hands
• Depending on your backend, it can scale (although Core Data’s failure to scale in certain situations is what started this whole thing off)
• You have all the other potential side benefits of a database – transactions, sharing data between clients, etc.

Sometimes this stuff is important. In Core Data’s case and in the case of almost kind of desktop application (where the database is local) this is all irrelevant or untrue. In the meantime you’ve set your self up with the biggest limitation of a SQL database: all of your objects must be mapped to tabular data. In the simple cases, this isn’t a big deal. However, the more complicated your object model gets, the more complicated this mapping gets or the more your constrained in how you can even design your object model. In short, the fewer translations your objects/data/whatever must go through, the better.

It’s easy to see the evidence of this impedance mismatch in Core Data. You want ordered collections? You’ll have some work to do. Or how about dealing with the synthetic ids of newly created objects? Or (common to all object-relational mappers) the hackery that goes on to deal with inheritance? All of these problems are solvable, but they’re just a few indications that SQL databases might not be ideal for storing objects.

So what’s the solution? I’m not sure. No, neither CouchDB nor my own CouchDB-inspired BRDocBase are really answers (in case you thought that’s where I was headed). They’re not intended for generic object persistence. Whatever happened to object databases? I know some are still around, and I’ve seen really cool stuff done in Smalltalk with them. Of course ’seeing cool stuff done’ is not at all the same as actually developing with one, which I have zero experience with. I’d love to see an Objective-C based OODB, and have often thought of developing one for fun, but I know that if I’m not using it in an actual product it wouldn’t come out right (that’s just me…I need to be actually using it to see if something is a good idea or horrible one).

As mentioned by a few others, Aaron Hillegass is working on BNRPersistence which sounds interesting. I haven’t looked deeply into it, and Brent Simmons mentioned the LGPL issue of Tokyo Cabinet. I’m under the impression that including an LGPL component is fine fine for non-free, close-sourced products, as long as you share the source (and your changes) of that component (but I’m definitely no lawyer). I’m certainly planning on taking a better look at it though.

Before I finish I just wanted to make clear that I am not at all saying ‘Core Data sucks, don’t use it’. It’s still the best general-purpose solution for persisting objects for a desktop Cocoa application (although I really wish they’d add ordered collections). I would say, that if your data is more tabular in nature and you do more of the kinds of operations SQL is good at (searching, scaling to lots of rows, etc) you might consider doing your own database access. But even if we had an awesome Cocoa OODB available to us, I’d say the same thing (the right tool for the right job). The main point of my lengthy, semi-rant is that it seems the development community as a whole has gone down this path of object-relational mapping and accepted it as The Way to be persisting our objects and I just wish we had better options.

Being a developer, all the non-coding tasks needed to release software are particularly painful. I’m not really done either, as I still have to figure out everything I’m going to do to market it. However, there were some fun coding-related challenges that I went through and I’ll probably have a post or two about those in the near future.

Right now, I’ve heaved my sigh of relief, and it’s time to get on that big, scary ‘market CrankTunes’ task on my todo list. I now regret all the times I’ve ranted against people in marketing departments.

Did I mention I’ve never done a custom control before? Some drawing in views, sure, but never a control. Let alone one with a weird menu that stays open while you type in a text field. I imagined it would be a learning experience, and I definitely learned a lot about being frustrated (my knowledge of making a custom control is still iffy).

The main functionality I was looking for from Safari’s search field was:

• menu includes search matches (Suggestions) as well as recent searches
• menu opens when the user enters text that has possible matches (rather than just when the button is clicked)
• menu will update the matches while the user continues to type
• change focus from the text view to the menu and back with the arrow keys

I started out by snooping around in Safari using F-Script. It’s using subclasses of NSSearchField and NSSearchFieldCell to do it’s job, so I figured that would be a good thing to try out first. Unfortunately, I just couldn’t figure out how to keep the menu open. After a bit of fruitless hacking to try an mimic Safari, I decided to give up on subclassing NSSearchField and start instead with NSTextField. NSSearchField just lets you provide a template menu and then at some point it uses that menu to build a new one with the placeholders fixed up with recent searches, etc. It’s a fine concept, but I thought it was getting in the way of my understanding on how to keep the stupid menu open.

Still I couldn’t figure out how so make menu open. Finally, I started making some progress when I gave up on using a menu at all and decided to fake it with a child window. I can’t be sure, but my suspicion is that Safari is doing the same thing (my F-Script skills aren’t advanced enough to figure it out).

Once I had the menu problem solved, my only barrier was my complete lack of control programming knowledge. I forged on fearlessly (or foolishly) ahead and got something that basically work. So am I happy with the results? Not really. I had enough issues in dealing with managing the control’s focus that I feel I just don’t get it yet and there should be better ways to do it. I know part of the problem is the funkiness of NSTextField and the associated field editor and how the first responder is set between the two. I’m also coming to somewhat regret not sticking with subclassing NSSearchField and NSSearchFieldCell since I had some issues drawing the control with the search icon inside the text field’s border.

In the end though, it works well enough for me (for now). I’m still planning on revisiting it at some point to try and clean up the messiness but right now I don’t think I’m in the right mind to look at it. There are also definitely some limitations that I don’t feel affect me (yet). Since the menu is completely fake, you don’t get much flexibility from it. You can’t have submenus or view-based menu items for instance. Also, there’s no asynchronous behavior in populating the menu. If, for some reason, the BRSearchFieldDataSource needs a long time to produce a set of results, there’s no built in support for handling that.

For the brave, insane, and those who wish to mock me, the source is available on github here: BRSearchField. Using it isn’t too bad. The only non-straightforward part that isn’t illustrated by the code is that you add an NSTextField in Interface Builder, then must change it’s class to BRSearchField and it’s cell’s class to BRSearchFieldCell. Other than that, set it’s data source and search menu prototype either in IB or in code and you should be (hopefully) good to go.

Sure enough, when I ran the application outside the debugger, there were no issues playing the songs in question. So, somewhere inside NSSound (I imagine pretty deep) it’s detecting that a debugger is attached and just exiting to prevent hacking the copy protection. It makes sense, but I wasn’t able to find any specific mention of this and I’m sure I never encountered this before Snow Leopard.

Perhaps it’s common knowledge and I just missed it, but I thought I’d mention it anyway. Okay, I admit, I actually first encountered it a few months ago, then again just the other day and I had forgotten what caused it. I figured if I posted about the problem it would help me remember the next time I hit it

As I was developing my app, I noticed that I was quickly outgrowing the measly 17 buckets I had originally hard coded for a database. It’s not too bad for a hundred or so documents, but probably not sufficient for several thousand documents. I knew it really need to be configurable. So in comes the docbase configuration file.

The configuration is just saved as another JSON dictionary in the document database folder and right now it only contains the setting for the number of buckets. There’s a few important changes that go along with this.

I moved anything that might give cause an error (within reason) outside of the initializer for BRDocBase. Now, if something is wrong, like it can’t create the database or the configuration is wrong it’ll give an error on the first call to one of the main methods (anything that has an error parameter). Basically, all the file-based initialization is done lazily. I’ve never seen an init method take an error parameter, and it just feels wrong to me.

If a configuration isn’t passed to the init method, one of two things will happen. If it’s a new database, a default configuration will be used. Otherwise, the configuration that already exists will be used. In either this case, the configuration property will return nil until it has actually completed its lazy initialization. If a configuration is passed on initialization and it doesn’t match the existing configuration, the lazy initialization will fail with a configuration mismatch error (this can be useful in updating a configuration).

Migrator

Without someway to migrate a database, there’s no way to change the configuration, so I added BRDocBaseMigrator. It’s pretty simple class right now that can update a database to use a new configuration. It just uses brute force to read and re-write all the documents in the database in the new configuration.

The migrator is smart enough to not do anything if the configuration matches. In that case, the original document database will be returned. So if fairly simple to create/open a database that always has a particular configuration:

NSString* path = ...;
NSDictionary* config = ...;
NSError* error;
BRDocBase* docBase = [[BRDocBaseMigrator docBaseMigrator]
    update:[BRDocBase docBaseWithPath:path]
    toConfiguration:config 
    error:&error];

Now if the configuration isn’t what is wanted, it’ll be updated. Otherwise it will just return.

In an earlier post, I talked about extending a class at runtime using (what Ruby calls) modules and I’ve added a github repository for it. It covers most of the functionality we need with the exception of undoing the changes it made. The implementation for this (along with absolutely no error checking) is pretty short:

+(void)revertMethod:(SEL)methodName onClass:(Class)cls
{
    NSString* savedSelectorName = [[NSString alloc] initWithFormat:
        @"%s_%s", class_getName(cls), sel_getName(methodName)];
    SEL savedSelector = NSSelectorFromString(savedSelectorName);
    Method savedMethod = class_getInstanceMethod(cls, savedSelector);
    IMP savedImp = method_getImplementation(savedMethod);
    Method existingMethod = class_getInstanceMethod(cls, methodName);
    method_setImplementation(existingMethod, savedImp);
}

If you read the other post, you’ll notice that we saved the original implementation of a method in a new method: <class name>_<method name>. This way, when revertMethod:onClass: is called, we can just reset the implementation. The one thing we can’t do is get rid of our backup method since this capability was removed from Objective-C 2.0 apparently…oh well.

Now that we have a way to reset our method back, using it is a piece of cake. Pretend our class under test, Foo, has a method readDataFromFile that returns a string from some configuration file. We don’t want to be bothered with this during a test, or we want to supply specific values for it while testing. So we might create our fixture as follows:

@implementation FooTest

-(void)setup
{
    [BRClassExtender 
        extendClass:[Foo class] 
        withModule:[FooTest class] 
        override:YES 
        method:@selector(readDataFromFile)];
}

-(void)tearDown
{
    [BRClassExtender 
        revertMethod:@selector(readDataFromFile) 
        onClass:[Foo class]];
}

-(void)testConfiguration
{
    // do something with foo
}

// this is our replacement implementation
-(NSString*)readDataFromFile
{
    return @"my configuration";
}
@end

Now rather than opening some file and reading it, we simply return the string specified. Note that the method can be private since the test case doesn’t care if it’s in the interface of Foo or not.

Using this kind of testing technique can be a slippery slope. Obviously, if we do this, we’re no longer testing the implementation of the method we overrode. I prefer only using it if I can keep the overridden methods really small and it’s easy for me to verify they work by some other means (like running the actual application).

I’ve been going back and forth between using an NSTableView and an NSCollectionView for a list of items, and I think I’m settling on the NSCollectionView, but there are definitely some interesting challenges. A couple of quick notes: I’m building for Snow Leopard so a few of my comments only apply to 10.6-only builds. Also, all my comments here come from my own experience with NSCollectionView rather than actual knowledge about how it works internally. So, I could easily be wrong when I mention how I think something is working.

NSCollectionViewItem

First up is NSCollectionViewItem. In 10.6 it’s now a subclass of NSViewController. Noticing this helped me understand a lot better how an NSCollectionView might handled in code. Previously I never viewed it as something I should subclass, but now it seems obvious that if I want to handle actions from controls an item view, I put it in my own NSCollectionViewItem subclass.

There’s a problem when I take this a bit further though. The NSCollectionViewItem that I set up in the nib isn’t actually the instance gets used when the application is running. That instance is just a prototype, underneath the hood it’ll be cloned, and Cocoa seems to take care of all the messy work of setting up the actions and bindings that I set up in Interface Builder.

It doesn’t, however, fix up my outlets automatically. Any outlets that I add to my subclass, I need to handle on my own. This can be done by overriding copyWithZone:. There’s a catch however: this won’t work with outlets pointing to objects on the controller’s corresponding view.

NSView

Conceptually, each NSCollectionViewItem cloned is paired with an NSView (actually, from my understanding, only enough views will be created to fill the NSCollectionView and they are reused). While the controllers are cloned with copyWithZone: the views are not, they are simply created anew from the data in the nib for each additional one that’s needed. When you set up outlets pointing to items in that view, they’re pointing to items that, I believe, are never even actually used.

Unfortunately, I didn’t stumble upon a good way to have outlets pointing to objects in the view. The only way I could think of doing this was to set the tag value on controls I’m interested in and search through subviews to find them. As I mentioned above, bindings seem to be handled by the framework as you’d expect, so hopefully outlets into the view won’t be needed.

The first thing I wanted was a change flag on documents. This will be the first optional method on the BRDocument protocol. If the property doesn’t exist on a document object, everything will proceed as it did before. I went with the property name isDocumentEdited since that’s the method name on NSDocument. I’m not really going for any kind of BRDocument to NSDocument parity but I decided a familiar name was best.

One important functional change is that if the document implements the property and it’s not set, the saveDocument:error: method will not save the document. At first, I had save doing absolutely nothing with the document if the flag was set, this caused a small issue with new documents though. Currently, a nil return value means an error occurred and the error object is appropriately set. So now there are a few options:

• Return nil with no error set
• Go ahead and set the id, but still don’t save the document
• Ignore the isDocumentEdited flag in this case and save the document
• Return nil with an error
• Raise an exception

I went with the 4th option. The first is the worst in my mind since it forces a user of BRDocBase to account for a nil error even when nil is returned from save. The other options aren’t really bad. Since this is really kind of an odd situation and I consider it a developer error (new documents should be automatically set as edited) I may at some point change it to raise an exception. Although exceptions aren’t used a lot in Cocoa, a developer error warrants an exception in my mind.

A New Hash Algorithm

While I was in the code I realized that my in-memory storage of documents that had been retrieved had a possible problem. To make saves quicker, internally I’m storing documents similar to the file based stored. There’s a top dictionary keyed by the bucket ids containing dictionaries itself mapping documentId to the document object. But since the bucket number is generated by hashing the documentId, if the inner dictionary happens to have the same number of buckets as the file based storage, we’d end up with 100% collisions in the dictionary.

That led me to think of a less likely, but way more serious problem. What if Apple changed the hash algorithm on NSString? I wouldn’t think this is likely, but the structure of BRDocBase storage is dependent on Apples internal implementation. So rather than depend on this, to assign bucket numbers I now use a a different hash algorithm: FNV-1. I saw other, more complicated and faster algorithms, but I don’t really need ultimate speed for my purposes. Of course this means that any document database I have is now useless. Since it’s still early in development, I didn’t bother writing something to convert my existing databases and just trashed them.

Finally, I put BRDocBase up on github. It’s available here BRDocBase