In some environments user's home directory is located on the mapped network drive which in unacceptable for IntelliJ IDEA. You'll notice the huge performance degradation.

What is the problem?

There are some good motives for keeping the home directory mounted to a network folder rather than on the local hard drive. For example, all user settings and user documents can be backed up centrally, and the user could potentially log in to any computer and have the same user environment. However, there are also some drawbacks. Depending on how the home folder is configured, there will either be a remote network call for each file access, or the file changes will be executed as a batch job at scheduled intervals or specific events such as logging off the computer.

The second part of the problem is that IntelliJ IDEA produces quite a lot of data, typically several hundred megabytes, besides the anticipated artifacts. The data consists of IDE plugins, configuration files, log files, but the vast majority is internal cache files. By default, the data is written to a hidden .IntelliJIdea10/ folder in your home directory.

If you combine these two factors, the performance will suffer. How much depends on the bandwidth of your network, something that I became painfully aware of when I logged in remotely to the customer's network via a slow VPN connection.

Solution

Avoid network overhead by configuring IntelliJ IDEA to cache data on your local computer instead of in your home folder. A recent customer used Windows XP as their working environment, the instructions below are based on that experience.

1. Create a new directory on a local harddrive that IntelliJ IDEA can use, e.g. C:/.IntelliJIdea10/. Hint, you cannot create a library starting with a "." from Windows Explorer, you have to use the mkdir command from the terminal.
2. Locate the idea.properties file in the bin directory in IntelliJ IDEA's installation directory. The default location is C:\Program Files\JetBrains\IntelliJ IDEA Community Edition 10.5.1\bin.
3. Change the relevant properties to point to the recently created directory:

   # path to IDEA config folder. Make sure you're using forward slashes
   idea.config.path=C:/.IntelliJIdea10/config
   
   # path to IDEA system folder. Make sure you're using forward slashes
   idea.system.path=C:/.IntelliJIdea10/system
   
   # path to user installed plugins folder. Make sure you're using forward slashes
   idea.plugins.path=C:/.IntelliJIdea10/config/plugins
   

Considerations

Consider not to change the idea.config.path property if you are using several different computers and you would like to have the same configuration on all machines. Additionally, if you are using IntelliJ IDEA Ultimate, i.e. the commercial version of the tool, you should also make a cautious decision about whether or not to keep this property unchanged, because the license file is stored in the denoted directory. On the other hand, if you always use the same computer you might as well change this property together with the other properties, so that all IntelliJ IDEA files are stored in the same location.

References

• IDEA files location: http://devnet.jetbrains.net/docs/DOC-181
• IDEA license key: http://devnet.jetbrains.net/docs/DOC-200

After listening to Hanselminutes show #262 I became curious and have looked a bit at Dapper to check what it really is.

First of all, it’s implemented as a single file you can drop in your project. It’s created for and at stackoverflow. I heard somewhere that it should be 400 lines, but it looks more like 1200. It’s nice that I can understand much of how the ORM works just by browsing this file. But there are a few methods with a lot of il.Emit, which I can just hope works fine.

Dapper works as an extension of the IDbConnection interface through extension methods. With the use of generics it can return typed objects otherwise it returns dynamic objects.

I thought that perhaps a simple ORM like this is only targeted to a single provider, but of course it uses ADO.NET, so all providers supported there should work with Dapper. But as I understand it, you write raw SQL, so it doesn’t help you if you want to switch provider. But how often do you really do that? And it also means that you don’t have to learn a new language.

The goal has been to create a really fast and simple mapper, and according to their benchmarks, they have succeeded to make it fast. I think I heard that stackoverflow first used Linq2sql and Dapper is 4 times faster than Linq2SQL and 13 times faster than Entity Framework for a SELECT statement with 500 rows to Poco’s in their test case.

Of course it has some limitations, for example it lacks a identity map, but I suppose it makes it a good fit when you use a CQRS architecture and separate commands and queries?

I like the simplicity, and the speed, so I might try it for real some day.

When Bob the tester comes, and tells me that the client application sometimes hangs for about 30 seconds, when Bob is doing nothing. This client, and a hand full of other clients are connected to a system of connected servers. The problem could be a busy server or some internal client problem. We have to find out what the client is doing when it get’s busy, for no good reason.

My first idea was that perhaps Bob could use the feature in the task manager in Vista and Windows 7, right click on a process and select Create Dump File.

Automate it?

It would of course be better if we could automate it, it’s not that easy to select a process, right click and choose the right item within 30 seconds.

So then I found a good tool for generating dump files when a windows application is none responsive.

The tool is ProcDump from SysInternals.

ProcDump is a command-line utility whose primary purpose is monitoring an application for CPU spikes and generating crash dumps during a spike that an administrator or developer can use to determine the cause of the spike. ProcDump also includes hung window monitoring (using the same definition of a window hang that Windows and Task Manager use), unhandled exception monitoring and can generate dumps based on the values of system performance counters.

I have used it to catch the cases where TheClient hangs for a number of seconds.

To use it in this case:

1. Start TheClient
2. Run ProcDump with the following command line:

"procdump.exe -h -ma TheClient.exe TheClient.hang.dmp"

It will then generate a dump file in the current directory when the application shows as "Not responding". Then it’s just to bring out WinDbg to find out what was going on.

I can understand the joys of making sure that every potential bug is removed, and the bragging rights of 100% automated test coverage as well. But I would argue that it is equally fraudulent to claim 100% automated test coverage to a client as a virtue. I am not arguing that automated tests should be abandoned, far from it. I am arguing that automated tests should be applied with the same care as optimizations.

Premature Automated Testing is Bad

We have all learned that premature optimizations are bad. Guessing what could fail is hard, the only safe route would be to write a check for everything! There is even a coding practice encouraging this, test driven development or TDD for short. Which should really be called ADD for assertion driven development. Since what is done is not testing, but checking by asserting conditions against a specification.

Take a moment and answer honestly; for any project with good automated test coverage what is the ratio between time spent writing the checks versus the time these checks have saved you? In my experience the more zealously automated test coverage is applied, the more time is spent maintaining the tests. For any project where automated tests have been written first, and code later, I have yet never experienced that the checks have saved more time than the time they cost to develop and maintain.

When optimizing it is hard to find the bottlenecks. Gut feeling is good many times, but often even your gut feeling is wrong and the performance bottle neck is somewhere you would never expect. It is the same with defects in code, your gut feeling is good for pin-pointing many defects, but often the cause is something you never expected. So instead of wasting time guessing what the problems may be down the road you should inspect and measure where the issues are. And when an issue is found then you write a check for it to ensure it does not break again.

Change the Algorithm

The best optimization is often not to optimize at all, but to change to a better algorithm. Writing a bubble sort algorithm in assembly will never yield a better result than a sloppy merge sort written in C anyway, so do not waste your time.

Same can be said about code quality. If you have many issues, and your code breaks often, adding automated tests will only be bandaid until it breaks the next time. Change the code into something more stable. If needed, write checks to ensure the new implementation conforms to the same public API when you refactor it. Take notice to write checks with as good coverage as needed when you are about to refactor your code not before.

If you architect your application with many small independent units, each with a clearly defined public API, then refactoring is not a big problem.

Perceived Quality

All green lights and a 100% automated test coverage looks good on a report. But it can never catch what is most important of all; is the application valuable to the user? In that regard the 100% automated test coverage is a false safety, it gives the illusion of quality. A dangerous illusion that can be used to blind managers, customers and other stakeholders from seeing the real issues with the application.

There is yet no testing framework made that is as efficient as a human being actually using the application. Not just people blindly following a test protocol, that is even worse. But stake holders using the application as intended. Not just short burst before committing code to source control, but also regularly on longer sessions. So that you get a feel for how the app is used, what works, or just what grinds your gears. Usability issues are also issues and should be dealt with.

Not every issue is created equal. With the limited time every project have fixing, or predicting, each and every one of them is never feasible. So do not waste time fixing the one in a million issue, or writing checks for the bread and butter code. But do make sure the application fails gracefully. Fail as gracefully as you can for the end-user, they are very forgiving if you fail with style and make sure no data is lost. But also fail gracefully for yourself, be generous with logging when you do fail. And remember to add an automated check so that the reason for the failure may never raise again, once you fixed it.

Model View Controller

All application logic can be divided into three categories. Model for the business logic of your product, View for the display and input to and from the user (or other application), and Controller that is the mediator between the two. Any application of any significant size will have smaller MVC patterns within their units as well. Well defined boundaries between what is a Model, View and Controller is not only good for re-usability, maintainability, and replacing parts. It is also great for testing. Have well defined and testable public APIs for the parts, and the private implementations tend to be simpler and of higher quality. Do not let the public API be defined by how something is implemented, let the implementation be defined by how you want to access it from the outside.

Writing unit tests for the Model of your application is seldom a waste of time. Often the Model is what you begin implementing, so using unit tests as the incubator until you have enough logic in place to implement the first draft of your application is only rational; code that is unused is seldom of any quality at all.

Writing unit tests for the View yield much less returns on investment. It is also highly probably that you will have to write these tests over and over again. I would go so far as call them harmful, too many automated tests on the Views may in fact discourage you from doing drastic but needed usability changes. And no automated test in the world can ever flag if a View is usable, looks good, and feels right to the end-user.

Writing unit tests for the Controller should be a waste of time, otherwise your Controller is doing too much. A Controller should only be the mediator between the View and Model, if it does more then it should be split up and the proper parts moved into Model and View layers. So writing unit tests for a good Controller is basically just verifying that your are using the APIs of the Model and Views correctly.

Conclusions

We have learned that premature optimization is bad, often counter productive or wastes our time for minimal gains. Let us all grow up and also learn that premature automated testing is equally bad. The time we have is to precious to waste on overzealous automated testing instead of adding real value to the product by applying tests where it makes real difference to the products quality.

Building the plugin

Building a plugin using maven is easy, simply add the following to your pom:

 
<properties>
	<jmeter.home>installation directory of jmeter</jmeter.home>
</properties>
 
<dependencies>
	<dependency>
		<groupId>org.apache.jmeter</groupId>
		<artifactId>jmeter-core</artifactId>
		<version>2.3.4</version>
		<scope>system</scope>
		<systemPath>${jmeter.home}/lib/ext/ApacheJMeter_core.jar</systemPath>
	</dependency>
	<dependency>
		<groupId>org.apache.jmeter</groupId>
		<artifactId>jmeter-jorphan</artifactId>
		<version>2.3.4</version>
		<scope>system</scope>
		<systemPath>${jmeter.home}/lib/jorphan.jar</systemPath>
	</dependency>
</dependencies>

You probably want to put jmeter.home in your settings.xml instead.

Testing the plugin

Whenever I develop something I want to have a quick build/test cycle. In this case I want to to start the JMeter UI without having to update the JMeter installation or any other manual work. I didn't get chronos to add my plugin to the UI. It seems chronos copied the jar files to JMETER_HOME/lib/junit and this probably works for running an already created test, but not for creating the test. I also failed to the the maven exec plugin to create the command line arguments I needed. Instead I added a small ant script to start JMeter:

<build>
<plugins>
<plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-antrun-plugin</artifactId>
      <configuration>
        <tasks>
<property name="runtime_classpath"
           refid="maven.runtime.classpath" />
          <exec executable="${jmeter.home}/bin/jmeter.bat">
            <arg value="-Jsearch_paths=${runtime_classpath}" />
          </exec>
        </tasks>
      </configuration>
    </plugin>
  </plugins>
</build>

To run the jmeter gui simply run mvn antrun:run. This only works for Windows, I leave other platforms as an exercise to the reader .

Using the plugin

I ended up copying the plugin jar and dependencies to the JMeter installation and used then this modified JMeter distribution when performing the tests. The reason was that I didn't have maven installed on some of the test clients. To copy dependencies you can use mvn dependency:copy-dependencies.
If you want to run JMeter using maven you probably want to look at chronos (although it is not released it seems to work fine):

Using Maven Chronos Without an External JMeter Install

Automated performance testing using JMeter and Maven

I found a simple solution to a very common problem. While profiling I found that remove in this little method took a lot of the time.

   public void MovePropertyFirst(IProperty property)
   {
      properties.Remove(property);
      properties.Insert(0, property);
   }

The reason was, that it's time consuming to compare the properties by equality, for complex types. I think that in most (?) cases we want to delete the very same instance we supply as parameter to the remove method.

So I implemented a ReferenceEquals based remove, and by that I cut the time spent in remove by about 72 times!

  public static bool RemoveFrom<T>(this IList<T> list, T itemToRemove)
  {
    if(list.Count == 0)
      return false;
    int index = 0;
    foreach (var item in list)
    {
      if (ReferenceEquals(item, itemToRemove))
      {
        list.RemoveAt(index);
        return true;
      }
      index++;
    }
    Debug.Assert(false, "Item to delete not found");
    //equality based fall back
    return list.Remove(itemToRemove);
  }

Dependency injection and early validation is crucial for an enterprise application, and Spring is a boon in this regard. But do not fool yourself, Spring is good, but it is not a universal cure. Especially not in mobile space where startup times, low memory consumption, and avoiding interfaces are virtues.

The bottleneck in an enterprise application is most probably the database, spending a few extra milliseconds here and there seldom matters. On much less performant mobile device those clock cycles are not only time the user has to spend waiting, it will drain the battery. A simple thing as using an interface instead of an abstract superclass will perform at least twice as slow. Even passing an extra argument in the constructors a few nestings deep will have an impact.

Lazy Singeltons by Choice

Class loading in Java is lazy, the Java VM will not load classes until they are referenced. Odds are that the user will not trigger a full coverage of all classes in your app for a short time running mobile app. If the user just checks for incoming messages and quits, then every class involved in composing messages do not need to be loaded. Early dependency injection breaks this, as all classes will be referenced at startup. Most components will be initialized in vain, as they are never actually used.

So what we should do in mobile application is to work more like the Java VM, and less like Spring. If possible do not create a component until it is requested. The best way to do this is using a singleton pattern. Not a normal enforced singleton pattern, but rather a singleton by choice. Let the constructor be public, trust your users, and name the getter getSharedFoo(), not getInstance(). Let me show you using a example of an URL cache component:

public class URLCache {
  private static URLCache sharedCache;

  public static URLCache getSharedURLCache() {
    synchronized (URLCache.class) {
      if (sharedCache == null) {
        sharedCache = new URLCache();
      }
    }
    return sharedCache;
  }

  public URLCache() {
    // More code...
  }
  // Allot more code here...

}

Using this shared URL cache component from our imaginary HTTP provider would then be super easy, but not mandatory:

public class HTTPProvider {

  public InputStream inputStreamForURL(String url) {
    URLCache cache = URLCache.getSharedURLCache();
    // Use the cache...
  }

}

The big win here is that if the code path of this run of the application never tries to open an input stream then the URL cache never has to be created. Saving several hundred milliseconds of reading cache indexes, validation, etc.

But What About Testing?

Are not singletons bad for unit tests and mocking components? They used to be. These days we have PowerMock, you really should use it. Turns out that not even PowerMock is really required, if we just change our singleton pattern very slightly, and allows the outside to set the shared component as well:

public class URLCache {
  private static URLCache sharedCache;

  public static void setSharedURLCache(URLCache cache) {
    synchronized (URLCache.class) {
      sharedCache = cache;
    }
  }

  public static URLCache getSharedURLCache() {
    synchronized (URLCache.class) {
      if (sharedCache == null) {
        sharedCache = new URLCache();
      }
    }
    return sharedCache;
  }

  // Allot more code here...

}

This small addition allows us for setup up our own mock cache in the setup of our unit tests. With something as simple as this:

public class SomeTest extends TestCase {

  public void setUp() {
    URLCache.setSharedURLCache(new NoOpURLCache());
  }

  public void testRemoteResource() {
    assertNotNull(HTTPProvider.getSharedHTTPProvider().inputStreamForURL(TEST_URL));
  }
}

It also allows us for explicitly override a singleton if our application has special needs, perhaps a more aggressive cache on a low bandwidth mobile data connection, or special implementation on a buggy Java ME platform. But most importantly for the normal case it will not waste time creating the component until it is actually needed, vastly improving the user experience for your mobile users.

Take Aways

Do not fear singletons, it is a great way of lazy creations that will greatly improve startup times and memory consumption on mobile applications.

Avoid using interfaces, they are at least twice as slow as classes, and can not have static methods for acquiring the lazy created components.

Do not enforce the singleton pattern, always use singleton by choice. To allow for testing and easy implementation of special cases.

Any sane developer avoids premature optimization, so the task we later want to execute in a separate thread is almost always available in our current context. I am sure all Java-developers has seen something like this:

 new Thread() {
    public void run() {
        OuterClass.this.someMethod(arg);
    }
}.start();

And often an anonymous Runnable is involved as well. Executing a task in a separate thread with Cocoa and Cocoa Touch is ridiculously easy given the dynamic nature of Objective-C. This is done with Cocoa with this one-liner:

[self performSelectorInBackground:@selector(someMethod:) withObject:arg];

The performSelectorInBackground:withObject: is a companion method for performSelector:withObject:, and will create a new NSThread instance, and execute the specified method there. There are more companions such as performSelectorOnMainThread:withObject: that is used to execute a method on the main UI thread.

This easy access to executing in new threads often can result in code that spawns too many threads. In Cocoa on Mac OS X this is seldom a problem, not for Cocoa Touch applications under development when running in the iPhone simulator either. But on the more resource constrained iPhone and iPod Touch devices this will be a problem. Spawning a dozen background threads to download images will bring the iPhone OS to a staggering mess.

Update: I have removed CWSelectorOperation from this post, since using NSInvocationOperation is preferred.

Operation Queues

The solution is to use a NSOperationQueue, that as the name implies handles a queue of operations. The max number of concurrent operations can be set, as well as dependencies and priority. The downside is that NSOperationQueue can only handle instances of NSOperation.

NSOperation is an abstract class, where you as a developer override the main method to execute your task. In short NSOperation is programatically the equivalent of the Java Runnable interface. It is even worse as Objective-C do not support anonymous classes, so we will be required to actually implement our queued tasks as separate classes.

Not as neat as one would like, and definitely not even very Cocoa-like.

What is Missing

As a Cocoa developer I expect that executing a task on background queue, would be just as easy as executing a task on a background thread. The expected needed code is something like this:

[self performSelectorOnBackgroundQueue:@selector(someMethod:) withObject:arg];

To make this work we need:

1. A category on NSObject to add performSelectorOnBackgroundQueue:withObject:.
2. Implement a concrete subclass of NSOperation to support execution of a selector on any target.Use NSInvocationOperation to perform selector.
3. A category on NSOperationQueue to add a shared operation queue.

This is in the order of how a client developer would see it (most developers will only care for the NSObject category). For us this is the reverse order when implement this functionality.

A Shared Operation Queue

Cocoa and Cocoa Touch used many shared instances, for example a shared UIAccelerometer instance, and a shared NSURLCache instance. So this pattern of single point dependency injection is common in Cocoa and works very well. So let us follow suit by introducing an interface like this:

@interface NSOperationQueue (CWSharedQueue)
+(NSOperationQueue*)sharedOperationQueue;
+(void)setSharedOperationQueue:(NSOperationQueue*)operationQueue;
@end

The shared operation queue will be created lazily if it do not exist, and the client can easily swap it out at startup if needed. The implementation is very small, and a good example of how we should implement class variables with proper memory handling.

@implementation NSOperationQueue (CWSharedQueue)
 
static NSOperationQueue* cw_sharedOperationQueue = nil;
 
+(NSOperationQueue*)sharedOperationQueue;
{
  if (cw_sharedOperationQueue == nil) {
    cw_sharedOperationQueue = [[NSOperationQueue alloc] init];
    [cw_sharedOperationQueue setMaxConcurrentOperationCount:3];
  }
  return cw_sharedOperationQueue;
}
 
+(void)setSharedOperationQueue:(NSOperationQueue*)operationQueue;
{
  if (operationQueue != cw_sharedOperationQueue) {
    [cw_sharedOperationQueue release];
    cw_sharedOperationQueue = [operationQueue retain];
  }
}
 
@end

Enter CWSelectorOperation

Update: CWSelectorOperation is no longer needed, our implementation instead uses NSInvocationOperation.

Wrapping up NSObject

And at last we reach the category on NSObject, that is what we initially wanted, and what 95% of all use cases will be limited to.

@interface NSObject (CWSharedQueue)
 
-(NSInvocationOperation*)performSelectorInBackgroundQueue:(SEL)aSelector
    withObject:(id)arg;
-(NSInvocationOperation*)performSelectorInBackgroundQueue:(SEL)aSelector
    withObject:(id)arg dependencies:(NSArray*)dependencies
    priority:(NSOperationQueuePriority)priority;
 
@end

Most performSelector* methods do not have a return value, I have chosen to return the resulting NSInvocationOperation so that it can be used to setup dependencies for future queued operations. I have also added a second method performSelectorInBackgroundQueue:withObject:dependencies:priority: so that such dependencies can easily be setup, as well as prioritisation of operations.

The actual implementation is a simple wrapper around the NSInvocationOperation class, and category on NSOperationQueue that we implemented above:

@implementation NSObject (CWSharedQueue)
 
-(NSInvocationOperation*)performSelectorInBackgroundQueue:(SEL)aSelector
    withObject:(id)arg;
{
  NSInvocationOperation* operation = [[NSInvocationOperation alloc]
      initWithTarget:self selector:aSelector object:arg];
  [[NSOperationQueue sharedOperationQueue] addOperation:operation];
  return [operation autorelease];
}
 
-(NSInvocationOperation*)performSelectorInBackgroundQueue:(SEL)aSelector
    withObject:(id)arg dependencies:(NSArray*)dependencies
    priority:(NSOperationQueuePriority)priority;
{
  NSInvocationOperation* operation = [[NSInvocationOperation alloc]
      initWithTarget:self selector:aSelector object:arg];
  [operation setQueuePriority:priority];
  for (NSOperation* dependency in dependencies) {
    [operation addDependency:dependency];
  }
  [[NSOperationQueue sharedOperationQueue] addOperation:operation];
  return [operation autorelease];
}
 
@end

And that is it, if you want to avoid cut and paste of all this code to your own project just download the source code here. And happy concurrent hacking.

Download the older source code here, with a concrete subclass of NSOperation called CWSelectorOperation.

Or are regular expressions really missing? Regular expressions can be used with NSPredicate that is part of Core Data, available since Mac OS X 10.4 and officially announced for iPhone OS 3.0. Cocoa's WebView and the equivalent UIWebView in Cocoa Touch both support JavaScript with regular expressions. So there sure is regular expressions available on the platforms, but how do you make it available for your own code?

An Ugly Solution

You can actually get access to the regular expression engine through JavaScript, unfortunately this requires a roundtrip through WebKit. On an iPhone this means you have to use an off-screen instance of UIWebView, and delegate execution of regular expression to it.

The complexity of an off-screen WebView or UIWebView could be hidden by a utility class. But the extra glue code needed to make something useful out of the single method stringByEvaluatingJavaScripFromString:, would be allot.

What Apple Recommends

For most problems the official stance is correct; do not use regular expressions. Instead use NSScanner, that is perfect for sequentially parse texts. It is very fast and can substitute any regular expressions that only relies on:

• Character sets
• Exact string matches
• Numerical matches
• Uniform input text

These conditions hold true to 95% of everything regular expressions is ever used for. For the other 5%, Apple leaves you to fend for your own.

Other Solutions

PCRE compiles perfectly with Cocoa, since it is written in C, one of the many advantages of Objective-C. PCRE is very capable, and almost a standard, but also very large. For an iPhone application the PCRE implementation could end up as the majority of your executables file-size. If this is something you can live with, then the open source RegexKit framework wraps PCRE in Cocoa and Cocoa Touch friendly Objective-C.

Another regular expressions framework is OgreKit. The advantage of OgreKit is full unicode support, with the same disadvantage of size. And the fact that the documentation is in Japanese.

A Pretty Solution

It turns out that Mac OS X for years, and iPhone OS since inception, has been shipped with a perfectly good regular expressions engine. This engine is based on the ICU specification, so it works perfectly with unicode and is well on par with PCRE for functionality. This framework is simply called ICU Core, and has a C interface. But for a Cocoa programmers the C interface is not nice enough, and thankfully John Engelhart has done this work for us, with RegexKitLite. RegexKitLite is a little brother to RegexKit that wrapps ICU Core instead of PCRE.

RegexKitLite is published under BSD license, and is simply two files you add to your project, fully compatible with all available versions of both Mac OS X and iPhone OS. The tricky part is that ICU Core is not a public API officially supported by Apple, even though it has existed unchanged for years. Good news is that using ICU Core is not a show stopper for publishing on the iPhone App Store, application out there already uses it, both well known and not so well known.

Setting Up RegexKitLite

1. Download the latest version from the sourceforge webpage, or SVN.
2. Add RegexKitLite.h and RegexKitLite.m to your project.
3. Link your project against ICU Core, by adding the linker flag -licucore to Other Linker Flags under your projects build settings.

Optionally you can also add the documentation to Xcode with these easy steps:

1. Open Help -> Documtantion.
2. Press the Gears button in the lower left corder, and select New Subscription....
3. Enter feed://regexkit.sourceforge.net/RegexKitLiteDocSets.atom as URL.

Using RegexKitLite

This post is not a tutorial on regular expressions, but a tutorial on a partical API for executing regular expressions. If you want to learn more about regular expressions themselves I would recomend you look at Regular Expressions.info.

RegexKitLite provides it's functionality as categories on NSString and NSMutableString. This way using regular expressions with Cocoa is just as easy and normal string manipulation. This is best described using examples.

A simple example that normalizes a text with single white spaces, kind of like how a HML renderer would do, so this is handy when scraping web pages:

NSString* source = @"One\t Two \nThree ";
NSString* result = [source stringByReplacingOccurancesOfRegex:@"\\s+"
    withString:@" "];
NSLog(source);
NSLog(result);

Or you can split a text, such as semi-colon delimeted data:

NSString* source = @"Test;12;Y";
NSArray* columns = [source componentsSeparatedByRegex:@";\\s*"];
NSLog([columns description]);

And you can extract more complex data using capture groups:

NSString* source = @"<foo no=\"12\">Name</foo>";
NSString* regex = @"<foo no=\"(.+?)\">(.*?)</foo>";
int no = [[source stringByMatching:regex capture:1] intValue];
NSString* data = [source stringByMatching:regex capture:2];
NSLog(@"no: %d data: %@", no, data);

This may look like it could be slow to perform matches on the same regular expression twice, but it is not. RegexKitLite is very smart, and will cache your previous matches for very high performance.

RegexKitLite is a very capable, and also much active open source project, with version 3.0 as a release candidate in SVN. Use it, and use it well.

NSArray admits to sorts being a slow operation, and adds a method pair for comultive sorts using hints. This way the operation is done in O(P*LOG(P)+N) time, instead of O(N*LOG(N)). Where N is number of elements, and P is number of additions and deletions since the last sort. Unfortunately that do not work on NSMutableArray. So even if memory consumption will not hit the roof, release retain cycles will take it's toll.

So why not add methods to find the insertion points, and insert new objects into already sorted NSArray and NSMutableArray object? Best case for inserting single elements should be O(LOG(N)^2), so lets hit that target. And on the way there, we will learn how to;

• Add functionality to standard classes using categories.
• Implement high performant Obj-C code for tight loops.

New Categories on NSArray and NSMutableArray

This will become a bit complex on the inside, but the public front should be as simple as this:

#import <Foundation/Foundation.h>
 
@interface NSArray (CWSortedInsert)
 
-(NSUInteger)indexForInsertingObject:(id)anObject
    sortedUsingfunction:(NSInteger (*)(id, id, void *))compare
    context:(void*)context;
-(NSUInteger)indexForInsertingObject:(id)anObject
    sortedUsingSelector:(SEL)aSelector;
-(NSUInteger)indexForInsertingObject:(id)anObject
    sortedUsingDescriptors:(NSArray*)descriptors;
 
@end
 
@interface NSMutableArray (CWSortedInsert)
 
-(void)insertObject:(id)anObject
    sortedUsingfunction:(NSInteger (*)(id, id, void *))compare
    context:(void*)context;
-(void)insertObject:(id)anObject sortedUsingSelector:(SEL)aSelector;
-(void)insertObject:(id)anObject sortedUsingDescriptors:(NSArray*)descriptors;
 
@end

Base Implementation

We want to follow suit with NSArray and NSMutableArray, so we too shall support inserts based on functions, selectors, and an array of sort descriptors. In practice we only need to implement indexForInsertingObject:sortedUsingfunction:context:, the other methods can use this one, each with a private compare method of their own. So we begin by implementing this method.

#import "NSArray+CWSortedInsert.h"
#import <objc/runtime.h>
 
@implementation NSArray (CWSortedInsert)
 
-(NSUInteger)indexForInsertingObject:(id)anObject
    sortedUsingfunction:(NSInteger (*)(id, id, void *))compare
    context:(void*)context;
{
  NSUInteger index = 0;
  NSUInteger topIndex = [self count];
  IMP objectAtIndexImp = [self methodForSelector:@selector(objectAtIndex:)];
  while (index < topIndex) {
    NSUInteger midIndex = (index + topIndex) / 2;
    id testObject = objectAtIndexImp(self, @selector(objectAtIndex:), midIndex);
    if (compare(anObject, testObject, context) < 0) {
      index = midIndex + 1;
    } else {
      topIndex = midIndex;
    }
  }
  return index;
}
 
// More code here...
 
@end

The Objective-C runtime header is included so we can go around the usual method dispatch, and instead use the method implementations as function pointers. In most cases this is a neglectable optimization, for this case we will be the bottle neck since we are intended to be the low level API.

The algorithm is a simple binary search, and we have a worst case O(LOG(N)^2) time for index search. Using this for implementing the sibling method in NSMutableArray is as simple as this:

@implementation NSMutableArray (CWSortedInsert)
 
-(void)insertObject:(id)anObject
    sortedUsingfunction:(NSInteger (*)(id, id, void *))compare
    context:(void*)context;
{
  NSUInteger index = [self indexForInsertingObject:anObject
      sortedUsingfunction:compare context:context];
  [self insertObject:anObject atIndex:index];
}
 
// More code here...
 
@end

Sort Using Selector

With functions in place, writing a utility compare function that uses selectors, and can be used with our indexForInsertingObject:sortedUsingfunction:context: method is a breeze.

static NSComparisonResult cw_SelectorCompare(id a, id b, void* aSelector) {
  return (NSComparisonResult)objc_msgSend(a, (SEL)aSelector, b);
}
 
-(NSUInteger)indexForInsertingObject:(id)anObject
    sortedUsingSelector:(SEL)aSelector;
{
  return [self indexForInsertingObject:anObject
      sortedUsingfunction:&cw_SelectorCompare context:aSelector];
}

Notice that objc_msgSend(id, SEL, ...) is used instead of the performSelector:withObject: method. This is a small performance improvement, as what performSelector:withObject: do behind the scene is just a call to objc_msgSend() anyway. We have removed one method invokation per compare.

Sort Using Sort Descriptors

Or rather sort with an array or NSSortDescriptor objects. This will be the least performant, but also the most flexible sort. An array of sort descriptors is used because we want to be able to sort using several criteria, one sort descriptor per criteria. For example when sorting on last name, then first name, or even more complex sorts.

A NSSortDescriptor is initialized with a key, sort order (ascending or descending) and optionally a selector to do the comparison with, by default compare:. There are quite a few method invokations involved, and it will be good for performance to avoid as many as we can.

Just as performSelector:withObject: uses objc_msgSend() behind the scene, so do objc_msgSend use a function pointer internally. After doing a hash-table lookup with the selector as key that is. This look-up is cached and very fast, but avoiding it all together is always faster. So let us fetch those when the category is first loaded, and do our calls straight through the functions pointers avoiding at least two or more function calls for each compare operation.

static IMP cw_compareObjectToObjectImp = NULL;
static IMP cw_ascendingImp = NULL;
 
+(void)initialize;
{
  cw_compareObjectToObjectImp = [NSSortDescriptor
      instanceMethodForSelector:@selector(compareObject:toObject:)];
  cw_ascendingImp = [NSSortDescriptor
      instanceMethodForSelector:@selector(ascending)];
}

The class method inilialize is kind of magic, it is called once for each class and category, when the class or category is loaded into the run-time. It is the Obj-C equivalent of static initializers.

With the function pointers to the hot-spot method implementations in hand, here is the fast implementation:

static NSComparisonResult cw_DescriptorCompare(id a, id b, void* descriptors) {
  NSComparisonResult result = NSOrderedSame;
  for (NSSortDescriptor* sortDescriptor in (NSArray*)descriptors) {
    result = (NSComparisonResult)cw_compareObjectToObjectImp(sortDescriptor,
        @selector(compareObject:toObject:), a, b);
    if (result != NSOrderedSame) {
      if (!cw_ascendingImp(sortDescriptor, @selector(ascending))) {
      	result = 0 - result;
      }
      break;
    }
  }
  return result;
}
 
-(NSUInteger)indexForInsertingObject:(id)anObject
    sortedUsingDescriptors:(NSArray*)descriptors;
{
  return [self indexForInsertingObject:anObject
      sortedUsingfunction:&cw_DescriptorCompare context:descriptors];
}

Conclusions

The fact that Objective-C is a strict superset of C have some good benefits. The obvious one is that your Obj-C code is both source-code and binary compatible with the vast collection of code that has been written in C over the past 30+ years.

The second benefit is that when performance is needed, Obj-C gives us as developers full access to the internals of the dynamic run-time.

But best of all, most developers never need to care. They can, blissfully unaware, reap the benefits of the performance tuning under the hood. Tuning by both Apple and considerate third party library developers willing to go the extra mile (or a few metres at least).

The code is released under BSD license, and works on both Mac OS X and iPhone OS. You can download it here.

On the Mac it was easy to get running (if you have MacPorts):

% sudo port install squid
% sudo squid -z (initialize stuff if you never ran Squid before)
% sudo squid -N -d 1 (no daemon, debug level 1)

Then it was running on the console, and you could test it like this:

% squidclient http://google.com
...
X-Cache: MISS from myhostname
Via: 1.0 myhostname:3128 (squid/2.7.STABLE2)

And again, this time fetching from the cache:

% squidclient http://google.com
...
X-Cache: HIT from myhostname
Via: 1.0 myhostname:3128 (squid/2.7.STABLE2)

You configure your browser by simply setting proxy server to localhost:3128, if you are running your Squid locally, that is.

Maven is probably best configured in ~/.m2/settings.xml:

<settings xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/xmlSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
    http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <proxies>
    <proxy>
      <active>true</active>
      <protocol>http</protocol>
      <host>localhost</host>
      <port>3128</port>
    </proxy>
  </proxies>
</settings>