UPDATE: There is now a live CodePlex project for the realization of this concept; A .NET Client Library for Neo4j over HTTP using REST and JSON; http://neo4jrestsharp.codeplex.com/

Intro

Neo Technologies have come out with a Neo4j REST API for their popular Neo4j graph database. Since I am employed at Jayway and a couple of colleagues working with Neo are too a little bird by the name of Peter Neubauer tweeted in my ear that there was a free T-Shirt involved in making a working sample for .NET that communicated with the Neo4j server. Not only was that a handsome offer – I wanted to check out what the Architecture of such a client library for a graph database would look like.

I’ve recently been involved in client library abstractions and good architecture with the RESTful interface to Windows Azure Storage. (More on Azure and storage on my blog here, here, here, here and here.) Enough with the side tracking (sorry)!

Said and done; We set out to code up a working sample for .NET. I hacked the client code. The Neo-guys cheered me on and offered very helpful assistance when I ran into trouble or was just plane stumped on how to proceed with the API. A very nice team effort, including a remote team debugging session, later and we have a small and running POC

Tweeting on this effort it is notable how a bunch of people answer back with an interest in this idea for potential applications in the .NET world. I’ve been approached both on twitter (@noopman) and on LinkedIn with questions on this.

This blog post is to submit our findings to the world and hope that someone will be inspired to get going with a real .NET client library for Neo4j.

Architecture

First let’s look at the architecture for creating a client library for a RESTful service. This is a more general discussion that leads to our client library POC below. If your main reason for reading is the actual sample; scroll down.

When doing a client library abstractions and domain thinking comes into play. What we want is for the forgotten users (developers) of your application to work with their normal domain objects in the business code and for them to use an API that makes sense to their domain while talking to a highly specific graph database backend store. This means you have to keep all the RESTfulness, HTTP and json away from the domain code. Let me show you a sketch and see if I can make this explanation a bit less wordy:

At the other side of the Cloud from our .NET code is a Java based graph database. It is friendly enough to expose a RESTful API so that we may speak to it.

Doing so we want to create a client library that is capable of a couple of things:1) Taking care of serializing objects to json (and deserializing them back again). 2) Knowing about HTTP requests and how to handle them. In our sample we employed the power of the HttpWebRequest class to handle things like HTTP Headers, Content Types, URIs etc. 3) Finally this library must understand to expose an interface to your code that you can use from a specialized Data Access Component.

To protect the business layer from knowing anything about the specifics of the Neo4j API you really should use a specialized Data Access Component. There is now law being broken if you don’t do that. However a good rule of thumb is to add abstraction layers in your system to shield your domain code in your business components from any specifics of an underlying storage structure. The details of serialization of objects to json and making requests over HTTP are not a natural fit to the business problem you are solving. The code that does this separation is code you write yourself in your application. Only you know how to adapt a foreign library and technology to your domain. The sole purpose of this code, in other words, is to abstract away from your business logic the underlying technologies used to persist and query data used in the application. This component translates between the two worlds of your business logic and the techniques used to store data in a graph database. Most of the specific technical details of storage are wrapped up in the Client Lib mind you but the Data Access Layer DAL knows about this problem domain. It understands that the persistence indeed is a REST based service over HTTP with all that this entails. This component in your code can handle synchonisity and take care of specific errors or faults that may occur when using the storage. All of this is done in a language that is fairly close to the API that Neo4j exposes. In the end this component translates into terms useful to the domain in which your business operates. Instead of saying .PostNode() in your code perhaps you instead expose a .StorePerson() action.

What we achieve is a good and sound (and classic) layering model: Business Logic – DAL – Client Lib.

There is one more observation to make in the picture and that is the matter of IoC - Inversion of Control. In order to keep your code testable, maintainable and untangled from the implementations at lower layers you should always employ the method of Inversion of Control – making your business code for instance not reference the DAL direct but instead use a common contract to do so. The Depencency Inversion Principle states briefly that a higher layer should never depend direct on a lower layer but instead depend on an abstraction. If this is true you are able to test your business logic without using the actual underlying storage. This is for instance key in a TDD – Test Driven Development development style.

Code that depends on abstractions can be tested very quickly and does not depend on specific data in a data store to be correct in order to test the intended behavior. The Business Logic uses a contract for the DAL and the DAL implements it. Then you use something called an IoC container (read on at Wikipedia in the link above) to create the DAL instances and hand them to the Business Logic employing the technique of DI -Dependency Injection. This separation enables you to modify, re-deploy or even exchange DAL components without having to touch your Business Layer at all. If separating the Business Logic from the DAL using contracts is important it is almost event more important to do so in the lower layer that in run time speaks direct to the data store! You never want to end up in the situation where in order to test your DAL code you have to have access to a specific data store with specific data in it. Not only is that an inconvenience; it also makes executing your tests run as slow as transferring data over the Internet compared to as fast as your CPU can execute code. From experience it is an obvious conclusion that tests that are slow to run are not in fact run. The conclusion from all of these IoC discussions can only be that it is imperative that your Client Library comes with an abstraction layer.

What did we do with the Architecture in our POC?

All of these architectural considerations are of course premature in a POC. This is why in our sample we did not really separate the DAL code from the Client Lib. However to prove the point of IoC and DI I did however separate out the Business Logic from the Data Access logic using a simple contract and an equally simple Factory Method Pattern. Instead of new:ing up a DAL component in my code I called a factory method to access an abstraction of the DAL.

Our POC sample

Our code sample creates a small node graph with a few relationships. Then it makes a traversal of the graph to find all who “knows” someone according to the relationships. Finally it deletes the graph again.

Here is the graph we create in our sample: http://wiki.neo4j.org/content/The_Matrix We only create names on nodes and types on relationships in our simple sample.

And here is also the code so that you may download and follow along with our explanation your self:

First, in accordance with architectural considerations (above) we get hold of our DAL using a factory method. We also create a new Person object and store him (in this case Thomas Anderson – Neo):

INeo4jStorageClient neo4JStorageClient =
Neo4jStorageFactory.GetNodeStorageClient(neo4jServerUri);

neo = new Person("Thomas Anderson");
neo4JStorageClient.StoreNode(neo);

As you can see in our business code this is just standard .NET classes being instantiated and a “Neo” Person is stored in a very normal way. In the business logic code there is no real details on how this storage happens. There is of course a “node” concept and you can consider for your self if that should be hidden away from business code too.

Since this is a POC the implementation of .StoreNode(neo) is very very trivial. If this were a real client library we would have included things like a library to transform (serialize) the neo-Person node into a json string. Now we just do it statically:

public void StoreNode(Person person)
{
    var name = person.Name;

    string jsonString = @"{""Name"":""" + name + @"""}";

    var nodeProtocol = new NodeProtocol(uri);

    var nodeId = nodeProtocol.PostNode(jsonString);
    person.Id = nodeId;
}

The interesting part here is the existence of a NodeProtocol implementation. I’ve found it useful working with a remote API like Windows Azure Storage and also with this Neo4j REST API to implement methods in C# code that correlate very closely to the API methods available. In our case we have a PostNode operation on the REST API and consequently we have a .PostNode() method in the C# implementation of that HTTP method call.

public int PostNode(string nodeData)
{
    var uri = HttpHelper.ConcatUri(baseUri, "node");

    HttpWebRequest request =
        Request.CreateWebRequest(HttpWebRequestMethod.POST, uri);

    HttpHelper.AddBody(request, nodeData);

    var response = requestMaker.MakeRequest(request);

    return HttpHelper.FindNodeId(response.Item3);
}

I won’t bore you with the exact details of all of the helper classes. To dig in to those it is better if you just download the code and look at the details your self.

As you can see we create a HttpWebRequest, add some headers to it (in the CreateWebRequest method) specifying things like request.Accept = "application/json"; meaning we accept a json formatted response. Also we specify that the body we send out is formatted as json with request.ContentType = "application/json";. Finally we make the call and return the id of the node we just created.

Again this is a solution bordering on triviality but still it is to the point of making a REST request and receiving a response. Also there is no code to handle any errors present here and the request is done synchronously.

What does the actual HTTP REST and json request and response look like for this simple sample?

Request:

POST http://try.neo4j.org:9999/node HTTP/1.1
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 26
Expect: 100-continue
Connection: Keep-Alive


{"Name":"Thomas Anderson"}

Response (note that the added line breaks in the response body were added by me):

HTTP/1.1 201 Created
server: grizzly/1.8.1
Content-Encoding: UTF-8
Location: http://try.neo4j.org:9999/node/1

Content-Type: application/json
Content-Length: 860
Date: Fri, 16 Apr 2010 08:36:45 GMT

{"incoming typed relationships":

http://try.neo4j.org:9999/node/1/relationships/in/{-list|&|types},

"incoming relationships":http://try.neo4j.org:9999/node/1/relationships/in,

"all relationships":http://try.neo4j.org:9999/node/1/relationships/all,

"create relationship":http://try.neo4j.org:9999/node/1/relationships,

"data":{"Name":"Thomas Anderson"},

"traverse":http://try.neo4j.org:9999/node/1/traverse/{returnType},

"property":http://try.neo4j.org:9999/node/1/properties/{key},

"self":http://try.neo4j.org:9999/node/1,

"properties":http://try.neo4j.org:9999/node/1/properties,

"all typed relationships":

http://try.neo4j.org:9999/node/1/relationships/all/{-list|&|types},

"outgoing typed relationships":

http://try.neo4j.org:9999/node/1/relationships/out/{-list|&|types},

"outgoing relationships":http://try.neo4j.org:9999/node/1/relationships/out}

As you can see the Neo4j REST API is self explanatory. Creating a node gives you back a response with all the things you can now do with this new node over the API. Since this quite a lot of data when the only thing that really matters is in the header (Location: http://try.neo4j.org:9999/node/1) I’ve already given feedback to the Neo team that I’d like a simple Uri switch (like APIDocumentation=0) to shut off the detailed response. But in fact I do like that you get so many Uris back that you may use direct. Depending on your needs this is either good or to verbose a payload to get back in your response. As you can see the node we created got the Id set to 1.

Now we proceed to create all the other nodes and in a very similar fashion the relationships between nodes. Here is the relationship that says that “Neo knows Trinity” (who has a node id of 2). This is a POST operation to /node/1/relationships:

POST http://try.neo4j.org:9999/node/1/relationships HTTP/1.1
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 56
Expect: 100-continue

{"type":"Knows","to":http://try.neo4j.org:9999/node/2}

Once the node graph for the Matrix is created we proceed to make a traversal of the graph. We want to find anyone who knows someone throughout the graph.

The request looks like this in REST (after a similar path through the code as above with a call from business logic to DAL and through to the client lib that makes the actual call using REST):

POST http://try.neo4j.org:9999/node/1/traverse/node HTTP/1.1
Accept: application/json
Content-Type: application/json
Host: try.neo4j.org:9999
Content-Length: 57
Expect: 100-continue
Connection: Keep-Alive

{ "relationships": { "type": "Knows" }, "max depth": 10 }

Again we simplified to a “max depth” of 10 relationships deep into the graph which works fine for our sample.

The result is a list of all the nodes that match. Because of the self explanatory API mentioned above the result also gives you actions to perform on each node that comes back in the response thus making the response body quite long. Rather than pasting the whole thing here into the blog post you can try it out your self (by downloading the application) and you will get a result exactly like this from the app:

Neo knows Trinity and Morpheus, who knows Cypher, who in turn knows Agent Smith.

Difficulties

We had only two real challenges in coding this POC.

The first one was an issue with streams and the optional so called Byte Order Mark (BOM) that you can put at the head of a stream. I’ve posted on this here: Would you like a Byte Order Mark to go with that?.

The other challenge was that I wrote a small app for Windows to show our results. But the guys on the Neo team are certainly not Windows users. The wanted to show the code on Mono. Just to be sure of compatibility I scaled back to a good old Windows Forms demo app in .NET Framework 3.0. This worked great on Mono which was our intent. But the drawback was that the library I’d found to handle object serialization didn’t respond well to that old version of the framework. That’s why I replaced my lib with static string creations (as you can see in the implementation of the .StoreNode() method above).

What have we learned from this?

First of all creating a client library for .NET to speak REST and json over HTTP to Neo4j is very doable; quite easy and straight forward.

Second (and that’s just a word of warning); keep your layers separated. This goes for all applications I guess but if you start to put code to create instances of web requests with web methods and content types and such in your business code you will end up in a very tangled and messy situation very fast which will be a pain and a high cost to maintain.

Finally we have considered looking into using WCF data services (and OData – the Open Data Protocol) and also possibly Linq – Language Integrated Query on the client side to make implementations in the library even more powerful and useful. And we do need a object to json serializer library.

Oh and here is another post on using the Neo4j REST API by the team itself: http://blog.neo4j.org/2010/04/neo4j-rest-server-part1-get-it-going.html

That’s it!

We hope this is inspirational enough for you to get going with your own samples or even client libraries in .NET to speak to Neo4j. If you do; please get back in touch with us!

Cheers,

M.

Context

Recently Neo4j graph database announced a REST interface. A couple of colleagues at Jayway are working on this. This was (very much) a too intriguing temptation to resist. I naturally had to write a small POC .NET based REST client to be able to talk to that server. (Blog post on this is pending within a few days.) While trying to send REST data in HTTP requests to the server we came across something interesting that caused compatibility issues between the Java and .NET worlds. The culprit was that optional byte order mark that I unintentionally embedded in my streams of bytes. The server did not recognize them. In a joint effort from both sides of the world (.NET client and Java server) we were able to find and identify the problem. Not that this is a new problem - It is very well documented in Wikipedia (link above). However I feel that one more post on this topic with a good test to show what’s going on is warranted.

A very good tool was also used to nail down the problem; Fiddler2. This little baby of a program is an http proxy that sits on your machine and listens to your http traffic. Using the tool you can inspect exactly what you are sending over the wire in your http requests and also view the responses coming back. Sweet stuff indeed. This is a MUST tool for you if you develop anything that sends data over HTTP.

Using Fiddler2 we were able to observe that when sending a stream of bytes as a body for an http POST the byte length, when the request came from my code, was 108 bytes. However when I used the built in request builder in Fiddler2 to reproduce and resend the exact same request it was reported that the length was 105. Why was there a difference in three (3) bytes?

Problem was that these tree bytes, as I stated above, caused the server to throw a fit. It could not recognize the body of the message to be the expected json string in UTF8 encoding. Json, I’ve learned, is always in UTF8 encoding.

The java guys googled and I binged ;~) and we found that these three bytes is an optional “byte order mark” or BOM. It is not wrong to send these three bytes and it is not wrong not to send them. Guess what? .The .NET world tends to send these extra bytes and everyone else tends not to. Why not have a bit of more incompatibility, right? It’s not like we want to talk to each other anyway!

What’s up with the BOM? What is it’s purpose?

The BOM is a way to add a kind of metadata inside of a stream of bytes instead of sending some actual metadata along next to your stream. Option one is saying “Here is my byte stream and btw it is in UTF8 encoding”. The other way to do it is saying “Here is my byte stream and if you look at the first three bytes you can read the encoding of it”. Which is better? I can’t say I care all that much other than the fact that it caused us a problem when I tried to send requests from .NET code to the Neo4j server.

How can you then – finally – handle these three bytes?

Well that depends… on what you intend to do. What I can show you is a piece of code that tells you exactly what this is and then you can copy that behavior into your code and modify it to serve your purpose. The code below should be pretty self explanatory but just to be sure. I take the string "foo" and encode it into a MemoryStream using a StreamWriter. The issue is that I tell the stream writer to use Encoding.UTF8 and this is where .NET Framework adds the BOM. The resulting stream is not 3 bytes as perhaps expected. Instead it is 6 bytes long. What you have to do if you read this stream ‘raw’ – by hand – in some library, is skip over the first three bytes. A better way to do it is to use a reader that handles the BOM. Finally if you don’t want to add the BOM in the first place you can write your bytes yourself in a more ‘raw’ fashion byte by byte to the stream. As you can see .NET Framework is good enough to have a way to find an actual BOM for different encodings. The UTF8 encoding does it this way: Encoding.UTF8.GetPreamble() (msdn library link to .GetPremable()).

[TestMethod]
public void StreamWriter_with_encoding_writes_ByteOrderMark()
{
    string str = "foo";

    var memStream = new MemoryStream();

    // Test
    var writer = new StreamWriter(memStream, Encoding.UTF8);
    writer.Write(str);
    writer.Flush();

    Assert.AreEqual(6, memStream.Length);

    var buffer = new byte[3];

    memStream.Position = 0;
    memStream.Read(buffer, 0, 3);
    string result = Encoding.UTF8.GetString(buffer);

    var preamble = Encoding.UTF8.GetPreamble();
    var byteOrderMarkUtf8 = Encoding.UTF8.GetString(preamble);

    Assert.AreEqual(byteOrderMarkUtf8, result);

    memStream.Position = 3;
    memStream.Read(buffer, 0, 3);
    result = Encoding.UTF8.GetString(buffer);

    Assert.AreEqual(str, result);

    // Alternative: Write the bytes yourself to skip the BOM

    memStream = new MemoryStream();

    foreach (var @byte in Encoding.UTF8.GetBytes(str))
    {
        memStream.WriteByte(@byte);
    }

    Assert.AreEqual(3, memStream.Length);
}

You can also if you like compare bytes for the premable one by one rather than converting to a string comparison.

As you can see this BOM can be handled easily if you like. The thing is I did not have a clue that it was there.

Oh – and btw now the Neo4j server accepts REST json bodys that both can have and skip the BOM! Good ‘bug’ to solve or ‘feature’ to have.

Cheers,

M.

Summary

With connected devices, JavaScript enabled web sites and the extensibility of the XMPP protocol, we are at the beginning of a new kind of applications that are operating in the social context of their owners. Google Wave is just the first well known application doing other-than-chatting things on top of XMPP. LinkedProcess is trying to lay a foundation for a much more general computing and distribution infrastructure for context-aware, social-spreadable and trusted application scenarios that potentially enable users to regain control over their own graph of information and control access to it using social connections.

What is XMPP?

XMPP is an XML instant messaging protocol that was developed by the Jabber foundation, as a reaction to the closed protocols of ICQ, Microsoft Messenger etc etc. It is based on the concepts of

• server side contact lists (Roster in XMPP terminology)
• open XML streams between the XMPP server and any connected client once the connection is established
• every connected XMPP client gets a unique ID (e.g. "neubauer.peter@gmail.com/phone1/xyz") under the common XMPP "bare" ID (neubauer.peter@gmail.com) and is uniquely addressable by XMPP packets. This can be used in situations where there is no direct IP number available or it is changing (e.g. mobile devices), making any Internet-connected device addressable as a potential server, with a constant address over time and space - an URI.
• extensible payloads of XML packets as defined by the XMPP Specification, the most important reflecting the 3 paradigms of asynchronous communication, and supported by all major XMPP servers
  • broadcast ("<presence>")
  • one-to-one or one-to-many ("<message>")
  • request/response with acknowledgement ("<iq>"), similar to the HTTP protocol
• handling of status, presence of resources and routing of messages to the right receivers by the XMPP servers
• from the beginning built to scale for the Internet with distributed infrastructure, as explained in this blog post
• large existing XMPP networks to tap into (GoogleTalk, iChat, Facebook Connect, holding part of your existing social graph in the contact lists) and open XMPP servers to set up your own XMPP network
• SSL support built into the base specification, distributed server-to-server communication even between XMPP networks like GoogleTalk and iChat.

XMPP principle setup

Why should I care about XMPP?

• XMPP is the currently most popular infrastructure for instant communication, used by systems like iChat, Google Talk and Google Wave. These are just a few of the possible applications we can do on top of XMPP, Twitter and TiVo are rumored to follow
• via the use of a server side contact list of peers in any XMPP communication, suddenly there is a social or trust-like aspect to any XMPP conversation between two or more participating systems, users or devices
• any modern mobile phone, javascript-enabled website, J2ME, Java or Flash-capable home automation device (e.g. BluRay players or media servers) or server application can talk XMPP and thus become a true peer to the other clients, without any new server side infrastructure or cloud service development more than existing XMPP servers for routing via e.g. http://www.kaazing.com, HTML5 web sockets or various XMPP libraries for JavaScript, ActionScript, for an XMPP server and Smack for Java.

What can we do with this?

Now, with all these characteristics in place, we can look at the different emerging platforms for XMPP communication:

Chat - simple text exchange

This is the simple tracking of presence for XMPP clients and the exchange of only human-readable text messages between them. However, a certain coolness is added by being able to distinguish between different clients logged in under the same XMPP account (e.g. different browser windows logging into GTalk and still sending and receiving in the right window). This has been in use for the past 15 years and is nothing new, even if the clients have gotten lighter and fancier over the years (see figure, marie@neubauer.se talking from the browser to neubauer.peter@gmail.com on iChat).

Normal XMPP chat setup

Google Wave - concurrent document editing on steroids

Google Wave is introducing a concept where a central XML document (the Wave), reflecting the total of a conversation, is simultaneously edited by several participants (via XMPP clients) of the conversation at the same time. While the incremental changes are sent to the XMPP Google Wave server for merging into the central Wave, they are at the same time even sent to the other online clients in the Roster of the current conversation just as chat messages, creating the real-time response impression of the Google Wave experience. However, these packages do not contain simple chat but actual machine-data interpreted by special Google Wave clients in e.g. JavaScript (see the Wave Protocol).
Google Wave even supports extensions to its functionality, see this blog post which means that a lot of the infrastructure can be used to e.g. support new functionality and data packets.

Google Wave setup

LinkedProcess - turn any connected device into a server with a social context

LinkedProcess (LoP), driven by Marko Rodriguez and Joshua Shinavier is taking the approach of providing a set of XMPP tags for sending scripts and code snippets from XMPP clients ("LoP Villeins") to other XMPP peers acting as servers ("LoP Farms"), get them executed there and the result sent back. The thought is to be able to speed up the processing aspect of the emerging Semantic Web/Web of Data by moving the processes to the data instead of the usual move of the data to the processor.
One huge application is the processing of the emerging Semantic Web or Linked Data (Tim Berners Lee talks about that at TED 2009) by sending processing instructions to the big servers holding large chunks of this emerging graph structure instead of having to pull all data onto your own device prior to processing it (see the LinkedProcess introduction).

LinkedProcess distributed processing of LinkedData setup

This has a number of interesting aspects:

• any XMPP capable client (mobile phones, websites etc) can be turned into a server being in direct contact with its peers
• data can stay at one logical place (e.g. the mobile phone) while others can send scripts to operate to the data in order to get ad-hoc queries executed
• code can be spread within the social context, so scripts can be shared and spread through the social network of resources and people rapidly, without special installation hassle, giving birth to truly viral "web- and mobile only" applications and social networks of both humans, content and hardware
• the access to the computing resources and data is controlled via the contact list of the XMPP client executing the script. Thus, it is always known who is trying to access the LinkedProcess server
• via the use of existing or new social chat networks, this extends the social aspect of your contact list to any XMPP speaking resource under your account, persons, devices and software alike.

So, what is social about LinkedProcess?

Well, with all this said, we have suddenly a couple of very interesting concrete scenarios:

1. Personal and Social Could Computing

This is the concept of spreading code to execute on other servers and devices that are available within the social context of the sender - either a number of servers being set up under the same bare account or friends and colleagues providing their resources to others in their social vicinity. E.g. if I want to transcode a HD movie from TED to fit my mobile phone, I could send a script to my home server instructing the server to download the appropriate encoder, download the original high-resolution link and put the result onto a URL and return that to me. Totally Ad Hoc, without any new programs installed in my phone or the server afterwards.

The Social Cloud

2. Totally distributed and viral social apps - my personal Facebook

A user writes, downloads or is recommended a script - "SOClet" (written in e.g. LUA or Python for Android phones, see Android Sctripting Environment) that asks all its friends phones to evaluate if there is any content related to the asking phones owner. If yes, return the URIs of matching content on the friends phones. That results in an aggregated feed on "Pictures of me by my friends, friends statuses etc" - a Facebook- like setup, totally up to every individual to customize and no central storage. Of course if desired there can be a central server as an equal XMPP client in the middle in order to help holding state, much like the Google Wave setup.

Viral apps spreading in the social device context across mobile devices and e.g. web pages

3. Installing servers on phones and other resources

The technical use setup is as the previous one, but here, long running server code is executed on your friends devices and thus, we create new services on top of the LinkedProcess infrastructure, and turn the devices into mini-servers. This serves potential use cases like Opera Unite, although not with a central namespace like Unite but totally distributed. Peter could for instalnce install a script on Maries phone serving an RSS feed back to Peter when he asks for new content every 5 minutes, including the current location. This can easily be displayed as a GeoRSS over a map on Peters phone or used to mash up the feed with other information using e.g. Yahoo Pipes. This could even be done from a web page (e.g. in Facebook), resulting in a Facebook application that talks to the phone directly and can mash up live phone data with the rest of the Facebook data.

4. Privacy and ownership of everybody's personal information

By turning the physical holder of your own data - the phone or your home desktop - into a server that accepts or denies requests for operations on that data, users regain control over the personal data. Imagining different protection zones (private, business, open), access can then be granted via the social contact list to individuals, groups of contacts (e.g. friends, co-workers in a project) or anyone.
If unknown or public clients like e.g. Google bots want to access that information, even commercial transactions can be attached and making the commercial client sending the script or access request pay for access to e.g. the GPS location, thus reversing the current notion of users spreading their data and its connections into multiple commercial cloud services, paying with their copyright, usage patterns and privacy only to be forced to aggregate the total personal information again via services like FriendFeed, Plaxo etc.
In the longer run, the total personal information, turned into a consistent personal graph of information with data models like Neo4j, a Graph Database would of course be held physically at more convenient locations than a non-backed up mobile phone and linked to the other personal sources of new information like desktop computers and embedded devices like cameras and GPS.

Socially controlled access to the personal information

5. Social home automation and media sharing, the Internet Of Things

With media and home automation devices having Internet access (TV, set top boxes, media servers, stereo), installing XMPP aware software on them or providing a DLNA/XMPP gateway on my home computer under my personal account will expose these as different accessible resources on the Internet, without the intervention of firewalls, wireless routers etc. So, now my friends LinkedProcess enabled phone marie@neubuaer.se/phone1 can discover my living room TV, peter@neubaeur.se/tv1 since I am in Maries Roster, and the TV is AVAILABLE in XMPP. It discovers the capabilities of the TV much like DLNA and UPnP and then either sends over the latest photo stream to my TV directly, or negotiates means of doing so, e.g. via FTP to a third server. Voila, the home automation is freed onto the Internet and visible and controlled by me and the friends I trust. The same goes for lamps, fridges, home security systems etc with the appropriate bridging in place on a convenient machine, e.g. your home media server. For an actual implementation of light-control via XMPP, see this bridging lab

Home autmomation devices, directly or via an UPnP/XMPP bridge exposed to LinkedProcess

Conclusion

These are just some use cases that one can think of when it comes to merging the key ingredients:

• status aware, realtime, lightweight messaging infrastructure with contact list support
• widely used infrastructure with existing contact networks (GTalk, iChat etc)
• Open Platforms like LinkedProcess providing the protocol for communication and exchange of machine-readable data packets

With the advent of Google Wave, I think the time has come to realize the great potential in Social Computing - introducing social aspects into core programming, security and distribution paradigms.

/peter neubauer

Co-founder, NeoTechnology

Consultant, Jayway

I was playing around with creating a Twitter clone, in my opinion the "Hello World" of graph databases, and I started out with the following simple service.

Spring configurating:

 
<beans ... >
	<context:annotation-config />
 
	<bean id="neo" class="org.neo4j.api.core.EmbeddedNeo"
		destroy-method="shutdown">
		<constructor-arg>
			<value>var/neo</value>
		</constructor-arg>
	</bean>
 
	<bean id="indexService" class="org.neo4j.util.index.NeoIndexService"
		destroy-method="shutdown">
		<constructor-arg ref="neo" />
	</bean>
 
	<bean id="twitterService"
		class="com.jayway.twitter.neo.NeoBasedTweetService">
<property name="neo" ref="neo" />
<property name="indexService" ref="indexService" />
	</bean>
 
	<!-- Transactions -->
	<tx:annotation-driven />
        ...
</beans>
 

The service:

 
public class NeoBasedTweetService implements TweetService {
 
	protected NeoService neo;
	protected IndexService indexService;
 
	//Setters for neo and indexService
 
	@Transactional
	public Tweet createTweet(User user, String tweet) {
		UserNode userNode = new UserNode(indexService.getSingleNode(
				TwitterIndexConstants.INDEX_USER, user.getUserName()));
		TweetNode tweetNode = new TweetNode(neo.createNode());
		tweetNode.setKvitt(tweet);
		userNode.createRelationshipTo(tweetNode, KvittRelationships.KVITTRAT);
		return tweetNode;
	}
 
	//Services to create User and more
 
}
 

This doesn't feel very DDD. I would really want to do the User to be responsible for creating the Tweet. Like this:

 
public class NeoBasedTweetService implements TweetService {
 
	protected NeoService neo;
	protected IndexService indexService;
 
	//Setters for neo and indexService
 
	@Transactional
	public Tweet createTweet(User user, String tweet) {
		UserNode userNode = new UserNode(indexService.getSingleNode(
				TwitterIndexConstants.INDEX_USER, user.getUserName()));
		return userNode.createTweet(tweet);
	}
 
	//Services to create User and more
 
}
 

The TweetNode and UserNode would look like this:

 
public class TweetNode extends NodeDelegate implements Tweet {
 
	TweetNode(Node underlyingNode) {
		super(underlyingNode);
	}
 
	//Methods for doing Tweet-stuff
 
}
 

 
public class UserNode extends NodeDelegate implements User{
 
	UserNode(Node underlyingNode) {
		super(underlyingNode);
	}
 
	//Methods for doing User-stuff
 
	public Tweet createTweet(String kvitt) {
		//Here comes a problem! I need a new Node
		//from the NeoService to create the TweetNode
		TweetNode tweetNode = new TweetNode(null);
		tweetNode.setTweet(tweet);
		createRelationshipTo(tweetNode, TwitterRelationships.TWEETED);
		return tweetNode;
	}
}
 

NodeDelegate, extended by both User- and TweetNode implements the Neo4j interface Node and delegates all the method calls to the underlying node ( for more on this, see my previous post) :

 
public class NodeDelegate implements Node {
 
	private Node delegate;
 
	protected NodeDelegate(Node underlyingNode) {
		delegate = underlyingNode;
	}
 
	//Methods for delegating all Node methods to the Node delegate
}
 

As you see I have a problem here. In the first version my NeoService is injected into NeoBasedTweetService and used when I create the new TweetNode. In the re-made, more DDD-style example I create the new TweetNode in the UserNode, where I don't have access to my NeoService. How do we solve this?

One alternative that I instinctively dislike is to inject the NeoService into the UserNode. The reason for me disliking this is that my UserNode IS a Node, and it feels very tangled if a Node holds a reference to its container. But I came to think of another solution thanks to a talk I did a while back together with Henrik Reinhold on what was new in Spring 2.5.

In Spring 2.5 load-time weaving (LTW) capabillites was introduced, and with LTW you can manipulate objects in load-time, as opposed to runtime weaving that is proxy based. This, combine it with @Configurable and @Autowired, proves to be very helpful when the Spring context is not in charge of creating the objects in question, as in the example above...

To enable LTW we add one tag in the Spring configuration:

 
<context:load-time-weaver />
 

After that we have to inform the Spring container of how we want thing done. To do we edit the TweetNode and NodeDelegate like this:

 
//Annotate the class with @Configurable so that Spring can find it.
@Configurable
public class TweetNode extends NodeDelegate implements Tweet {
 
	TweetNode(Node underlyingNode) {
		super(underlyingNode);
	}
 
	//Add a default constructor both in the TweetNode
	//and the NodeDelegate, it is new:able without a Node.
	public TweetNode() {
		super();
	}
 
	//Add a setter for the underlaying Node and annotate it with @Autowired.
	@Autowired
	public void setNode(Node nodeDelegate) {
		super.setNode(nodeDelegate);
	}
 
	//Tweet-methods as before
 
}
 

 
public class NodeDelegate implements Node {
 
	private Node delegate;
 
	protected NodeDelegate(Node underlyingNode) {
		delegate = underlyingNode;
	}
 
	protected NodeDelegate() {
	}
 
	protected void setNode(Node nodeDelegate) {
		if (delegate == null) {
			delegate = nodeDelegate;
		}
	}
 
	//Methods for delegating all Node methods to the Node delegate
 
}
 

Now Spring only has to be able to create Nodes. To do that we create a simple NodeFactory, like this:

 
public class NodeFactory implements FactoryBean{
 
	private NeoService neo = null;
 
	public void setNeo(NeoService neo) {
		this.neo = neo;
	}
 
	@Transactional
	public Object getObject() throws Exception {
		return neo.createNode();
	}
 
	public Class getObjectType() {
		return Node.class;
	}
 
	public boolean isSingleton() {
		return false;
	}
 
}
 

... and add it to the configuration, like this:

 
	<bean id="node" class="com.jayway.neo.utils.NodeFactory">
<property name="neo" ref="neo" />
	</bean>
 

The result of this is that Spring now has a NodeFactory that returns a new node for every request, I have annotated the TweetNode so that Spring knows of it (@Configurable) and what to do with it (@Autowired on setNode) and I've enabled load-time weaving in order to get the a new Node woven into a TweetNode every time a new instance of it is created in the code.

In UserNode I really don't have to make any changes. "setNode(Node)" will get a new Node after the instantiation of a TweetNode, even if the constructor that takes a Node is called. For clarity I still changed the UserNode as follows:

 
public class UserNode extends NodeDelegate implements User{
 
	UserNode(Node underlyingNode) {
		super(underlyingNode);
	}
 
	//Methods for doing User-stuff
 
	public Tweet createTweet(String kvitt) {
		//This TweetNode will get a Node set by Spring
		TweetNode tweetNode = new TweetNode();
		tweetNode.setTweet(tweet);
		createRelationshipTo(tweetNode, TwitterRelationships.TWEETED);
		return tweetNode;
	}
}
 

The problem I had when trying to make the UserNode responsible for creating its TweetNode's was that I needed to inject something that could create Nodes into the UserNodes. By using load-time weaving and instructing Spring of how to wire a TweetNode I don't have to pass my NeoService around to all the UserNodes created in my code. Instead I can rely on Spring to inject what is needed, when it's needed.

When I first started looking at Neo4j I was blown away by how precise the graph database structure matched my mental model of information. You have blobs of informations (nodes with properties) that relates to other blobs of information. Perfect!

When I started playing around with the API I couldn't do what I wanted, namely extend NodeImpl and make the objects I want to persist Nodes. Now, Neo4j has a very nice API which is hard to abuse (if you don't place your classes in the same namespace as they use), and understandably they don't want you to extend NodeImpl. The design guide gives this example of use:

 
public class CustomerImpl implements Customer{
 
        private final Node underlyingNode;
 
        private static final String KEY_FIRST_NAME = "firstName";
 
        CustomerImpl( Node underlyingNode ){
                this.underlyingNode = underlyingNode;
        }
 
        public void setFirstName( String firstName ){
                underlyingNode.setProperty( KEY_FIRST_NAME, firstName );
        }
 
        public String getFirstName(){
                return ( String ) underlyingNode.getProperty( KEY_FIRST_NAME );
        }
 
        // ...
}
 

That's great, but it doesn't match what I see in my head... In order to make things match I made a subtle change.

 
public class NodeDelegate implements Node {
 
        private Node delegate;
 
        protected NodeDelegate(Node node) {
                delegate = node;
        }
 
        // Delegate all methods of the Node interface to the
        // Node received in the constructor...
}
 
public class CustomerNode extends NodeDelegate implements Customer {
 
        private static final String KEY_FIRST_NAME = "firstName";
 
        CustomerNode(Node node) {
                super(node);
        }
 
        public void setFirstName( String firstName ){
                setProperty( KEY_FIRST_NAME, firstName );
        }
 
        public String getFirstName(){
                return ( String ) getProperty( KEY_FIRST_NAME );
        }
 
        // ...
 
}
 

By doing this I can work with my domain objects as Nodes. I don't know if it is "correct" but with this small change I can say that "A Customer IS a Node" and this is what I see in my head. Good? Bad? You be the judge, but it fits me