Thursday, 8 September 2011

Starting with Lift and CouchDB: Resolving Some CouchDB-Record Inconsistencies

In recent months I have been introduced to Lift web framework by my scala-enthusiastic colleagues @ OpenCredo. The scala expressiveness and conciseness along with view-first web approach looked very interesting from the start, especially since great part of my previous experience comes from verbose Java frameworks with traditional MVC architecture.
After initial interest and research I did my first project, using a relational database and the well-known Lift-Mapper framework. Much of the data we needed to store represented documents, so I started experimenting with CouchDB. While the REST interface to Documents and Views in Couch does not require a rich mapping layer (as opposed to the richness  of SQL ORM frameworks), I still found it cumbersome to work directly with HTTP all the time in order to store/query my model objects to/from CouchDB.
One of the nice features of Lift's Record framework is it's agnostic approach to the underlying persistence mechanism, which means that it is relatively easy to plug in any storage solution. Currently the Lift-Record framework has a Record implementation for relational databases (Squeryl), as well as  Mongo-Record and CouchDB-Record implementations which gained prominence with recent increased interest in NoSQL storage. While the CouchDB-Record implementation is not complete, it has enough features to make it useful when working with CouchDB and Lift.  Using CouchDB-Record I could easily map my domain model to the underlying database, and abstract the HTTP layer when I wanted to. The CouchDB-Record implementation did not offer everything that you might expect from a typical O(R)M framework - specifically there are some missing features such as associations mapping, lazy loading, caching. However these features weren't required for this specific project, and some would argue that those are not even a features that a non-relational storage should use (there is no R in ORM if using CouchDB).
First thing to do when using the CouchDB with lift is to add database configuration to the Lift's Boot class. To do this, you need only three lines of code, as Listing 1 illustrates
Listing 1:
val couch = new Database("127.0.0.1",5984, "lift-test")
couch.createIfNotCreated(new Http())
CouchDB.defaultDatabase = couch

To map a model object to CouchDB persistence, you'll need to extend CouchRecord class which is part of the couchdb-record framework. In addition, you define its companion object with the same name, which mixes CouchMetaRecord trait.
Listing 2 shows the sample User class mapped to the CouchDB database using Lift's Record framework
Listing 2:
import net.liftweb.record.field.{DateTimeField, PasswordField, StringField}
import net.liftweb.couchdb.CouchRecord
import com.opencredo.nicheplatform.couchdb.CouchMetaRecord
class User extends CouchRecord[User] {
  def meta = User

  object username extends StringField(this, 20)
  object fullName extends StringField(this, 100)
  object birthDate extends DateTimeField(this)
}
object User extends User with CouchMetaRecord[User] {
}

We demonstrated 2 different field types here: StringField and DateTimeField. For a full list of supported filed types in couchdb-record implementation, you can check the online documentation at http://exploring.liftweb.net/master/index-8.html.
Ok, let's persist a user now. To check that the user is actually peristed we're going to write a test, as illustrated in Listing 3:
Listing 3:
@RunWith(classOf[JUnitRunner])
class UserTest extends FunSuite with LazyLoggable {
  // reset database and boot lift
  if (!Boot.initialized_?) {
    try{
      Http(new Database("127.0.0.1", 5984, "lift-test").delete)
    }catch {
      case e : Exception => Console.print(e)
    }
    val boot: Boot = new Boot()
    boot.boot
  }

 test("Create a new User") {
   val userRecord = User.createRecord #1
   userRecord.username.set("mrbob") #2
   userRecord.fullName.set("Bob Bobic") #3
   val user: Box[User] = userRecord.save #4
   assert(user.isDefined, user.compoundFailMsg("Failure")) #5
 }
}


To create a persistable couchdb-backed document, we simple call the createRecord fuction on our user object (#1). After we set the properties we require (#2, #3), we simply save the created object (#4).
The save function persists the object (by issuing a POST request to the CouchDb server) and returns a Box. The Box will be full if the operation succeeded, in which case the Box will contain the peristed object - with assigned id and rev fields from the CouchDb server.
If the POST operation fails to create the new document the box will be empty, and will contain the Failure message (#5)

The code looks nice, so we run the test, excited about our first couch db persisted object - but the test FAILS miserably!!! What could be wrong.
The test fails at the assertion point (so at least we did the configuration right), and the message is as follows:
Failure(User not defined,Empty,Full(Failure(ok not present in reply or not true: JObject(List(JField(ok,JBool(true)), JField(id,JString(706a63a5ae0adc3325347c8a1d020160)), JField(rev,JString(1-8eaacb2ca383a965037251676b25dbd6)))),Empty,Empty)))

What does this mean? "ok not present in reply or not true", but the JSON object referenced in the rest of the message clearly has the OK message (JField(ok,JBool(true))?
I found this confusing.
What was even more confusing was that the object was actually saved to CouchDB successfully, as I could find it using the Futon web interface.
The fact that the object was actually persisted, and the error message suggested that there was a bug in response parsing code within Lift. I wasn't that familiar with the Lift code base, so after numerous tries,
I found the thread that suggested someone else was having a similar problem (http://groups.google.com/group/liftweb/browse_thread/thread/15d17b9348d4d88e#)
What was the problem? Unfortunately, CouchDB Record implementation didn't follow the work done on other parts of lift framework. The culprit is the following code in CouchDB Record implementation (net.liftweb.couchdb.Database.scala file):
Listing 4:
private[couchdb] object DatabaseHelpers {
  /** Handles the JSON result of an update action by parsing out id and rev, updating the given original object with the values and returning it */
  def handleUpdateResult(original: JObject)(json: JValue): Box[JObject] =
    for {
      obj <- Full(json).asA[JObject] ?~ ("update result is not a JObject: " + json)
      ok  <- Full(json \ "ok" ).asA[JField].map(_.value).asA[JBool].filter(_.value) ?~ ("ok not present in reply or not true: "+json)
      id  <- Full(json \ "id" ).asA[JField].map(_.value).asA[JString].map(_.s)    ?~ ("id not present or not a string: " + json)
      rev <- Full(json \ "rev").asA[JField].map(_.value).asA[JString].map(_.s)    ?~ ("rev not present or not a string: " + json)
    } yield updateIdAndRev(original, id, rev)
}

The argument json: JValue used to be an instance of JField, which the code above correctly maps to concrete JBool/JString instance in order to check the response of CouchDB update operation.
However, some changes to Json parsing since liftweb 2.2 improved this, so that \ operator on the JValue object returns the correct JBool/JString objects directly. This method is called whenever the document is created/updated in CouchDB.
The correct code should look like this:
Listing 5:
private[couchdb] object DatabaseHelpers {
  /** Handles the JSON result of an update action by parsing out id and rev, updating the given original object with the values and returning it */
  /** This is a fix as the original code still depends on old JSON library -- Aleksa**/
  def handleUpdateResult(original: JObject)(json: JValue): Box[JObject] =
    for {
      obj <- Full(json).asA[JObject] ?~ ("update result is not a JObject: " + json)
      ok  <- Full(json \ "ok" ).asA[JBool].filter(_.value) ?~ ("ok not present in reply or not true: "+json)
      id  <- Full(json \ "id" ).asA[JString].map(_.s)    ?~ ("id not present or not a string: " + json)
      rev <- Full(json \ "rev").asA[JString].map(_.s)    ?~ ("rev not present or not a string: " + json)
    } yield updateIdAndRev(original, id, rev)
}

Unfortunately, this change hasn't propagated to CouchDB Record code, and at the time of the writing, there is an open ticket for this bug (http://www.assembla.com/spaces/liftweb/tickets/961-saving-to-couchdb-doesn-t-fetch-the-document--id--property)
Since DatabaseHelpers is the private scala object, it cannot be extended - so you have 2 options:
1. Checkout liftweb, fix the source, build it and use the custom built artefacts in your project. Of course, you'll lose ability to follow future releases of liftweb until the bug is fixed - but hopefully that won't be for too long
2. Override the methods that depend on it, by injecting the correct implementation.
You can easily overwrite the post method on the Database class, which is used by CouchRecord, which all of your CouchDB-mapped objects extend:
Listing 6:
val couch = new Database("127.0.0.1",5984, "lift-test") {
      override def post(doc: JObject): Handler[Box[JObject]] = {
        (JSONRequest(this) <<# doc) ># handleUpdateResultCorrected(doc) _
      }
      def handleUpdateResultCorrecred(original: JObject)(json: JValue): Box[JObject] = {
        //bug-free implementation, as above
      }
}
couch.createIfNotCreated(new Http())
CouchDB.defaultDatabase = couch

This solves the problem of storing new documents to CouchDB - however updates are not handled by this overridden method.
Updates are defined in the Document trait buried within the Database.scala, and this trait is then extend and used for composition quite a few times - not easy to follow, let alone to figure out the clean way to overrirde its behaviour.
You could argue that it is not a significant problem for updates, as you know the document's id and revision values up front, you do not need to confirm them from the response when it succeeds. However, you still have incorrect behaviour, and that is something I cannot live with :)
Rather then spending too much time figuring out how to cleanly override the post method in Document trait, I went down the hard route, checked out the code and fixed the bug, and from there on I used my own custom build lift-couchdb_2.9.0-2.4M3-Aleksa.jar

Now that we have basic tests passing, let's see how we can load the persisted object from CouchDB using its id.
We will continue working on the same test:
Listing 7:
test("Create a new User") {
   val userRecord = User.createRecord
   userRecord.username.set("mrbob")
   userRecord.fullName.set("Bob Bobic")
   val user: Box[User] = userRecord.save

   assert(user.isDefined, user.compoundFailMsg("User not defined"))

   expect("mrbob"){
      user.open_!.username.get
   }
   assert(!user.open_!.id.get.get.isEmpty) #1
   assert(!user.open_!.rev.get.get.isEmpty) #2

   val loadedUser: Box[User] = User.fetch(user.open_!.id.get.get) #3
   assert(loadedUser.isDefined, loadedUser.compoundFailMsg("User not defined"))
   expect("mrbob"){
      loadedUser.open_!.username.get
   }
   expect("Bob Bobic"){
      loadedUser.open_!.fullName.get
   }
   expect(user.open_!.rev.get){
      loadedUser.open_!.rev.get
   }
}


First, we check that the ide and revision fields are now populated as expected (#1, #2).
And the we load the user using the existing id using the fetch method (#3) - it's that simple!
The assertions that follow prove that we loaded all user properties, and that we loaded the document with the latest (expected) revision.
NOTE: Because the CouchRecord's id field is optional, the get function returns the Option object, so we need another get to get to the actual id.

You can use the save function to update the existing record as well. The document to be updated must have id and rev fields set.
Listing 8:
val u2: User = loadedUser.open_!
u2.username.set("bobby")
val updatedUser: Box[User] = u2.save
expect("bobby") {
  loadedUser.open_!.username.get
}
expect(loadedUser.open_!.id.get) {
  updatedUser.open_!.id.get
}


And that's it. Even with few challenges from the Lift's CouchDb implementation, the persistence and fetching of Scala objects to/from CouchDB worked nicelly, using Lift's Record framework.

I will be talking about my experiences on Lift/Scala-CouchDB integration at the NoSQL Exchange at SkillsMatter in London on November 2nd 2011.


15 comments:

  1. Hi Aleksa
    My name is James Sugrue, editor at JavaLobby.
    I just read through your blog and think you'd be a great addition to our MVB program.
    You can find out more about the MVB program at dzone.com/aboutmvb. If you're interested, please send me an email to james at dzone dot com.
    Thanks
    James

    ReplyDelete
  2. With havin so much content and articles do you ever run into any issues of plagorism or copyright violation? My blog has a lot of completely unique content
    I've either created myself or outsourced but it appears a lot of it is popping
    it up all over the internet without my authorization. Do you know any ways to
    help reduce content from being ripped off? I'd genuinely appreciate it.


    Here is my web blog ... air conditioner system Dallas

    ReplyDelete
  3. The great issue with these stores is that they permit you to order your choice without the requirement to be physically gift there.

    Men like lingerie because women look gorgeous and erotic
    while wearing them and women like lingerie because it makes them feel sexy and desirable.

    If you are tired of going to your local store and finding that
    they have just sold out the lingerie of your size then you can consider online shopping You will not have to
    face this issue while shopping online.

    Also visit my site ... looking for online Sex shop

    ReplyDelete
  4. pada sebuah kehidupan berumah tangga pasangan selalu membutuhkan alat bantu sex
    untuk wanita yang maniak berhubngan intim tapi di tinggal pasangannya pakai aja alat bantu sex wanita minimal alat bantu sex pria
    untuk menghindari seorang suami berselingkuh dalam perjalanan luar kota alangkah baiknya bawa bekal boneka sex | boneka full body atau alat bantu sex
    dan alat bantu seks
    dan bagi pria atau suami yang penisnya sudah loyo pakai aja penis ikat pinggang penyangga jual alat bantu sex
    dan bagi wanita yang ingin berfantasi sex engan pasangannya alat bantu sexualitas wanita
    jika anda mencari alat bantu wanita jangan ragu klik aja link ini alat bantu seks
    kebanyakan orang dalam berhubungan intim mengunakan alat bantu seksual wanita atau alat seksual wanita
    dan rata rata wanita yang doyan ngesex biasanya menggunakan alat sex wanita atau menggelitik anunya dengan sex toys
    untuk pria yang ingin mememuas kan pasangannya tapi malu karena penisnya kecil alat pembesar penis

    ReplyDelete
  5. bagi pria yang males pakai alat oalah raga penis bisa menggunakan obat pembesar penis atau pembesar penis
    yang bermerk neo size xl | vimax
    dan kalau pria itu alat vitalnya sudah gede tapi spermanya kuran dan tidak subur bisa menggunakan obat penyubur sperma
    jangan lupa saat berhubungan intim menggunakan kondom silikon polos | kondom bergerigi untuk menghindari penyakit gunakan juga kondom sambung silikon
    bagi pria yang ingin memuaskan pasangannya gunakan obat tahan lama dan obat perkasa cocok di gunakan bagi pria yang ingin berhubungan intim
    untuk pria yang sudah loyo disarankan menggunakan obat kuat atau obat sex biar stamina badan selau terjaga obat vitalitas pria
    seorang wanita yang dilihat pertama dari pemutih wajah dan body badan obat pelangsing dibagian penumbuh rambut itu juga mendukung pemutih gigi apa lagi tinggi semapai obat peninggi badan
    bagi seorang wanita yang ingin memaksimalkan penampilan pemerah bibir atau ingin menghilangkan penghilang bekas luka atau perontok bulu
    penting juga untuk pria maupun wanita yang ingin obat penggemuk dan membersihkan pemutih selangkangan atau menghilangkan pengghilang selulit

    ReplyDelete