Mar 05

Scala 2.10 Runtime Reflection from a Class Name

In the examples I have seen on Scala 2.10 reflection, the starting point is usually a known class. In the, example below, “List[Int]” is known.

scala> import scala.reflect.runtime.universe._
import scala.reflect.runtime.universe._

scala> typeOf[List[Int]]
res0: reflect.runtime.universe.Type = scala.List[Int]

What if at runtime, you just have a class name? How would you reflect that class? It turns out that I needed to:

// 1. Get a mirror
scala> val mirror = ru.runtimeMirror(getClass.getClassLoader)
mirror: reflect.runtime.universe.Mirror = ...


// 2. Convert class name into a class
scala> val clz = Class.forName("scala.collection.immutable.List")
clz: Class[_] = class scala.collection.immutable.List


// 3. Convert class into class symbol
scala> val listClassSymbol = mirror.classSymbol(clz)
listClassSymbol: reflect.runtime.universe.ClassSymbol = class List


// 4. You can now find out info about the class 
scala> listClassSymbol.isTrait
res3: Boolean = false

scala> listClassSymbol.isAbstractClass
res5: Boolean = true


// 5. To find out info about it members, inheritance, etc, 
//    you need to convert the class symbol into a Type
scala> val listType = listClassSymbol.toType
listType: reflect.runtime.universe.Type = List[A]

scala> listType.members
res6: reflect.runtime.universe.MemberScope = Scopes(method removeDuplicates, method foreach, method toStream, method stringPrefix, method reverse, method span, method dropWhile, method takeWhile, method splitAt, method takeRight, method slice, method drop, method take, method toList, method +:, method ++, method mapConserve, method reverse_:::, method :::, method ::, method companion, constructor List, method super$sameElements, method lastIndexWhere, method indexWhere, method segmentLength, method isDefinedAt, method lengthCompare, method sameElements, method dropRight, method last, method reduceRight, method reduceLeft, method foldRight, method foldLeft, method find, method contains, method exists, method forall, method apply, method length, method $init$, method productPrefix, method...


Once you have a Type, you can use a ClassMirror to instance it.

Here’s the official document on ClassSymbol, Type and Mirror.

Jan 29

Reflecting Annotations in Scala 2.10

Playing around with the idea of using annotations for Socko’s REST handler.

The official documentation can be found here.

However, there is a lack of examples. To fill the gap, there are some examples from my investigations using the Scala REPL.

A Simple Annotation

//
// You need to import the reflect package
//
scala> import scala.reflect.runtime.{universe => ru}
import scala.reflect.runtime.{universe=>ru}

//
// To save the annotation metadata, you need to extend from StaticAnnotation
//
scala> case class SimpleAnnotation() extends scala.annotation.StaticAnnotation
defined class SimpleAnnotation

//
// Define a class and specify the annotation
//
scala> @SimpleAnnotation case class SimpleClass()
defined class SimpleClass

//
// Use reflection to determine if the class has the annotation
//   1. Get the ClassSymbol for the class we want to check
//   2. Get the list of annotations from the ClassSymbol
//   3. Get the expected annotation type to match
//   4. Check the type of annotation equals the expected type
//
scala> val simpleClassSymbol = ru.typeOf[SimpleClass].typeSymbol.asClass
simpleClassSymbol: reflect.runtime.universe.ClassSymbol = class SimpleClass

scala> val simpleAnnotations = simpleClassSymbol.annotations
simpleAnnotations: List[reflect.runtime.universe.Annotation] = List(SimpleAnnotation)

scala> val simpleAnnotationType = ru.typeOf[SimpleAnnotation]
simpleAnnotationType: reflect.runtime.universe.Type = SimpleAnnotation

scala> simpleAnnotations.find(a => a.tpe == simpleAnnotationType)
res1: Option[reflect.runtime.universe.Annotation] = Some(SimpleAnnotation)

A Complex Annotation with Parameters

//
// You need to import the reflect package
//
scala> import scala.reflect.runtime.{universe => ru}
import scala.reflect.runtime.{universe=>ru}

//
// Declare a complex annotation with parameters
//
scala> case class ComplexAnnotation(p1: String, p2: Int) extends scala.annotation.StaticAnnotation
defined class ComplexAnnotation

//
// Define a class and specify the annotation with parameters
//
scala> @ComplexAnnotation(p1 = "test1", p2 = 2) case class ComplexClass()
defined class ComplexClass

//
// Use reflection to extract the parameter details
//   1. Get the ClassSymbol for the class we want to check
//   2. Get the list of annotations from the ClassSymbol
//   3. Get the expected annotation type to match
//   4. Find the annotation
//   5. Retrieve the args. These are returned as a list of Tree.
//
scala> val complexClassSymbol = ru.typeOf[ComplexClass].typeSymbol.asClass
complexClassSymbol: reflect.runtime.universe.ClassSymbol = class ComplexClass

scala> val complexAnnotations = complexClassSymbol.annotations
complexAnnotations: List[reflect.runtime.universe.Annotation] = List(ComplexAnnotation("test1", 2))

scala> val complexAnnotationType = ru.typeOf[ComplexAnnotation]
complexAnnotationType: reflect.runtime.universe.Type = ComplexAnnotation

scala> val complexAnnotation = complexAnnotations.find(a => a.tpe == complexAnnotationType)
complexAnnotation: Option[reflect.runtime.universe.Annotation] = Some(ComplexAnnotation("test1", 2))

scala> val complexAnnotationArgs = complexAnnotation.get.scalaArgs
complexAnnotationArgs: List[reflect.runtime.universe.Tree] = List("test1", 2)

scala> complexAnnotationArgs.foreach(a => println(ru.showRaw(a)))
Literal(Constant("test1"))
Literal(Constant(2))

//
// Bind the arguments back to the annotation class so that it is easy to use.
// Use reflection to instance the annotation class with the arguments.
//   1. Convert list of Tree to list of argument values
//   2. Get runtimeMirror
//   3. Get ClassSymbol for the annotation class
//   4. Get classMirror for the class
//   5. Get MethodSymbol for the constructor method
//   6. Get constructorMethodMirror for the method
//   7. Use the constructor to instance the annotation class.  Pass in the arguments.
//      Note that we use ": _*" so that each value in the argument list is passed 
//      as parameter (as opposed to passing the list as just 1 argument).
//
scala> val complexAnnotationArgValues = complexAnnotationArgs.map(a => a.productElement(0).asInstanceOf[ru.Constant].value)
complexAnnotationArgValues: List[Any] = List(test1, 2)

scala> val runtimeMirror = ru.runtimeMirror(getClass.getClassLoader)
runtimeMirror: reflect.runtime.universe.Mirror = JavaMirror with scala.tools.nsc.interpreter.IMain$TranslatingClassLoader@b85c3e6 of type class scala.tools.nsc.interpreter.IMain$TranslatingClassLoader with classpath [(memory)] and parent being scala.tools.nsc.util.ScalaClassLoader$URLClassLoader@7b0036f2 of type class scala.tools.nsc.util.ScalaClassLoader$URLClassLoader with classpath [file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar,file:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar,file:/home/vibul/bin/scala-2.10.0/lib/akka-actors.jar,file:/home/...

scala> val classSymbol = complexAnnotationType.typeSymbol.asClass
classSymbol: reflect.runtime.universe.ClassSymbol = class ComplexAnnotation

scala> val classMirror = runtimeMirror.reflectClass(classSymbol)
classMirror: reflect.runtime.universe.ClassMirror = class mirror for ComplexAnnotation (bound to null)

scala> val constructorMethodSymbol = complexAnnotationType.declaration(ru.nme.CONSTRUCTOR).asMethod
constructorMethodSymbol: reflect.runtime.universe.MethodSymbol = constructor ComplexAnnotation

scala> val constructorMethodMirror = classMirror.reflectConstructor(constructorMethodSymbol)
constructorMethodMirror: reflect.runtime.universe.MethodMirror = constructor mirror for ComplexAnnotation.<init>(p1: String, p2: scala.Int): ComplexAnnotation (bound to null)

scala> val myInstance = constructorMethodMirror(complexAnnotationArgValues: _*).asInstanceOf[ComplexAnnotation]
myInstance: ComplexAnnotation = ComplexAnnotation(test1,2)

scala> assert(myInstance.p1 == "test1")

scala> assert(myInstance.p2 == 2)

Jun 12

Switching Java Versions on Ubuntu

$ sudo update-java-alternatives -l
java-1.6.0-openjdk-amd64 1061 /usr/lib/jvm/java-1.6.0-openjdk-amd64
java-1.7.0-openjdk-amd64 1051 /usr/lib/jvm/java-1.7.0-openjdk-amd64
java-6-sun 63 /usr/lib/jvm/java-6-sun

$ sudo update-java-alternatives -s java-1.7.0-openjdk-amd64

This changes ALL java files to the specified JDK.

The following only changes the “java” program and NOT “javac”, etc.

$ sudo update-alternatives --config java






May 06

Installing MySQL 5.5 on Ubuntu 11

The package manager for Ubuntu 11 installs MySQL 5.1.

Read instructions from mysql and here to install 5.5.

Step #1. Download

1. Browse to: http://dev.mysql.com/downloads/mysql/5.5.html
2. Choose: Linux Generic
3. Download: Linux - Generic 2.6 (x86, 64-bit), Compressed TAR Archive

Step #2. Create mysql user and group

$ sudo groupadd mysql
$ sudo useradd -r -g mysql mysql

Step #3. Install files to /usr/local/mysql

$ cd /usr/local
$ sudo tar zxvf /path/to/mysql-VERSION-OS.tar.gz
$ sudo ln -s full-path-to-mysql-VERSION-OS mysql
$ cd mysql
$ sudo chown -R mysql .
$ sudo chgrp -R mysql .

Step #4. Install Async IO library

$ sudo apt-get install libaio1

Step #5. Run mysql install script

$ sudo scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql

Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/local/mysql/bin/mysqladmin -u root password 'new-password'
/usr/local/mysql/bin/mysqladmin -u root -h ubuntu password 'new-password'

Alternatively you can run:
/usr/local/mysql/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr/local/mysql ; /usr/local/mysql/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd /usr/local/mysql/mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/local/mysql/scripts/mysqlbug script!

Step #6. Add to path

$ sudo vi /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/mysql/bin"

Step #7. Create the socket directory

Setting the correct permission on the socket directory is very important, otherwise MySQL would not run.

$ sudo mkdir /var/run/mysqld/
$ mkdir: cannot create directory `/var/run/mysqld/': File exists
$ sudo chown -R mysql:mysql /var/run/mysqld/

Step #8. Config Settings

Create configuration

$ cp support-files/my-medium.cnf /etc/my.cnf

Add settings in /etc/my.cnf

[client]
#password = your_password
port = 3306
socket =  /var/run/mysqld/mysqld.sock
#/tmp/mysql.sock

# The MySQL server
[mysqld]
user = mysql
port = 3306
socket = /var/run/mysqld/mysqld.sock
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
tmpdir = /tmp
log_error = /var/log/mysql/error.log

Step #9. Remove the MySQL files from the older version

$ sudo rm -R /var/lib/mysql
$ sudo rm -R /etc/mysql
$ sudo rm -R /usr/lib/mysql

Step #10. Start Server

From the terminal …

$ sudo /usr/local/mysql/bin/mysqld_safe --user=mysql &

At startup as a service …

$ cd /usr/local/mysql/support-files/ 
$ sudo cp mysql.server /etc/init.d/mysql 
$ sudo chmod +x /etc/init.d/mysql
$ sudo update-rc.d mysql defaults
# Starting
$ sudo service mysql start
# Stopping 
$ sudo service mysql stop

Log file will be kept at /var/log/mysql/error.log

Step #11. Secure Server

$ sudo /usr/local/mysql/bin/mysql_secure_installation
May 02

Just released Socko v0.1

Got together with my good friend Dave to do some “brogramming”. We are trying to learn scala/akka and use it for a new open source project called Mashupbots (more on Mashupbots later).

The first component of Mashupbots is now out!

It is an embedded Scala web server powered by Netty networking and Akka processing.

We’ve called it Socko. Check it out.

Mar 28

Don’t get fooled by Akka Logging Thread Name – Logging is Asynchronous

In my scala/akka web server, I’m doing all the processing in akka Actors.

I’m doing a lot of blocking IO so, as per documentation, I’ve setup my Actor to run as a router with a pinned dispatcher so that each worker gets its own thread.

   val router = actorSystem.actorOf(Props[StaticFileProcessor]
      .withRouter(FromConfig()).withDispatcher("mydispatcher"), "myrouter")

In my application.conf file, I have the following setting:

mydispatcher {
  executor = "thread-pool-executor"
  type = PinnedDispatcher
  thread-pool-executor {
    # minimum number of threads to cap factor-based core number to
    core-pool-size-min = 25
    # No of core threads ... ceil(available processors * factor)
    core-pool-size-factor = 2.0
    # maximum number of threads to cap factor-based number to
    core-pool-size-max = 30
  }  
}

akka {
  event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
  loglevel=DEBUG
  
  actor {
    deployment {
      /myrouter {
        router = round-robin
        nr-of-instances = 5
      }
    }
  }  
}

Log statements from my Actor is getting printed as follows:

22:15:44.471 [TestActorSystem-akka.actor.default-dispatcher-3] DEBUG o.m.w.processors.StaticFileProcessor akka://TestActorSystem/user/file-processor-test-router/$a - Compressed file name: /tmp/Temp_7097448265494079716/f6310e6f4d77113539594e81c62601cd.deflate

Note that the thread is: TestActorSystem-akka.actor.default-dispatcher-3.

This lead me to think that my pinned dispatcher configuration is not working. Why would the thread be named default-dispatcher-3? Does this mean that my Actors are using the default dispatcher rather than mydisptacher?

After discussions with the Akka team, I found that the misunderstanding comes from akka logging.

Akka logging is async and the thread printed in the log statement is the thread that is displaying the log message – NOT the thread that wrote or initiated the log message.

The fix is to add “%X{sourceThread}” to the logback pattern. For example: %d{HH:mm:ss.SSS} [%thread %X{sourceThread}] %-5level %logger{36} %X{akkaSource} – %msg%n.

Now my log message is:

22:15:44.471 [TestActorSystem-akka.actor.default-dispatcher-3 TestActorSystem-1] DEBUG o.m.w.processors.StaticFileProcessor akka://TestActorSystem/user/file-processor-test-router/$a - Compressed file name: /tmp/Temp_7097448265494079716/f6310e6f4d77113539594e81c62601cd.deflate

The thread in which my Actor is running and writing the log message is: TestActorSystem-1.

The thread displaying the log message is: TestActorSystem-akka.actor.default-dispatcher-3.

Mar 28

Apache Benchmark (ab) hanging and timeout with -k Keep-Alive switch

Here’s one for newbies …

I’ve been trying to learn scala and akka. I decided to write a netty based web server in scala and using akka.

To test my new server, I used ab.

ab works fine with no “keep alive”.

However, with “keep alive”, ab kept hanging after the first request. It eventually timed out.

Thanks to Sean’s post, I found that the problem is that my server needed to return the “Connection: keep-alive” header in the response!

According to Sean,

This behaviour isn’t specified anywhere, but it is suggested in an Internet Draft for HTTP/1.1 at w3.org:
The Connection header field with a keep-alive keyword must be sent on all requests and responses that wish to continue the persistence.

I’ll be sending a pull request to Netty shortly to add this to the http examples.

Nov 12

My Git Hub Shortcuts

A good Cheet Sheet.

Fork

http://help.github.com/fork-a-repo/

# Clone to Spoon-Knife directory
git clone git@github.com:username/Spoon-Knife.git

# Clone to SpecificFolderName directory
git clone git@github.com:username/Spoon-Knife.git SpecificFolderName

cd Spoon-Knife
git remote add upstream git://github.com/octocat/Spoon-Knife.git
git fetch upstream

 

Committing Changes

# To commit changes to local repository
git commit -a

# Send changes to remote repository:
git push origin master

 

Update a Forked Repository

#Fetches any new changes from the original repo
git fetch upstream

# Merge changes retrieved from the fetch
git merge upstream/master

 

Creating a Branch

# Creates a new branch called "mybranch"
git branch mybranch

# Switch to mybranch
git checkout mybranch

# Push new local branch
git push origin mybranch

# Merge changes from mybranch into master and delete mybranch
git checkout master
git merge mybranch
git branch -d mybranch

# Delete remote branch after local delete
git push origin :mybranch

Getting a Remote Branch

http://stackoverflow.com/questions/1897475/github-how-to-checkout-production-branch

# To list branches:
git branch -a

#You could checkout directly the remote branch after fetching it
git fetch origin
git branch -f remote_branch_name origin/remote_branch_name
git checkout remote_branch name

# or shorter:
git checkout -b production origin/production

# Merge changes from remote production into local production
git co -b production
git pull origin production

Tagging

#Create local tag
git tag -m"Tag version 1.0" V1.0 

# Push tag to origin
git push --tags

Merge Patch Request Locally to Test it Out

# Every pull request has a .patch URL where you can grab a textual patch file to feed into the git-am command
$ git checkout master
$ curl http://github.com/github/jobs/pull/25.patch | git am
$ git push origin master

If more than 1 person is working on a branch …

To avoid “Merge branch ‘master’ of …” in your commit history, use the following to get the latest changes. See this link for more info.

$ git pull --rebase origin

Get latest from master into development branches

$ git checkout -b develop

#...make some changes...
#...notice master has been updated...

$ git checkout master
$ git pull

#...bring those changes back into develop...

$ git checkout develop
$ git rebase master

#...make some more changes...
#...commit them to develop...
#...merge them into master...

$ git checkout master
$ git pull
$ git merge develop