Veeb's Brain Dump

The package manager for Ubuntu 11 installs MySQL 5.1.

Read instructions from mysql and here to install 5.5.

Step #1. Download

1. Browse to: http://dev.mysql.com/downloads/mysql/5.5.html
2. Choose: Linux Generic
3. Download: Linux - Generic 2.6 (x86, 64-bit), Compressed TAR Archive

Step #2. Create mysql user and group

$ sudo groupadd mysql
$ sudo useradd -r -g mysql mysql

Step #3. Install files to /usr/local/mysql

$ cd /usr/local
$ sudo tar zxvf /path/to/mysql-VERSION-OS.tar.gz
$ sudo ln -s full-path-to-mysql-VERSION-OS mysql
$ cd mysql
$ sudo chown -R mysql .
$ sudo chgrp -R mysql .

Step #4. Install Async IO library

$ sudo apt-get install libaio1

Step #5. Run mysql install script

$ sudo scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql

Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

/usr/local/mysql/bin/mysqladmin -u root password 'new-password'
/usr/local/mysql/bin/mysqladmin -u root -h ubuntu password 'new-password'

Alternatively you can run:
/usr/local/mysql/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default.  This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:
cd /usr/local/mysql ; /usr/local/mysql/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl
cd /usr/local/mysql/mysql-test ; perl mysql-test-run.pl

Please report any problems with the /usr/local/mysql/scripts/mysqlbug script!

Step #6. Add to path

$ sudo vi /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/mysql/bin"

Step #7. Create the socket directory

Setting the correct permission on the socket directory is very important, otherwise MySQL would not run.

$ sudo mkdir /var/run/mysqld/
$ mkdir: cannot create directory `/var/run/mysqld/': File exists
$ sudo chown -R mysql:mysql /var/run/mysqld/

Step #8. Config Settings

Create configuration

$ cp support-files/my-medium.cnf /etc/my.cnf

Add settings in /etc/my.cnf

[client]
#password = your_password
port = 3306
socket =  /var/run/mysqld/mysqld.sock
#/tmp/mysql.sock

# The MySQL server
[mysqld]
user = mysql
port = 3306
socket = /var/run/mysqld/mysqld.sock
basedir = /usr/local/mysql
datadir = /usr/local/mysql/data
tmpdir = /tmp
log_error = /var/log/mysql/error.log

Step #9. Remove the MySQL files from the older version

$ sudo rm -R /var/lib/mysql
$ sudo rm -R /etc/mysql
$ sudo rm -R /usr/lib/mysql

Step #10. Start Server

From the terminal …

$ sudo /usr/local/mysql/bin/mysqld_safe --user=mysql &

At startup as a service …

$ cd /usr/local/mysql/support-files/
$ sudo cp mysql.server /etc/init.d/mysql
$ sudo chmod +x /etc/init.d/mysql
$ sudo update-rc.d mysql defaults
# Starting
$ sudo service mysql start
# Stopping
$ sudo service mysql stop

Log file will be kept at /var/log/mysql/error.log

Step #11. Secure Server

$ sudo /usr/local/mysql/bin/mysql_secure_installation

Got together with my good friend Dave to do some “brogramming”. We are trying to learn scala/akka and use it for a new open source project called Mashupbots (more on Mashupbots later).

The first component of Mashupbots is now out!

It is an embedded Scala web server powered by Netty networking and Akka processing.

We’ve called it Socko. Check it out.

In my scala/akka web server, I’m doing all the processing in akka Actors.

I’m doing a lot of blocking IO so, as per documentation, I’ve setup my Actor to run as a router with a pinned dispatcher so that each worker gets its own thread.

   val router = actorSystem.actorOf(Props[StaticFileProcessor]
      .withRouter(FromConfig()).withDispatcher("mydispatcher"), "myrouter")

In my application.conf file, I have the following setting:

mydispatcher {
  executor = "thread-pool-executor"
  type = PinnedDispatcher
  thread-pool-executor {
    # minimum number of threads to cap factor-based core number to
    core-pool-size-min = 25
    # No of core threads ... ceil(available processors * factor)
    core-pool-size-factor = 2.0
    # maximum number of threads to cap factor-based number to
    core-pool-size-max = 30
  }
}

akka {
  event-handlers = ["akka.event.slf4j.Slf4jEventHandler"]
  loglevel=DEBUG

  actor {
    deployment {
      /myrouter {
        router = round-robin
        nr-of-instances = 5
      }
    }
  }
}

Log statements from my Actor is getting printed as follows:

22:15:44.471 [TestActorSystem-akka.actor.default-dispatcher-3] DEBUG o.m.w.processors.StaticFileProcessor akka://TestActorSystem/user/file-processor-test-router/$a - Compressed file name: /tmp/Temp_7097448265494079716/f6310e6f4d77113539594e81c62601cd.deflate

Note that the thread is: TestActorSystem-akka.actor.default-dispatcher-3.

This lead me to think that my pinned dispatcher configuration is not working. Why would the thread be named default-dispatcher-3? Does this mean that my Actors are using the default dispatcher rather than mydisptacher?

After discussions with the Akka team, I found that the misunderstanding comes from akka logging.

Akka logging is async and the thread printed in the log statement is the thread that is displaying the log message – NOT the thread that wrote or initiated the log message.

The fix is to add “%X{sourceThread}” to the logback pattern. For example: %d{HH:mm:ss.SSS} [%thread %X{sourceThread}] %-5level %logger{36} %X{akkaSource} – %msg%n.

Now my log message is:

22:15:44.471 [TestActorSystem-akka.actor.default-dispatcher-3 TestActorSystem-1] DEBUG o.m.w.processors.StaticFileProcessor akka://TestActorSystem/user/file-processor-test-router/$a - Compressed file name: /tmp/Temp_7097448265494079716/f6310e6f4d77113539594e81c62601cd.deflate

The thread in which my Actor is running and writing the log message is: TestActorSystem-1.

The thread displaying the log message is: TestActorSystem-akka.actor.default-dispatcher-3.

Here’s one for newbies …

I’ve been trying to learn scala and akka. I decided to write a netty based web server in scala and using akka.

To test my new server, I used ab.

ab works fine with no “keep alive”.

However, with “keep alive”, ab kept hanging after the first request. It eventually timed out.

Thanks to Sean’s post, I found that the problem is that my server needed to return the “Connection: keep-alive” header in the response!

According to Sean,

This behaviour isn’t specified anywhere, but it is suggested in an Internet Draft for HTTP/1.1 at w3.org:
The Connection header field with a keep-alive keyword must be sent on all requests and responses that wish to continue the persistence.

I’ll be sending a pull request to Netty shortly to add this to the http examples.

A good Cheet Sheet.

Fork

http://help.github.com/fork-a-repo/

# Clone to Spoon-Knife directory
git clone git@github.com:username/Spoon-Knife.git

# Clone to SpecificFolderName directory
git clone git@github.com:username/Spoon-Knife.git SpecificFolderName

cd Spoon-Knife
git remote add upstream git://github.com/octocat/Spoon-Knife.git
git fetch upstream

Committing Changes

# To commit changes to local repository
git commit -a

# Send changes to remote repository:
git push origin master

Update a Forked Repository

#Fetches any new changes from the original repo
git fetch upstream

# Merge changes retrieved from the fetch
git merge upstream/master

Creating a Branch

# Creates a new branch called "mybranch"
git branch mybranch

# Switch to mybranch
git checkout mybranch

# Push new local branch
git push origin mybranch

# Merge changes from mybranch into master and delete mybranch
git checkout master
git merge mybranch
git branch -d mybranch

# Delete remote branch after local delete
git push origin :mybranch

Getting a Remote Branch

http://stackoverflow.com/questions/1897475/github-how-to-checkout-production-branch

# To list branches:
git branch -a

#You could checkout directly the remote branch after fetching it
git fetch origin
git branch -f remote_branch_name origin/remote_branch_name
git checkout remote_branch name

# or shorter:
git checkout -b production origin/production

# Merge changes from remote production into local production
git co -b production
git pull origin production

Tagging

#Create local tag
git tag -m"Tag version 1.0" V1.0 

# Push tag to origin
git push --tags

Merge Patch Request Locally to Test it Out

# Every pull request has a .patch URL where you can grab a textual patch file to feed into the git-am command
$ git checkout master
$ curl http://github.com/github/jobs/pull/25.patch | git am
$ git push origin master

If more than 1 person is working on a branch …

To avoid “Merge branch ‘master’ of …” in your commit history, use the following to get the latest changes. See this link for more info.

$ git pull --rebase origin

Well, finally shipped a version of Chililog with decent websocket support. Can now stream console.log from user’s browser to my desktop browser.

Checkout the video and live demo at http://blog.chililog.org/2011/10/24/remote-javascript-logging/.

I’ve been working on a new open source project called Chililog.

I’ll be blogging about how I’ve used Netty, HornetQ, MongoDB and Sproutcore to build Chililog here.

What are WebSockets?

Current web browser communications protocol is limited to the HTTP request and response paradigm – i.e. the browser requests and the server responds.

What if we want to use a different communications paradigm?  For example, what if we want to perform 2 way communications where the server sends a request and the browser responds?  A common use case would be the server notifies the client that an event has occurred.

This is where WebSockets come into play. WebSocket is a technology providing for bi-directional, full-duplex communications channels, over a single Transmission Control Protocol (TCP) socket.

In addition, because WebSockets can co-exist with other HTTP traffic over port 80 and 443, firewalls will not have to be re-configured.

Version Confusion

WebSockets is an evolving standard.  Just have a look at the different implementations and the different versions each support.

There have been numerous version of the WebSocket standards under different names.  So far, browser have converged on 2 versions.

• Hixie-76/HyBi-00
• Safari 5+, Chrome 4-13 and Firefox 4 supports this standard.
• There are two names for this version because the Hixie-76 documentation is used as input into the new HiBi IETF working group.
• A flaw in this standard was discovered in the handshaking which requires exchange of binary data in the HTTP body.  This did not work across some proxy servers.
• HyBi-10
• Chrome  14, Firefox 7 and IE 10 Developer Preview supports this standard.
• Handshaking is performed in HTTP request and response headers
• Uses wire protocol version 8.  You will see “Sec-WebSocket-Version: 8″ in the HTTP header.

Hybi-00 and Hybi-10 both represents versions of the specification document.  The version of the wire protocol are actually 0 and 8 respectively.

Typically, the wire protocol (sequence of bits and bytes sent over the network) does not change between different versions of the specification document.  As such, the wire protocol version is set by the version of the specification document at which the change was made to the wire protocol.  So version 8 of the wire protocol was made in Hybi-08.

What changes are made between different versions of the specification document? Corrections of typos, clarification of concepts and adjustments in handshaking.

The latest version is Hybi-17 (with a wire protocol version of 13).  So far, no browsers have supported that version.

Netty WebSocket Support

I’m using Netty 3.2.5 in building Chililog (topic for another blog post).

Netty 3.2.5 supports Hixie-75 and 76 but NOT Hybi-10.

I’ve compiled together my own WebSocket package for Chililog to support both “Hixie-75/76/Hybi-00″ and “Hybi-10″.  This allows my Chililog server to support all of today’s major browsers (except for IE which does not support Web Sockets at all).

I’ve used the word “compiled” because I’ve extensively used code from Netty and Webbit (for which Aslak Hellesøy has written hybi-10 support).  I’ve also used code from cgbystrom to help build web socket clients.

I’ve submitted the work back to Netty as pull request #26.

Points of interest:

1. I’ve not changed the existing org.jboss.netty.handler.codec.http.websocket package.  I’ve found quite a few frameworks using this package.  I think it would be best if the next version of Netty can be a “drop-in” replacement.
2. I’ve put all my changes in org.jboss.netty.handler.codec.http.websocketx.  The “x” is intended to represent multiple versions.
3. The websocketx package supports both WebSocket versions (“Hixie-75/76/Hybi-00″ and “Hybi-10″)  for both client and server.
4. Frames
• Data is sent between client and server in frames.
• The old websocket package implements only the DefaultWebSocketFrame.  Text, binary and closing frames are encapsulated into this single class.
• The new websocketx package implements frames as a different class: TextWebSocketFrame, BinaryWebSocketFrame, CloseWebSocketFrame, PingWebSocketFrame and PongWebSocketFrame.  I felt that this made the code easier to read and understand.
5. Encoders and Decoders
1. Hixie-75/76/Hybi-00 is implemented as WebSocket00FrameDecoder and WebSocket00FrameEncoder.
2. Hybi-10 is implemented as WebSocket08FrameDecoder and WebSocket08FrameEncoder.  The version #8 is used because the wire protocol #8 is used in conjunction  with the specification document version #10.
6. Server Handshake
• Implements the handshaking protocol on the server side.
• Hixie-75/76/Hybi-00 is implemented in WebSocketServerHandshaker00  
• Hybi-10 is implemented in WebSocketServerHandshaker10  
• WebSocketServerHandshakerFactory picks the correct handshaker to use based on the handshaking request sent by the client.
• See org.jboss.netty.example.http.websocketx.server.WebSocketServer  for an example.
7. Client Handshake
• Implements the handshaking protocol on the client side.
• Hixie-75/76/Hybi-00 is implemented in WebSocketClientHandshaker00
• Hybi-10 is implemented in WebSocketClientHandshaker10  
  
• WebSocketClientHandshakerFactory picks the correct handshaker to use based on the version of the specification passed in as a paramter.
  
• See org.jboss.netty.example.http.websocketx.client.App  for an example.

Hope this helps anyone looking for Hybi-10 support in Netty.

27 Oct 2011 – Update

Pull #26 request has been merged into Netty.  This feature should be in upcoming Netty release 3.3.

20 Jan 2012 – Update

This has now been released in Netty 3.3. See blog post.

Kill the process TGitCache.exe.