Recently I stumbled upon the book Writing Idiomatic Python by Jeff Knupp, and I wanted to grab a copy of it. It wasn’t something I could pay for, so I emailed Jeff asking him if he could give me a free copy, and he was kind enough to agree. Well, I just finished the book and I really loved it !

The book is short and sweet and conveys powerful idioms in a clear and concise way. I have read quite a lot about what is good and bad in Python and how to write good code in it, and yet I was able to learn some new things out of this book. That being said, if you don’t know the basics of Python, then this is not the book for you.

Well, I’m glad that I’m subscribed to the updates of this book. Oh yea, and whether you’re a fanboy or an expert at Python, you’ve gotta read this book atleast once.

The settings I use for Sublime Text 2:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

{
    "color_scheme": "Packages/User/Tomorrow-Night.tmTheme",
    "detect_slow_plugins": false,
    "draw_white_space": "all",
    "find_selected_text": true,
    "fold_buttons": false,
    "folder_exclude_patterns":
    [
        ".svn",
        ".git",
        ".hg",
        "CVS",
        "_build",
        "dist",
        "build",
        "site"
    ],
    "font_face": "Ubuntu Mono",
    "font_options":
    [
        "subpixel_antialias"
    ],
    "font_size": 13,
    "highlight_line": true,
    "ignored_packages":
    [
        "Vintage"
    ],
    "rulers":
    [
        80,
        100
    ],
    "soda_classic_tabs": true,
    "theme": "Soda Dark.sublime-theme",
    "translate_tabs_to_spaces": true,
    "trim_trailing_white_space_on_save": true
}

Plugins

• Emmet-Sublime
• SublimeLinter
• SublimeREPL
• SideBarEnhancements
• Markdown Preview
• Git

Others

• Tomorrow-Night color scheme
• Soda Dark theme

Recently I stumbled across an article talking about the importance of understanding type and object in Python, so thought of writing a small post on it.

Looking indepth, one may get the impression that this is more of a chicked-egg problem, but I’ll give an overall view of this.

- An object is basically an instance of type
- type is an instance of type itself
- object is the subclass of all other objects
- a type is an instance of object itself

So in a way, you have only two objects in Python, types and non-types.

Note: type can also be called class from Python >= 2.3

Well there you go. It wasn’t confusing now, was it ? :)

Before you install Pylibmc on your machine, there are a few dependencies that you need to take care of, so as to ensure a successful installation.

Install python-dev

sudo apt-get install python-dev

Install libmemcached

sudo apt-get install libmemcached-dev

Now you can successfully install pylibmc using pip

Install pylibmc

sudo pip install pylibmc

Public Repo Creation

• Create a Sonatype Repo which can be used for the deployment of the project jars. For this, follow Step 2 and 3 from here.

• After u create the JIRA ticket, the guys responsible will get back to you with all the relevant details including the repositories to which you can deploy

• Now we have a snapshot and release repo ready. So let’s ready our project for deployment.

Changes in your POM file

• Make sure you have all these elements in your POM file

1
2
3
4
5
6
7
8
9
10
11
12

<modelVersion> 
  <groupId>
  <artifactId>
  <version>
  <packaging>
  <name>
  <description>
  <url>
  <licenses>
  <scm><url>
  <scm><connection>
  <developers>

• Make sure that you append -SNAPSHOT to <version>. Don’t worry, when your project jar is being deployed to the release repo, the “-SNAPSHOT” string will be removed.

• Add the following xml snippets in your POM

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

<plugins>
  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-compiler-plugin</artifactId>
      <version>2.3.2</version>
      <configuration>
          <source>${java.version}</source>
          <target>${java.version}</target>
      </configuration>
  </plugin>
  <plugin>
      <artifactId>maven-assembly-plugin</artifactId>
      <version>2.3</version>
      <configuration>
          <descriptorRefs>
              <descriptorRef>jar-with-dependencies</descriptorRef>
          </descriptorRefs>
      </configuration>
      <executions>
          <execution>
              <phase>package</phase>
              <goals>
                  <goal>single</goal>
              </goals>
          </execution>
      </executions>
  </plugin>
  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-source-plugin</artifactId>
      <version>2.1.2</version>
      <executions>
          <execution>
              <id>attach-sources</id>
              <goals>
                  <goal>jar</goal>
              </goals>
          </execution>
      </executions>
  </plugin>
  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-javadoc-plugin</artifactId>
      <configuration>
          <quiet>true</quiet>
          <nonavbar>true</nonavbar>
          <notree>true</notree>
          <nocomment>true</nocomment>
          <nohelp>true</nohelp>

      </configuration>
      <executions>
          <execution>
              <id>attach-javadocs</id>
              <goals>
                  <goal>jar</goal>
              </goals>
          </execution>
      </executions>
  </plugin>
  <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-gpg-plugin</artifactId>
      <executions>
          <execution>
              <id>sign-artifacts</id>
              <phase>verify</phase>
              <goals>
                  <goal>sign</goal>
              </goals>
          </execution>
      </executions>
  </plugin>
</plugins>

• Add the distribution info like this:

1
2
3
4
5
6
7
8
9
10
11
12
13

<distributionManagement>
  <!-- Repository for snapshots -->
  <repository>
      <id>sonatype-nexus-snapshots</id>
      <url>https://oss.sonatype.org/content/repositories/snapshots</url>
  </repository>
  <!-- Repository for releases -->
  <repository>
      <id>nexus-releases</id>
      <name>Nexus Release Repository</name>
      <url>http://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
  </repository>
</distributionManagement>

• Add the Sonatype parent info:

1
2
3
4
5

<parent>
  <groupId>org.sonatype.oss</groupId>
  <artifactId>oss-parent</artifactId>
  <version>7</version>
</parent>

• Create a file settings.xml inside your local Maven repository folder (which is usually ~/.m2/ ), and add the following xml snippet

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

<settings>
  <servers>
      <server>
          <id>sonatype-nexus-snapshots</id>
          <username>your username</username>
          <password>your password</password>
      </server>
      <server>
          <id>nexus-releases</id>
          <username>your username</username>
          <password>your password</password>
      </server>
  </servers>
  <profiles>
      <profile>
          <id>gpg</id>
          <properties>
              <gpg.passphrase>your passphrase</gpg.passphrase>
              <gpg.keyname>your pgp key</gpg.keyname>
          </properties>
      </profile>
  </profiles>
</settings>

The username and password refers to the one you used for creating the jira account earlier.

Using Public key to sign artifacts

• You can sign your project jars using pgp signatures by following the instructions here

• Make sure to upload your key to the public keyserver as mentioned in the link above.

• The maven-gpg-plugin plugin above will help to sign all your project jars with a pgp signature.

Upload to Repository

• Before you proceed, make sure to push all the changes made locally to your SCM. This is a very important step.

• Create a public SSH key in github, if you don’t have one. If not created, you will face the following error while uploading

1

Permission denied (publickey). error.

• To upload your project to the snapshot repository simply run

1

mvn clean deploy

• To upload your project for staging before the actual release, run the commands below

1
2
3
4
5

//Cleanup before releasing 
 mvn release:clean

//Preparing for the release
 mvn release:prepare

• If your project was prepared successfully, use the following command to upload the jars to the staging repository.

1

mvn release:perform

If everything ran well, you have now successfully deployed your project artifacts to the sonatype staging repository.

Promote staged project to Maven repo

• Login to Sonatype Repository browser
• Follow the instructions given here
• Before you release your jars into Maven Central, make sure to download and test them locally.
• Do not release your artifacts without ensuring that they work as expected

Once your staged release has been promoted and synced to the Central Repo, do not forget to close the JIRA ticket you created in the beginning.

That’s it. You’re done ! Go and chillout :)

Well, I’ve never actually given importance to the term ”profiling” till a few days ago. The real deal was that I was just too lazy to actually run a profiler and then to try interpret the stats and hopefully find the bottleneck in the code. So to avoid all this hassle, I tried to be careful while writing code to make sure I don’t leave any open doors for memory leaks or poor CPU utilization.

But a few days back, I stumbled into a performance issue in my code at work, leaving both me and my project lead puzzled about the culprit. Atlast desperation lead us to setup YourKit profiling in our server to have a peak under the hood of the JVM. And in the process, we happened to learn some new terms and functionalities of the profiler and I must say that I’m slowly beginning to like the idea of profiling code. I was really surprised at what we could learn about our code just by looking at the profiler stats. So here below, I’ve merged in some of our learnings along the whole process.

JVM Memory Structure

It’s important to first understand the structure of the JVM memory itself. Simply put:

JVM Memory = Heap Memory + Non-Heap Memory + Other (JVM code itself, JVM internal structures, loaded profiler agent code and data, etc.)

Profiling Terms

Next, lets look at a few terms you need to know in order to make some sense out of all those weird graphs.

• Heap Memory - Stores Java objects
• Non-Heap memory - Stores per-class structures and other meta-data
• Stack Memory - Java only stores primitives on the stack. Stack values only exist within the scope in which they were created. Java objects are created on the heap, and only references (which in turn are primitives) are passed around on the stack.
• Allocated Memory - All memory that is allocated by the JVM
• Used Memory - Amount of memory that is actually in use
• Eden - Objects that are newly created reside here
• S1(Survivor Space 1) or S2 - All objects that survive a GC cycle go here
• Young Generation - Eden + S1 + S2 section of Heap memory

• Old Gen Space - Objects that survive a fixed number of GC cycles go here

  Ex: Singletons, Cached Objects and other application global data

• Shallow Size(object) - Amount of allocated memory to store the object
• Retained Size(object) - Shallow size of object under consideration + Shallow size of other objects that can be directly or indirectly accessed from this object

Things to watch out for while profiling

• In a typical application there will be less variation inside the Old Gen space.
• If the Old Gen space grows linearly with time even after the GC cycle, it would lead to a OutOfMemoryError. This occurs mainly due to JVM memory space depletion (Java Heap, Native Heap, etc.). This might be a indication of a memory leak inside the code.

The OutOfMemoryError mainly occurs due to the following:

• Lack of understanding of current JVM utilization
• Inappropriate JVM GC policy used. (Monitor JVM GC closely. Enable verbose GC option and analyse the data)
• Lack of performance and load testing

From the above, you probably would understand most of the important stats when you look into your profiler output next time. Now comes the important question

How do I generally find/prevent bottlenecks in my code ?

Well, I’ve tried to provide a few ideas below:

• Understand your application’s memory footprint
• Provide Connection Timeouts for any sort of communication API that you use.
• Cache data whereever possible to avoid more I/O operations on disk
• Monitor your threads closely to make sure there aren’t any bottlenecks or, well Deadlocks (gulp)
• Monitor Network Latencies. They are the hidden culprits of many performance problems.
• Implement application logging. This will help you to internally monitor the performance issues and hotspots in your code.
• Use Client JVM for development and Server JVM in production. Most people ignore this change. Even I did so. That was mainly because I never knew the difference between the JVM’s. The Server JVM does heavy optimization of your code to increase performance and reduce memory and CPU footprints, in production.
• Increase JVM Heap by specifying -Xms and -Xmx while running your application
• Analyse what’s happening under the hood in the heap, using jmap and jhat (Haven’t really tried this out)

In this post, I’ll share a simple way to download CSV files in Play Framework. This is especially useful, if you want to export out some reports or analytics data for offline use. So let’s get started.

1. First create a new Play! application (I’m skipping all this for the sake of sticking to the point)
2. Have a seperate controller/method which handles the generation of CSV files.
3. In that controller add the following code:

1
2
3
4
5
6
7
8
9

response.contentType = "text/csv";

response.setHeader("Content-Disposition", "attachment;filename="" + <filename> +".csv" + """);

response.writeChunk(<header>); 

response.writeChunk("\n"); 

response.writeChunk(<content>);

Note: Remember to separate your content with a “,” so that it will be columnised in the file

This is my first post, so thought of starting up with the most widely used programming lingo: Hello World. In this blog, I’ll share my hacks,ideas and even mistakes :)