Refactoring as Discovery

Refactoring usually falls into two modes:

• Small, local changes that make later improvements possible
• Large, structural changes that improve the overall design

Before the big changes can happen, you often need to do a lot of small, mechanical edits. These help clean things up, but they also serve as a way to re-learn the codebase. What developers often won’t admit is that they usually don’t know what the big change is going to be when they start. They just know the code isn’t right.

Semgrep: AutoFixes using LLMs

Semgrep:

Semgrep is an incredible tool that allows you to search code by matching against the Abstract Syntax Tree (AST). For instance, if you want to find all method calls named get_foo, you can write a pattern like this:

$A.get_foo(...)

Test your own patterns using the playground: https://semgrep.dev/playground/new

While there are other tools like this, semgrep is currently the most capable:

• https://ast-grep.github.io/
• https://codeql.github.com/
• https://comby.dev/

AutoFix:

Semgrep not only searches using patterns but also supports rewriting the matches. Here’s a simple rule definition from their documentation:

Git: programmatic staging

In the past year, I’ve been using a lot of tools to automatically rewrite/refactor code. These include semgrep, ast-grep, LLMs, and one-off scripts. After running these tools on a large code-base, you usually end up with lots of additional unintended changes. These range from formatting/whitespace to unrequested modifications by LLMs.

The subsequent “cleanup” step is a very manual and tedious process. I’m essentially running git add -p and staging hunks one at a time. At times it feels like this step offsets the productivity gain from the rewrite tool itself.

3D Printing

About a year ago, I purchased a BambuLab P1P 3D printer for about $1000 CAD (after taxes and shipping). This is something I’ve wanted to do on-and-off for the past 10 years and I finally decided to pull the trigger.

For my first few projects, I decided to print existing STL models I found on the internet instead of designing my own. This was a good idea at first, but eventually led me down the (very expensive) rabbit hole of 3D printed RC (Radio Control) vehicles. This was not ideal because it resulted in learning (and paying for) two separate hobbies at the same time: 3D printing + RC. The cost and complexity of the project was off-putting and the printer sat idle for a few months after that.

AWS X-Ray: Force Sample

Background

At my current workplace, we use X-Ray configured with a sample rate of 0.01. This means that a random 1% of requests will be traced. The low rate is great at keeping costs down, but it’s not useful for debugging specific failed requests. Fortunately, you can force X-Ray to sample your request by generating a trace id, and setting the X-Amzn-Trace-Id header.

Generating trace IDs

The id format is documented here along with a Python code sample.

Debugging a flaky Go test with Mozilla rr

This is how you debug a test that only fails once every 1000 times.

The Test

package my

import (
	"math/rand"
	"testing"
	"time"
)

func init() {
	rand.Seed(time.Now().UnixNano())
}

func TestRandFail(t *testing.T) {
	if n := rand.Intn(1000); n == 50 {
		t.Fatalf("finally got %d", n)
	}
}

• This is obviously a pedagogical example.

Get the newest version of rr

git clone https://github.com/rr-debugger/rr.git
cd rr
git checkout 5.4.0 # change this to the latest release (DO NOT BUILD HEAD)
mkdir build
cd build
cmake ..
make -j8
sudo make install

• Warning: rr does not work in VirtualBox.
• https://github.com/rr-debugger/rr/wiki/Building-And-Installing

Compile your failing test to a binary

go test -gcflags 'all=-N -l' -c

• The -gcflags 'all=-N -l' disables optimizations and inlining.

Install the following rrloop script.

#!/bin/sh

while :
do
  rr $@
  if [ $? -ne 0 ]; then
    echo "encountered non-zero exit code: $?";
    exit;
  fi
done

• rrloop is a wrapper around rr which keeps looping until it sees a non-zero exit code.
• This works because rr exits using the recorded process’ exit code.

Record the test execution in a loop until it fails

echo -1 | sudo tee -a /proc/sys/kernel/perf_event_paranoid
echo 0 | sudo tee -a /proc/sys/kernel/kptr_restrict

rrloop record ./my.test

• If you have multiple tests, you can pass -test.run=TestRandFail to only run a specific one.
• The --chaos flag often increases the chances of the failure.

Example Output:

Merging Multiple Git Repositories Into A MonoRepo

Note: Replace thing with your own repo name in the examples.

1. Create a repository which will store all your code

mkdir monorepo && cd monorepo
git init .
echo "# MonoRepo" > README.md
git add .
git commit -m "first commit"

2. Clone one of the existing repositories to a temporary location

Example Remote: ssh://git@code.company.com/thing.git

mkdir /tmp/thing
git clone ssh://git@code.company.com/thing.git /tmp/thing
cd /tmp/thing

3. Use git filter-branch to rewrite the history into a sub-directory.

Note: this step can take a very long time

Go: Composable http.Handler

When using net/http, handling errors is kinda annoying.

http.HandleFunc("/foo", func(w http.ResponseWriter, r *http.Request) {
	thing, err := storage.Get("thing")
	if err != nil {
		http.Error(w, err.Error(), 500)
		return
	}
	_ = json.NewEncoder(w).Encode(thing)
})

Ideally, I could just return an error from the handler. Let’s create a type to let that happen.

type MyHandlerFunc func(w http.ResponseWriter, r *http.Request) error

func (f MyHandlerFunc) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	if err := f(w, r); err != nil {
		http.Error(w, err.Error(), 500)
	}
}

Here’s the previous code refactored to use the new handler.

http.Handle("/foo", MyHandlerFunc(func(w http.ResponseWriter, r *http.Request) error {
	thing, err := storage.Get("thing")
	if err != nil {
		return err
	}
	return json.NewEncoder(w).Encode(thing)
}))

Better, but what if we want to control the error status code? There could be a special error type that MyHandlerFunc checks for.

Angular Events

I’ve been trying to find an elegant way of dealing with events in AngularJS recently. If you’re not farmiliar with Angular, that’s ok, this is a pretty common pattern.

Here I have a controller that registers an event listener:

function MyController($rootScope) {
  $rootScope.$on('event1', () => {
    console.log('event 1 occured');
  });
}

There’s an issue with this code. It doesn’t unbind the listener when the controller (or its scope) is destroyed. Let’s take care of this.

TypeScript: Working with JSON

EDITS:

• Calling toString on Date is for illustrative purposes.
• There’s a full commented example at the end.
• Use toJSON method as suggested by Schipperz.
• Add reviver method as suggested by Anders Ringqvist.

So you have a User type in your code.

interface User {
  name:    string;
  age:     number;
  created: Date;
}

At some point you’re going to want to encode this as JSON. This works as you’d expect.

 > JSON.stringify({ name: "bob", age: 34, created: new Date() });
'{"name":"bob","age":34,"created":"2016-03-19T18:15:12.710Z"}'

The problem is that the created field is no longer a Date when you parse it back.

Custom JSON Marshalling in Go

Go’s encoding/json package makes it really easy to marshal structs to JSON data.

package main

import (
	"encoding/json"
	"os"
	"time"
)

type MyUser struct {
	ID       int64     `json:"id"`
	Name     string    `json:"name"`
	LastSeen time.Time `json:"lastSeen"`
}

func main() {
	_ = json.NewEncoder(os.Stdout).Encode(
		&MyUser{1, "Ken", time.Now()},
	)
}

Output:

{"id":1,"name":"Ken","lastSeen":"2009-11-10T23:00:00Z"}

But what if we want to change how one of the field values are displayed? For example, say I wanted LastSeen to be a unix timestamp.

The simple solution is to introduce another auxiliary struct and populate it with the correctly formatted values in the MarshalJSON method.

TypeScript completion in Vim

One of the main advantages of using static types is that you get much better support from your tools. I recently got TypeScript auto-completion working in vim and I’m documenting how to do it here.

Demo:

1. Install TSS

git clone https://github.com/clausreinke/typescript-tools.git
cd typescript-tools
git checkout testing_ts1.4
sudo npm install -g

2. Install Vim Plugin

I’m using Vundle to manage my plugins.

Bundle "icholy/typescript-tools.git"
au BufRead,BufNewFile *.ts  setlocal filetype=typescript

3. Install TSD

sudo npm install tsd@next -g

4. Create Project

mkdir project
cd project
tsd init

tsd install jquery --save
tsd install angularjs --save

5. Create tsconfig.json

{
  "compilerOptions": {
    "target": "es5",
    "noImplicitAny": false,
  },
  "files": [
    "typings/tsd.d.ts",

    "all.ts", "your.ts",
    "other.ts", "files.ts"
  ]
}

6. Start TSS in vim

Make sure you’re cwd is somewhere in the directory containing tsconfig.json

interactive filtering with less

I discovered a cool little feature in less (not less.css) today. You can interactively filter the data.

&pattern

	Display only lines which match the pattern; lines which do not match the pattern are not displayed. If pattern is empty (if you type & immediately followed by ENTER), any filtering is turned off, and all lines are displayed. While filtering is in effect, an ampersand is displayed at the beginning of the prompt, as a reminder that some lines in the file may be hidden.

	Certain characters are special as in the / command:

Activate: &pattern hit enter
Disable: & hit enter

C++: Make Repl

One of the things I really like about dynamic languages like javascript & python is the repl. After you’ve gotten used to that type of exploratory programming, it’s hard to go back to the edit/compile/run cycle.

Luckily that has finally changed with cling. It’s an interactive C++ environment that behaves pretty much like a repl. In my recent projects I’ve been adding a new make rule: repl which lets me interactively play with the code I’m working on and it has drastically improved my productivity.

C++: Inline Functions

Even though overuse of getter and setter functions can be frowned upon, they can help a lot if you’re looking to provide a intuitive api. However the overhead the additional function call introduces is undesirable. Thankfully, there’s the inline keyword. It tells the compiler to replace each invocation of the function with the body of the function.

struct Foo {
  int m_number = 123;

  inline int number () {
    return m_number;
  }
};

int main () {
  Foo foo;

  // used like a regular function
  std::cout << foo.number() << std::endl;

  // compiled to almost identical assembly as this
  std::cout << foo.m_number << std::endl;

  return 0;
}

However the inline keyword isn’t a guarantee that the compiler will do this. It’s more of a hint to the compiler. So keep in mind that the compiler is free to ignore the fact that a function is declared inline and it can inline functions that haven’t been delcared as such. But in examples similar to the one above, you can assume it will behave as expected.

Vim Marks

Marks are a feature that I’ve never really used enough. Hopefully writing about them will change that for the better.

Make a basic, file local, mark called a

ma

Jump back to that mark

'a

Now I try to be pragmatic. So use cases are what motivate me to learn new thing. I think that marks are a good replacement for a lot of the things I use visual line V mode for now.

C++ Extending Classes via the Stream Operator

Vision

Looking for a way to create a class which behaved like one of the std::ostream classes.

MyClass obj;

obj << "foo" << 123 << some_string.c_str();

Problem

Implementing all those operator<< overloads would be redundant because something like std::stringstream already does it. However inheriting from std::stringstream is more complicated than it should be.

struct MyClass : public std::stringstream {
    /* not that simple ... */
};

Solution

You can use a simple template to achive the desired effect.

struct MyClass {

    std::stringstream m_ss;

    template <class T>
    MyClass & operator<< (T const& rhs) {
        m_ss << rhs;
        return *this;
    }
};

This comes with the benefit being able to ‘hook’ into each invocation.

C++ Log4cxx vs Glog vs Boost.log vs Wrapper

It seems that logging in C++ isn’t a much discused topic when compared to a language like java. In a recent C++ project, I needed to add real logging support. Up till this point, the following was good enough (don’t judge).

#ifdef DEBUG
    std::cerr << "some error" << std::endl;
#endif 

I started googling and the following to be the most popular and mature.

glog

homepage

glog was my first choice because it’s the simplest one to set up and it has hardly any dependencies. The interface is also nice to use.

Libpq: PQexec Timeout

1. Establish the connection

PGconn *pg_conn = PQconnect("info");

// error check
if (PQstatus(pg_conn) != CONNECTION_OK) throw "invalid connection";

2. Grab the socket file descriptor

int socket_fd = PQsocket(pg_conn);

// error check
if (socket_fd < 0) throw "invalid socket";

3. Set the timeout

// 5 second timeout
struct timeval timeout = { 5, 0 };

// recv timeout
int setopt_result_1 = setsockopt(
    socket_fd,
    SOL_SOCKET,
    SO_RCVTIMEO,
    (char *)&timeout,
    sizeof(timeout)
);

// send timeout
int setopt_result_2 = setsockopt(
    socket_fd,
    SOL_SOCKET,
    SO_SNDTIMEO,
    (char *)&timeout,
    sizeof(timeout)
);

// error check
if (setopt_result_1 < 0 || setopt_result_2 < 0) throw "failed to set timeout";

SWAPM: Code generation made easy.

I finally got around to reading the Pragmatic Programmer book. One thing that really interested me was the section on Code Generation. So in a recent C++ project I was interfacing with postgres and there was a LOT of code repetition. The sql query, class members, getters/setters, response parsing logic. They all contained the same information. Perfect I thought, here was the ideal chance to give code generation a shot. My first incarnation was a very ugly perl script (aren’t they all .. ?) which used mustache as the templating engine. It worked, But I had to copy and paste the generated code into my source every time it changed which was a pain.

A week with Vim

During the past week I’ve been learning to use Vim (gVim). Day 1 and 2 weren’t fun to say the least. But now I’m completely hooked. I’m the type of person who will sit there for hours customizing my development environment until I think it’s perfect. I’ve been playing with almost every cool plugin i can find (and wasting a lot of time in the process).

So Vim is, without a doubt, the best text editor I’ve ever used. However, that’s all it is. It’s just a text editor and I know a lot people don’t agree with me on this one, but IDE’s do help. They just make everything better. Therefore the ultimate combination would be an ide with vim as the text editor.

Ember.js with Brunch

I’ve recently discovered the brilliant Ember.js library and the first major issue I ran into was how to organize/modularize this thing!? At first I just opted into RequireJs because that’s what I know but I started hitting walls fast. I decided to try out the Brunch build system since I had heard good things about it before and this was a great opportunity to learn how to use it. Brunch uses skeletons which are essentially project templates to get rid of the boilerplate. Google search “ember brunch” and I found charlesjolley/ember-brunch perfect!

CSS Compass Gradient Generator

This is a css gradient generator that i’ve been using for a while:

• http://www.colorzilla.com/gradient-editor/

CSS Output

background: #1e5799; /* Old browsers */
background: -moz-linear-gradient(top, #1e5799 0%, #2989d8 50%, #207cca 51%, #7db9e8 100%); /* FF3.6+ */
background: -webkit-gradient(linear, left top, left bottom, color-stop(0%,#1e5799), color-stop(50%,#2989d8), color-stop(51%,#207cca), color-stop(100%,#7db9e8)); /* Chrome,Safari4+ */
background: -webkit-linear-gradient(top, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%); /* Chrome10+,Safari5.1+ */
background: -o-linear-gradient(top, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%); /* Opera 11.10+ */
background: -ms-linear-gradient(top, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%); /* IE10+ */
background: linear-gradient(top, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%); /* W3C */
filter: progid:DXImageTransform.Microsoft.gradient( startColorstr='#1e5799', endColorstr='#7db9e8',GradientType=0 ); /* IE6-9 */

However I just noticed the switch to scss option!

SCSS Ouput

@include filter-gradient(#1e5799, #7db9e8, horizontal);
$experimental-support-for-svg: true;
@include background-image(
    linear-gradient(left, #1e5799 0%,#2989d8 50%,#207cca 51%,#7db9e8 100%));

This makes implementing complex cross-browser css gradients painless.

VMware Workstation Ubuntu problems

I just tried starting up vmware workstation and was greeted with a message saying it needed to compile some modules and then went on to fail this step no matter what. This is an issue I’ve encountered before on Ubuntu 11.04 and now on 11.10.

This is a bug with all v7.x of workstation and can be fixed with a simple patch I found today at http://weltall.heliohost.org/wordpress/2011/05/14/running-vmware-workstation-player-on-linux-2-6-39-updated/

How to use it:

QML is Awesome

QML is Nokia’s recent addition to its well known Qt framework and comes part of the Qt Quick Suite

The way I describe it to people is:

it’s like html and css combined with the power of Qt in a extremely simple syntax.

Why?

I have used Swing, WinForms, and GTK in the past and never really liked anything to do with GUI work. QML is the first time I’m actually enjoying designing user interfaces and this results in a better end result. It feels a lot more like web development except without browser specific issues.

Qt Creator + Boost on Ubuntu 11.04

1. make a home for boost

sudo mkdir -p /code/include
sudo chown -R YOUR_USER_NAME /code
cd /code/include

2. download boost

sudo apt-get install subversion
svn co http://svn.boost.org/svn/boost/trunk boost
cd boost

3. compile boost

sudo ./bootstrap.sh
sudo ./b2

note: this will take a while, go get some coffee.

4. Include in qt project

Add the following to your .pro file.

INCLUDEPATH += /code/include/
LIBS += -L/code/include/boost/bin.v2/libs/ -lboost_system -lboost_filesystem -lboost_asio

In this example i’m linking boost::filesystem and boost::asio. boost::system is required by other boost libraries but if you can compile without it, then trash it.

Compile CompassApp on Ubuntu 11.04

1. Install RVM

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm);
echo 'if [[ -s "$HOME/.rvm/scripts/rvm" ]]  ; then source "$HOME/.rvm/scripts/rvm" ; fi' > ~/.bashrc
rvm install 1.9.2

2. Install jRuby

rvm install jruby
cd ~/.rvm/bin/jruby-1.6.4 -S gem install rawr

3. Get and Compile CompassApp

git clone https://github.com/handlino/CompassApp.git
cd CompassApp
bin/build-all.sh

4. Run CompassApp

cd package/compass.app/
./run.sh

Cloud9 IDE

I’ve always wanted to like web based IDE’s. However, there was one thing that always prevented it: they’ve always been terrible.

Until now that is. http://cloud9ide.com/ Cloud9 is epic. It’s built on node.js and has support for coffeescript and sass syntax highlighting and real time error checking. I can’t even find an desktop ide to do that right!

It gets better though. It’s 100% open source so you can install it on locally. I kinda used this as a general guide but didn’t need most of it

Sass Compass Blueprint

So I just realized that I hadn’t actually written anything about compass. Now I feel a little dumb about the title of the Formalize post but w.e shit happens. Anyway I’ll be talking about css in this post. I started using these a while back so I don’t really know why I haven’t posted anything about it. Better late than never.

Let’s start with SASS. Syntactically Awesome Style Sheet

well actually I’ll be talking about SCSS which is sass syntax which is more like css. Well actually it’s exactly like regular css except you can add way more stuff. but regular css is still valid scss. I’m not going to try to explain so here’s an example.

CoffeeScript

I just spent the last 5 hours learning CoffeeScript and I feel like I have pretty much everything down. My brain is kinda dead right now, but at the same time I’m pretty excited to actually try it in a real project. In case you don’t know CoffeeScript is a python-esque language which ‘compiles’ into javascript. classes, list comprehension inheritance, ranges, semantic code etc…. dream come true. http://jashkenas.github.com/coffee-script/

One thing I was worried about was being able to use 3rd party libraries with it. It’s actually not different at all… Once you figure it all out, you realize that it’s still the exact same js you’re working with and you can do just as much. It’s just a lot less shitty. Yes, the learning curve is balls, but it’s definitely worth it (i hope).

Formalize [More Compass]

I think forms and all things related ( inputs, buttons, etc… ) are probably one of the more annoying things when building a website. They’re just not consistent and it takes a lot of effort to make them look decent. I spent some time looking for a tool to help me with this and I ended up with formalize which comes as a compass plugin and integrates with any web framework you’re using. http://formalize.me/ It actually didn’t jump out at me the first time around when I checked out the site. It didn’t ‘dazzel’ me so I kept looking, but what I realized was that it shouldn’t stand out. It’s a form… You don’t want it to be shiny with crazy animations. So I went back and used it for the project I’m currently working on. It was perfect… enough said.

Reloader - multi browser live web preview

I recently started developing on linux and unfortunately stylizer 5 does not support linux. So I’m back to using kate. However, one thing that I really missed right away was the instant preview feature. Having to go and refresh multiple browsers every time you change a line of code blows. I searched around for a bit and found a few tools but none of them were any good. I needed something that would work in multiple browsers at the same time and I couldn’t find anything to my liking so I wrote my own.

JPProxy - tiny jsonp proxy

JPProxy is a very simple yet powerful JSONP script. It allows you to make ajax like requests to any page on a server that has the jpproxy.php script on it. I tried really hard to make it as simple and generic as possible so the source is tiny.

1. Client

A script tag is injected into the DOM and all the values are added to the url as GET parameters.

Skybound Stylizer 5 - CSS Editor

Lets start with a little preface. Prior to finding Stylizer, I was completely happy using a regular text editor (gedit, notepad++) with firebug to do my css coding. I don’t really know how I found stylizer, or what motivated me to download it, but I am glad I did. Stylizer is, by far, the best css editor. I went on to try 10+ different editors in hopes of finding a free alternative and nothing even comes close. This is the first piece of software I bought a legitimate licence for and $80 is a lot for a broke ass student.

Balsamiq Mockups - wireframing done right

Senario:

You’re designing some type of user interface. Clients never know what they want (even when they think they do) so it’s usually a good idea to come prepared with a basic design to go off. You quickly whip together something in photoshop and think you’re good to go.

This is how the conversation goes:

Me: I threw together this mockup of a potential design. It’s very rough so just look at the basic layout and ignore details.

absoluteFudge - ie6 absolute positioning

I don’t know about you, but here is a snippet of css that I love.

div#selector {
   position: absolute;
   left: 10px;
   right: 10px;
   top: 10px;
   bottom: 10px;
}

assuming that the parent element has either relative or absolute positioning, the child div will fit inside with a 10px margin. This is a very powerful technique for creating liquid css layouts.

Problem

The problem is, that ie doesn’t support giving values to all sides (top,bottom,left,right) so you were forced to have a separate stylesheet for ie with a static layout.