Github Pages and Yard

A quick post to document how I set up my yard documentation to use with Github pages. For an open source project I’ve been working on called switch_gear I wanted to host my code documentation on Github pages.

Github has a few options for this. If you go to the repo settings, you’ll see the area in question:

Here I’ve set this project to render a page from the /docs folder on the master branch. Normally however when you execute:

$ yardoc

The output is actually put in the directory doc. In order to change that you need to run:

$ yardoc -o docs

Now that we’ve got the right folder we can push up to Github (either on a feature branch or directly on master). It can take a few minutes but you should see your yard documentation served.

In order to automate this, I created a git hook that lives in your local project under .git/hooks/pre-push. Make sure you give executable permissions to that new file:

$ chmod u+x .git/hooks/pre-push

Then in that file paste:

#! /bin/bash

echo "Running documentation"

files=$(git diff --name-only)
yardfiles=$(sed 's/ /,/g' <<< $files)
yard doc -o docs $yardfiles

echo "Including doc changes in a separate commit"

git add docs/
git commit -m "Updated documentation"

Now with every push you’ll ensure your documentation is re-rendered for files changed. There is a downside to this though, something I haven’t been able to figure out yet. When yard generates docs it puts a timestamp in the HTML that git picks up as new changes. Ideally we could suppress those file changes if that’s all that changed. Looking at Yard’s source, it appears you’d have to write your own templates to make that work. I’ll update this post if I go down that route but for now this is good enough for me.

Thanks for reading.

6 months without a debugger

Six months ago I started a new job at commercetools and while I had written a few small node.js apps, I did not have anywhere close to the dev setup I had with ruby. With ruby, I had become an expert, no a master, rather I was reliant on pry for my day to day.

Pry was so ingrained in my normal workflow that I set up vim shortcuts to put a binding.pry under my cursor whenever needed, via:

" add a require 'pry' and binding.pry at current cursor location
map <leader>bp :s/\(^.*\n\)/require 'pry'\rbinding.pry\r\1/g<cr>:noh<cr>3k==2.2j:w<cr>

Later to get rid of them, I made another vim shortcut to grep through all the text to delete any trailing binding.pry’s in the code. So, you can imagine my workflow:

1. Run test!
2. Failure
3. Darn it
4. move to file:line in stacktrace
5. add a binding.pry
6. re-run test
7. Oh.
8. Fix.
9. Repeat

That worked really well and I got fancy with the pry prompts. For instance, show-doc, show-source, ls and my favorite whereami. I’d do the same when debugging gems, if my stacktrace pointed to a gem I’d do:

1. bundle open some_gem
2. move to file:line in stacktrace
3. add a binding.pry

I’ve found bugs in the mongo ruby driver using this technique.

Now though, in new territory, using node.js I decided to stop being so reliant on a debugger. This wasn’t a “I’m too good for a debugger”, it was just me reevaluating whether a tool I was using had become a crutch. I can remember times where I was in that 10 step loop (see above) endlessly and I thought to myself, this is a really slow way to develop software.

Of course, the internet has no shortage of opinions on the matter. I know what Linus has said, what Robert Martin said and that Kernighan once wrote that the most effective debugging tool is still careful thought, coupled with judiciously placed print statements.

All of those opinions are completely valid, but in my time developing software I can say with confidence there is no absolute truth to finding and fixing issues in code. I’ve done it 100 ways, some which took me seconds and some which took me weeks.

So I decided to not look for a node.js equivalent to pry and that meant I fell back on the ol’ C way of debugging that I grew up on, print statements. Or rather in node.js case, console.log statements. It was not an easy transition. There were two main pain points early on, 1. I was still learning node.js and learning a language with print statements is slow. Very, very slow. 2. Console.log did not always give me the output I was expecting. In fact many objects in javascript don’t really respond to toString() in a meaningful way.

I had to start creating some tooling again to help me because either I’d get buried in documentation or I’d go digging through source code just to find out what methods an object had. Here comes more vim keys…

map <leader>jsm :s/\(^.*\n\)/console.log(`Available methods: ${Object.getOwnPropertyNames(someObj)}`);\r\1/g<cr>:noh<cr>3k==2.2j:w<cr>

I’d hit <leader>jsm and it adds this line:

console.log(`Available methods: ${Object.getOwnPropertyNames(someObj)}`);

I’d then replace sombObj with the variable in question and rerun program. I’d then get a print out of methods to the screen. How about inspecting an object? Well if it was my own object I was forcing myself to always write a toString() method on any class. That’s fine for my own objects but for objects I didn’t want to monkey patch, I’d have another vim command that did this:

console.log(JSON.stringify(someObj));

You’ve all done that before I’m sure.

I’d gotten to the point where I couldn’t run the program from the commandline anymore because my thumbs were going to fall off. That led me to making sure I could run a node file at any time via <leader>e. That command would then know how to execute the file based on the extension, it looks something like this:

" execute file if we know how
function! ExecuteFile(filename)
  :w
  :silent !clear
  if match(a:filename, '\.rb$') != -1
    exec ":!bundle exec ruby " . a:filename
  elseif match(a:filename, '\.js$') != -1
    exec ":!node " . a:filename
  elseif match(a:filename, '\.sh$') != -1
    exec ":!bash " . a:filename
  elseif match(a:filename, '\.c$') != -1
    exec ":!gcc " . a:filename . " ; ./a.out"
  elseif match(a:filename, '\.go$') != -1
    exec ":!go run " . a:filename
  else
    exec ":!echo \"Don't know how to execute: \"" . a:filename
  end
endfunction

I also set up a similar command to run tests via <leader>t. It would look at the file extension and run the test file.

So what was my 10 step process with pry is now a little different with node.js.

1. Run test!
2. Failure
3. Darn it
4. move to file:line in stacktrace
5. add a console.log statement
6. re-run test
7. See output.
8. Fix.
9. Remove console.log statement
10. Repeat

So still 10 steps. However, I did notice a conscious change in behavior. Before with pry, I wouldn’t start to debug my code until I was at the familiar pry prompt. Yes, you read that right. I would instinctively add the binding.pry and just rerun and wait for the prompt before starting to check local variables, method definitions, etc. I imagine that just manifested out of habit and speed - there are many times in the day where my fingers are faster than my brain. With my node.js workflow, I started to notice myself combing through code line by line with much greater detail because it was so slow to debug with print statements. I didn’t want to keep adding print statements for every variable I forgot and I was tired of adding toString() to every single object when I was primarily just enforcing that API because of debugging.

Now 6 months later, I decided to see if using the node debugger would be a welcome change to my workflow. I’ve found that I don’t care to use it when it’s my own code. I’m now in the habit of really spending time with the error or test failure, going back through the code in a meaningful way with a bit more information than I had before. It’s slowed me down in terms of key commands but that metric is useless in programming. Instead, I’m focusing on the problem and it’s leading me to better results. If you find yourself rifling through code or tests but anchored to a debugger to get the job done, take a step back for a few days and see what comes of it. I find I’m writing a lot less ‘quick code’.

I’ve set a new goal now - I’m going to try to not use the debugger on code that I have written. For code that I didn’t write, such as the zillion npm packages that are out there - I’m giving myself a bit more wiggle room. These packages, most of which are still in ES5, are not easy for me to navigate. I’d say I’m proficient in javascript now but for large libraries, there is no way for me to debug someone else’s code quickly. Instead, I’m allowing myself to use the debugger when the stacktrace is pointing to something in my node_modules folder. I haven’t automated this just yet but I will and when I do, I’ll update this blog post. Thanks for reading!

Jump to github from your shell

I’ve been going through the ‘Learning the bash Shell: Unix Shell Programming’ by Cameron Newham and ‘Classic Shell Scripting’ by Arnold Robbins over the last few months. I’ve learned a lot about the *nix way and I’m confident there is a tremendous amount more to learn.

The books encourage you to think about repetitive tasks you make on your day-to-day and try to automate them. I find myself constantly checking in new code locally with git and then going to github to either find the issue number I’m working on, or to reference a pull request by another team member. That means I’m sitting at my bash prompt, I leave to go to chrome, I go to chromes address bar and hope I remember the repository name. Wait, was this on github or bitbucket? Is the local directory the same as the repository name on github? That’s when I realized I should try to automate this.

This is not rocket science and I’m merely sharing because it’s saving me a lot of time, especially for it being so simple. Hopefully it helps others as well.

Here’s the bash script, go ahead and copy this into where you have user-wide scripts available. I like to put mine in ~/scripts, go ahead and name this remote.

#! /bin/bash

endpoint=$1

get_ext() {
  case $1 in
    *github*)
      case $endpoint in
        pr)
          echo "/pulls";;
        issues)
          echo "/issues";;
        *)
          echo ;;
      esac;;
    *bitbucket*)
      case $endpoint in
        pr)
          echo "/pull-requests";;
        issues)
          # issues aren't enabled by default for bb repos
          # so this might not always work
          echo "/issues";;
        *)
          echo ;;
      esac;;
  esac
}

open_site() {
  ext=$(get_ext $1)
  url=""
  if [[ $remote == *@* ]]; then
    # extract path from ssh
    name=$(echo "$remote" | sed  "s/^.*:\(.*\)\.git/\1/g")
    url="$1/${name}${ext}"
  else
    # extract uri from http
    name=$(echo "$remote" | sed "s/^.*: \(.*\).git/\1/g")
    url="${name}${ext}"
  fi
  open "$url"
}

# get remote from origin
remote=$(git remote show -n origin | grep Fetch.URL)

case $remote in
  *github*)
    open_site "https://github.com";;
  *bitbucket*)
    open_site "https://bitbucket.org";;
esac

You’ll want to alias that script in your ~/.bashrc or ~/.zshrc like this:

alias github="~/scripts/remote"
alias bitbucket="~/scripts/remote"
alias remote="~/scripts/remote"

For whatever reason my brain thinks everything is on github so I generally just use the first alias from the command line. Go ahead and re-source your shell via . ~/bashrc (or zshrc depending on what you use).

Go to a directory with a git repository:

$ github
$ github pr
$ github issues

There you have it! Right now the script assumes the origin remote has the repo information (ie whether it lives on github or bitbucket). That is 99% true for my projects but it might not be for yours.

Why your Vim hotkeys might be slow...

I have been using ctrlp with ag as my file fuzzy finder for well over 2 years now. It has been fantastic - it’s accurate, easy to navigate and mostly importantly fast.

Whenever I open vim, I instantly hit <leader> f to bring up a file list. Here’s a quick screenshot:

Using the fuzzy finder:

Great - that is a joy to code with. If you see my post below on my vim shortcut to replace function’s with ES6 fat arrows, I ran into something with my vim setup that was unsuspected.

Prior to that post, I had my fuzzy finder mapped to <leader>f, as shown here in my ~/.vimrc:

" set leader key to comma
let mapleader = ","

" silver searcher config
let g:ackprg = 'ag --nogroup --nocolor --column'

" ctrlp config
let g:ctrlp_map = '<leader>f'
let g:ctrlp_max_height = 30
let g:ctrlp_working_path_mode = 0
let g:ctrlp_match_window_reversed = 0
let g:ctrlp_show_hidden = 1

Then, as my post below mentions I added a new mapping <leader>fa (fa meaning ‘fat arrow’). In my .vimrc that looked like:

" replace function(*args) with fat arrows =>
map <leader>fa :%s/(\(.*\)function\(.*)\)/(\1\2 =>/g<cr>

I’ve been using both of those hotkeys just fine over the last few days. However, one thing I did notice is that my vim began to lag when I wanted to search files. I’d hit ,f and it would hang for a second then display the file list. I’m so used to that being very fast that I was miss hitting files or worse leaving the prompt and entering text.

After some googling around I learned about a vim setting that I’ve never had to tweak before called timeout.

In vim, go to :h timeout. The first sentence describes exactly what I was running into:

[timeout and ttimeout] - These two options together determine the behavior when part of a mapped key sequence or keyboard code has been received

Most importantly it states in the next paragraph that vim waits 1 second for another character to arrive before it will default to the lesser mapping. So in my case I never had another hotkey mapped to f before but now that I have f and fa vim is waiting to see that once I hit f will I also hit a?

You’re probably wondering if there is a way to turn off the timeout and sure enough there is:

:set notimeout
:set nottimeout

I suggest going through :map and seeing if you could benefit from adjusting your timeout based on your own key mappings. Now you need to figure out what you’re going to do with all those free seconds I just gave you back - hopefully you’re in the giving spirit as we are just a few weeks away from code.org’s hour of code. I volunteered last year and it was a blast, I hope you do the same!

Search and replace for ES6

I’m here to share the simplest of tips for vim users that are transitioning to ES6 fat arrows.

It’s really common to still see examples on npm packages like this:

sftp.open(config.ftpPath + "/" + m_fileName, "r", function(err, fd) {
  sftp.fstat(fd, function(err, stats) {
    var totalBytesRead = 0;
    function callbackFunc(err, bytesRead, buf, pos) {
      if(err) {
        writeToErrorLog("downloadFile(): Error retrieving the file.");
        errorOccured = true;
        sftp.close(fd);
      }
    }
  })
});

I prefer to not have a linter fix my code for me, however, there are somethings I’m starting to automate. The first is the use of fat arrows over function(params) syntax.

I made a quick vim alias to search and replace the use of function(*args) to (*args) =>. Prepare for an anticlimactic code snippet…

:%s/(\(.*\)function\(.*)\)/(\1\2 =>/gc<cr>

Try that with the interactive prompt and see if it looks ok. The breakdown of the regex looks complicated but it’s not. It could likely be improved but so far it’s working for me.

(\(.*\)function - look for the word function within a set of parens.  That way function `myFunc(*args)` does not get replaced but `something.myMethod('arg', function(*args) {` does.  Create a match group for the args before the word function
\(.*)\) - create a match group for the arguments within `function(*args)` so we don't lose the arguments
/(\1\2 =>/ - replace with the first group and second group followed by our fat arrow
g<cr> - do this globally and leave the vim command prompt

Assuming that worked for you, let’s add it to our ~/.vimrc:

" replace function(*args) with fat arrows =>
map <leader>fa :%s/(\(.*\)function\(.*)\)/(\1\2 =>/g<cr>

Resource your vimrc in vim (either restart vim or :source ~/.vimrc). Now on your old ES5 code hit your leader key and then fa. My leader key is the comma so that means: ,fa.

The code above should turn into:

sftp.open(config.ftpPath + "/" + m_fileName, "r", (err, fd) => {
  sftp.fstat(fd, (err, stats) => {
    var totalBytesRead = 0;
    function callbackFunc(err, bytesRead, buf, pos) {
      if(err) {
        writeToErrorLog("downloadFile(): Error retrieving the file.");
        errorOccured = true;
        sftp.close(fd);
      }
    }
  })
});

If that didn’t work for you let me know and I’ll try to improve the regex. Enjoy!