Sunday, 23 August 2015

Reading USB controllers in Ruby (or What to do when you don't know what to do)

Disclaimer: I'm writing a blog on this subject because I couldn't find a more useful tutorial online. The truth is that I have twice worked out how to read a USB device in ruby, but I do not have a good understanding of the USB standard. I am sharing both how I reached a working code-base and the code I wrote, but there will be people out there who understand this better. If you're one of them, please write a simpler tutorial so this one won't be needed any more.

Our story starts at the Brighton Ruby conference in 2015, where I presented a technology-themed version of the popular panel game, Just A Minute. I had put together a simple system in Pure Data to keep track of the scores, topics and the timer. Long story short, this system let me down, it crashed half way through the session and I had to re-construct the scores on the fly.

In retrospect, I found that Pure Data was not the right tool for the job, so I set about rebuilding the same system in Ruby, with the Gosu library.

One of the reasons I had chosen Pd in the first place was the simplicity of using USB HID devices (you can learn about that in my earlier tutorial). So half way through this process, I ran up against the fact that is is not quite so simple in Ruby.

Firstly, I started with libusb, a standard library for reading USB.

I'm using the controllers from the trivia game Buzz, which look like they should be a particularly simple USB device, no output, no continuous controllers, just 20 buttons, should be simple, right?

First order of business was to find out what the USB device was. There's none of the PD index-based HID identifiers, instead I had to use the linux command 'lsusb' to find them. And the output from this includes my laptops keyboard and mouse, as well as what appears to be some internal USB hubs. It looks a little like this:

ajfaraday@ajf-samsung:~$ lsusb
Bus 002 Device 005: ID 0cf3:3004 Atheros Communications, Inc. 
Bus 002 Device 003: ID 054c:0002 Sony Corp. Standard HUB
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 2232:1029  
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

To this day, I have no idea what bus 1, device 3 is.

Okay, so we can't see the word Buzz, or playstation, or HID. The only clue is that it's made by Sony. The only way I can see to confirm this is to un-plug the USB and re-run lsusb. Sure enough, the Sony Corp. Standard HUB does not appear now. So I know this is the device I'm looking for.

Bus 002 Device 006: ID 054c:0002 Sony Corp. Standard HUB
It looks like this isn't enough information to actually use it, tho. I'm going to need more. The help page for lsusb tells us we can use the -s flag to choose a specific device, and the -v device to get verbose information.

ajfaraday@ajf-samsung:~$ lsusb -h
Usage: lsusb [options]...
List USB devices
  -v, --verbose
      Increase verbosity (show descriptors)
  -s [[bus]:][devnum]
      Show only devices with specified device and/or
      bus numbers (in decimal)
So we need to get the bus, and devnum (device) from our previous lsusb command. Here's the full information from my Buzz controllers.
ajfaraday@ajf-samsung:~$ lsusb -s 002:006 -v

Bus 002 Device 006: ID 054c:0002 Sony Corp. Standard HUB
Couldn't open device, some information will be missing
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 (Defined at Interface level)
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0         8
  idVendor           0x054c Sony Corp.
  idProduct          0x0002 Standard HUB
  bcdDevice           11.01
  iManufacturer           3 
  iProduct                1 
  iSerial                 0 
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength           34
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              100mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           1
      bInterfaceClass         3 Human Interface Device
      bInterfaceSubClass      0 No Subclass
      bInterfaceProtocol      0 None
      iInterface              0 
        HID Device Descriptor:
          bLength                 9
          bDescriptorType        33
          bcdHID               1.11
          bCountryCode           33 US
          bNumDescriptors         1
          bDescriptorType        34 Report
          wDescriptorLength      78
         Report Descriptors: 
           ** UNAVAILABLE **
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            3
          Transfer Type            Interrupt
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0008  1x 8 bytes
        bInterval              10

Okay, there's a load of extra information here that we don't need. Just to cut a bit of this out, we're going to need idVendor, idProduct and the Endpoint Descriptor.

Back to libusb, the README for the library quickly gives me a bit of example code, which I've changed to include the information for my Buzz controllers.
require "libusb"

usb = LIBUSB::Context.new
device = usb.devices(:idVendor => 0x054c, :idProduct => 0x0002).first
device.open_interface(0) do |handle|
  handle.control_transfer(:bmRequestType => 0x40, :bRequest => 0xa0, :wValue => 0xe600, :wIndex => 0x0000, :dataOut => 1.chr)
end
But as soon as I try running this code I see this error.
/var/lib/gems/1.9.1/gems/libusb-0.5.0/lib/libusb/constants.rb:62:in `raise_error': 
LIBUSB::ERROR_BUSY in libusb_claim_interface (LIBUSB::ERROR_BUSY)
The stack trace points us at line 5, 'device.open_interface'. I did a lot of googling on this subject, and got a lot of confusing answers. I'm skipping a lot of trial and error here, but the answer here seems to be that something else, possibly the operating system, is reading the USB port already. So to use it with my Ruby app, I need to use detach_kernel_driver. Which looks a little like this:
  def reset_device_access
    usb_context = LIBUSB::Context.new
    device = usb_context.devices(
      idVendor: 0x054c, idProduct: 0x0002
    ).first
    handle = device.open
    handle.detach_kernel_driver(0)
    handle.close
  rescue => er
    puts er.message
    # nothing needs doing here
  end

There's a couple of things here, firstly, I'm finding the usb context, then finding the device by as in the example. I then need to open a device handle, run detach_kernel_driver, then close the handle. Only, if there is no kernel driver to detach, this throws an error. The rescue block here is something of a hack, I can't find out how to detect if I need to run detach_kernel_driver or not, so I simply catch the error, but don't halt the code. If it throws an error, then it didn't need to be done. This shows my ignorance of the deeper, darker parts of libusb, but it works.

I don't like this, but some times this is something we need to do, pending a better understanding of what we're working with.

So, I can actually get to my device, that's a start. Although, I want to use my device in and amongst some other code, I didn't want it to be limited to a single code block. I had to go digging around in the libusb API documentation and found that instead of using 'open_interface', which opens the interface, uses it within a block, and then closes it at the end of the block, I could instead use the 'open' method. My new code looks a bit like this:

class Controller
  def initialize
    @usb_context = LIBUSB::Context.new
    @device = @usb_context.devices(
      idVendor: 0x054c, idProduct: 0x0002
    ).first
    reset_device_access
    @handle = @device.open
  end
end

So, I've wrapped it in a class, run my reset method to ensure it's free, and saved the device handle as an instance variable, @handle. The next thing we need is a method to read that handle. For this we need the endpoint data from back in our lsusb data, specifically the attributes named bEndpointAddress, bLength and bInterval. Again, after a lot of trial and error, I found it easier to use interrupt_transfer, which is specific to input endpoints (the information coming back from a USB device). So this is also in the Controller class

  def raw_data
    data = @handle.interrupt_transfer(
      :endpoint => 0x81,
      :dataIn => 0x0005,
      :timeout => 10
    )
    data.bytes
  end
Again, I had a great deal of difficulty reading the result of interrupt_transfer, and the code shows where I wound up after it. The data variable is a string, but it's not human readable, the result looks a little bit like this.


 

 

 
0
 

 


Each line of this is a single output from the raw_data method, while I'm pushing some buttons on the Buzz controllers. There's no understanding this. A lot of trial and error later, I discovered that this is actually an array of numbers, each one is a byte (a number made up of 8 binary bits, which translates to a number between 0 and 255. So, with the method defined above, I wrote a script to watch what happens when I push the buttons...
require 'libusb'
require './controller.rb'

c = Controller.new

loop do
  begin 
    puts c.raw_data.inspect
  rescue
  end
  sleep 0.01 
end  

Oops, there's another one of those unhandled rescue blocks. This is bad practise! Usually you would either stop the program completely on an error, and guard earlier on in the code against situations which will cause an error to be thrown. Only I can't find out if it'll work or not without just trying it. So this will have to do for now.

What I've found by probing in this way, is that when a button is pushed between one call of the raw_data method and the next, there is no error, but when no buttons are pushed, it throws the error 'error TRANSFER_TIMED_OUT'. So I just ignore the timeouts, and use the data from when there is a change. So I can now see the output from the raw data method, only when buttons are pushed. Here's what happens when I push a button at random:
[127, 127, 0, 0, 240]
[127, 127, 0, 4, 240]
[127, 127, 0, 0, 240]
[127, 127, 0, 4, 240]
[127, 127, 0, 0, 240]
Okay, so the first thing I've noticed is that the first two numbers (bytes 0 and 1) don't seem to change at all. I have no idea why, but I can easily isolate the input I was looking for. The fourth item in the list (byte 2) goes up by 4 when it is pushed.

Now, 4 is a binary number, and a quick check of the other buttons proved a definite pattern. Pushing any button on the controllers will increment byte 2 or byte 3 by a binary number (1, 2, 4, 8, 16, 32, 64 or 128). The long and the short of it is this, as I mentioned before, these numbers are made up of 8 binary bits, each representing one of the numbers listed above. The number each active bit represents is added up to make that number.

If I select the index of a number in Ruby, it gives me that indexed bit of the number. For instance:
# bit 0 represents 1, so for number 1, this is active.
1[0] 
# => 1
# but if the number will not contain a 1 in it's binary makeup, this is 0.
2[0]
# => 0
# This means that if we add up some binary numbers, the bits for these numbers is a 1:
n = 4 + 32
n[2] # the bit for 4
# => 1
n[5] the bit for 32
# => 1

So, applying this to the data to
c = Controller.new

loop do
  begin 
    puts c.raw_data[3][0] == 1
  rescue
  end
  sleep 0.01 
end  
So, raw_data[3] is byte 3 of our raw data and raw_data[3][0] is bit 0 of byte 3. If this bit is a 1 instead of a 0, that button is pushed. By pushing each of the buzzer buttons (the ones I'm interested in), I found this information out.

# buzzer | byte  | value | bit
# 1      |  2    |  1    |  0
# 2      |  2    |  32   |  5
# 3      |  3    |  4    |  2
# 4      |  3    |  128  |  7
That is, for buzzer 1 we see byte 2 increment by 1, which is bit 0 (the one on the far right) etc. So to check all 4 buttons, I narrowed this down to an array with which byte and bit to look for for each button. Which I've stored as a constant on my Controller class.

class Controller

  BUZZ_BITS = [
    [2, 0],
    [2, 5],
    [3, 2],
    [3, 7]
  ]

  def check
    data = raw_data
    BUZZ_BITS.each_with_index do |lookup, i|
      if data[lookup[0]][lookup[1]] == 1
        puts "buzzer #{i + 1} pushed"
      end
    end
  rescue
    # no input, just ignore the error
  end
  ...

This is really the end-point for this tutorial. What I've done in the end is store which bit I'm looking for for each button and at each call of the check method, I grab the raw data. Then I iterate through the bits I'm looking for, and print the index of that bit (with a + 1 so people aren't confused by the zero index).

Okay, so this is pretty confusing, but it works, honest. You can see the full example code at www.github.com/ajfaraday/usb-controller-demo and see it in the wild at www.github.com/ajfaraday/ruby-jam.

Thanks for bearing with me on this one, I really hope you found it helpful. Any comments, questions? Feel free to get me on twitter at @MarmiteJunction

Monday, 16 March 2015

An open letter to recruitment consultants, on their relationship with developers.

To: The incumbents of the IT recruitment industry

Recruitment consultants are a reality, it can be difficult for companies to find the right candidate, and for developers, particularly those who decide to work a series of short term contracts to find their next position. In this climate recruiters are regularly in contact with tech companies and programmers both searching for jobs and employees and attempting to find matches for them.

But it's not always a match made in heaven, often developers are not looking for work, and many happily hold a single job for a number of years. However my skills as a Ruby on Rails developer are currently in a lot of demand, so sometimes as many as five different agents, often more than one from the same agent, will get in contact and take up some of my valuable time attempting to tempt me into a new position.

Many people in the computer industry share the experience of being frustrated by over-zealous recruitment agents, often getting in touch during working hours, advertising unrelated jobs and using hyperbole to exaggerate the desirability of positions. In more extreme cases, recruiters can be patronising, unduly persistent or completely dishonest to both potential employers and employees.

I appreciate that recruitment is a goal-driven sector, and there is pressure to perform, but making more contact is not, necessarily, the best way to fill jobs, or find them.

-----------------

Allow me to give one example of the mistakes which recruiters make:

My boss works hard, he is a director of the company, and has worked hard to make it the success it is. He's a rails developer, but also handles database and system administration, as well as liaising with customers, regularly working to capacity.

One of my colleagues walks through the door with the phone, someone, giving their name, but not the reason for their call, has asked to speak to the head of Rails development. The boss answers, politely, and quickly becomes annoyed "No, I'm a director of the company, I can't just leave to do another job."

The recruiter has learned the piece of information they phoned to ask about, taking around 30 seconds of the time they are uniquely placed to know the value of. They've also told a director of the company that they want to poach developers from what is a small, tight-knit team, of course he's annoyed.

The correct response would be to give it up as a bad job, politely withdraw and cross his name off the list, permanently.

The next thing I hear is a raised voice "No, we are not currently hiring. Good bye!". It's hard to slam a cordless phone, but my boss had a jolly good try at pushing the red button with attitude.

Having phoned the office phone number in the hope of getting to a senior developer, told the boss that they're attempting to poach developers and persisting in a conversation which is clearly over. This recruiter decided to switch modes from "we need developers" to "we've got developers for you" without missing a beat. Besides a devious method of getting in touch, they're must be an untruth in there somewhere, or at least speculation. They're either lying to the developer in him, or to the director.

This having taken place, is it any surprise the boss didn't want to enter in to a business relationship with the individual on the phone?

----------------

Here's another example, in which a lengthy and complicated bug fix was interrupted with a 30 second phone call which put my own thought processes back by at least half an hour.

The phone rings, undisclosed number, I step out of the room and answer, already expecting recruitment, PPI claims or some form of "get rich quick" scheme.

The voice comes through, "Hello I'm (name) from (company name, it was an acronym, unhelpful)"

I reply, "Sorry, I don't know that company, what do you do?"

"We're a company of IT specialists."

"But what do you do?"

"I'm looking for a developer for a job in..."

Enough of my time has been wasted, I raise the tone of my voice a little "I'm not looking for work at the moment, thanks."

Again, the conversation has clearly ended, but the killer instinct which recruitment agents all seem to develop kicked in, but uncertainly...

"But what if we could..." he pauses, "Offer you more money". There was a noticeable rising, uncertain tone to this last word. As if he'd only just realised that the word money might not be a magic bullet in his fight.

"No thanks", and I make another spirited attempt at slamming my mobile phone.

I couldn't stop thinking about the fact that this total stranger, who wants to start an actual business relationship with me, seriously thought he was going to tempt me away from my current position with a pay cut.

This fact, on it's own, was enough to distract me for some time, but in truth any interruption is disruptive.

----------------------------

I don't just want to shout in the direction of recruiters, but to offer some genuine constructive advice on how to avoid alienating developers who aren't interested.

There are three principles I'd like to suggest you bear in mind when contacting developers.

Concision

Recruiters will often provide a lot of extraneous information about a position, for instance, that it has investment, inflated perks (office furniture and powerful work computers are not perks, they're necessities), or who their clients are. Developers are busy, we don't have a lot of time to read emails in detail, especially when we haven't decided to change jobs at all.

We are not likely to be tempted away by the fact you've taken more time to fill out some of the unimportant details of a job, and to exaggerate the benefits, and quite possibly hide some negaitve points.

Mostly, we'd like to find out who the job is for, so we can do our own research, and bypass all this text.

Okay, so there's a business case for keeping that piece of information secret. Although it does try to start a real business relationship by openly showing a lack of trust in potential clients.

Here's what developers actually want to know:
  • What's the job, don't just say "may be relevant to you", what does the successful applicant have to do? 
  • Where is it?
  • Is it a contract? (this isn't always a good thing), how long will it last?
  • How much is it? We're not completely money obsessed, but it's a good guide for the level a job is at, if we can go for it, and if it's worth the considerable disruption of changing a job. 
How about a bullet point list of this information, instead of paragraphs and paragraphs of nebulous, imprecise information?

Relevance

A javascript developer will rarely be interested in a java position.

A new developer will not be interested in a highly responsible job which doesn't provide some amount of training and employee development. (Senior developers don't happen without being juniors first).

If a CV has not been updated for years, this is probably because it's owner is not looking for work.

Recruiters tend towards a scatter-shot approach to recruitment, the theory being that recruitment emails have a hit rate, so a higher volume of emails lead to the same proportion of a higher number of people. Keywords don't always result in candidates who fit the bill.

I have personally blocked a number of recruiters from contacting my email address when I get more than one email from them a week. I have been known to reply tersely when I get multiple emails concerning the same position from multiple recruiters belonging to the same agency.

Try to gain a little more understanding of the industry you are working in, be a little quicker to take names off the list, or at the very least wait a year before getting back in touch.

Remove hyperbole and patronisation

Developers are intelligent, hard-working people who understand our work affects the public image, productivity and stress levels of our clients. We are professionals, we've worked hard to develop our skills and persist in improving skills to the benefit of our employers.

We do not need to have our egos stroked in order to begin a serious business relationship with an agent. We do not need to be called ninjas, rockstars or jedi to pique our interest. In around half a decade, there will be many programmers who are actually twelve years old, presently, however very few are.

Please don't tell me the job you're representing is an exciting opportunity. Near enough every email from a recruiter starts this way. It's a job, I can decide for myself whether or not it's an exciting prospect for me.

Any other hyperbole says so much more about the author than the subject, facts really are more important.

--------------------

So how would I like to be contacted?

Dear Mr Faraday

I'm currently looking for a Ruby developer for my client, a digital advertising agency based in Rotherham. It's a 6 month, contract, which may become permanent, and a £150 day rate.

Let me know if you're interested and I'll send over a job spec.

Kind Regards

Ms Joan Q Recruiter - Doohickey IT

As a prospect, that's really all I want to know. It's not hard to ask for more, and initiate a dialogue, but it's very off-putting to be fed lots of information outright.

Just let me know who you are and what you're representing to me.

Oh, and you'll rarely get a good response by phoning my mobile during work hours. Interrupting my job to try and take me away from it is not a considerate thing to do. It's like an estate agent saying "It looks like you have a home, would you like a home".

--------------------

In conclusion, think about who you're talking to, and how this unsolicited contact will be received.

Thanks!

Andrew Faraday