Monday, 29 September 2014

Definitive Guide to Recovering from a Full Disk

Cheap, stingy guy that I am, I allocate really small system partitions to my Ubuntu servers. This means that periodically my disk fills up. It fills up because every kernel upgrade takes a fair amount of space, and old kernels aren’t cleaned out automatically. Unfortunately, the disk usually fills up when trying to do an upgrade, so apt-get fails, and terminates with a partially installed package. You’ll know that has happened when you get a message like this when you run an apt command:
E: Unmet dependencies. Try using -f.
Once that happens, you can’t use any apt command.
There’s lots of advice out there about what to do, but the pages I’ve found always seem to leave something out, or assume knowledge of apt or dpkg that I don’t have.
So based on the last time this happened, here’s how I plan to recover the next time I run out of space. Warning: lots of Terminal commands coming up. I do everything in the Terminal for a few reasons:
  • The happens to me most often with servers, as I’m trying to save space, especially for virtual machines. My servers don’t have a GUI
  • Terminal works for both desktop and server machines
  • It’s easier to document commands for the Terminal
First, I have to make sure the problem really is that I’m out of space. (Looking for 0 in the “Avail” column, on the line that has “/” under the “Mounted on” column):
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       3.7G  3.7G     0 100% /
Then I find out what version of the kernel I’m running:
$ uname -a
Linux ixmucane 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Next, I find out what kernels are installed:
$ dpkg --list | grep linux-image
If I have at least two versions older than the one I’m currently running, I can remove the oldest one (replacing the “n.nn.n-nn” with the version number I want to remove):
$  sudo dpkg --purge linux-headers-n.nn.n-nn-generic 
$  sudo dpkg --purge linux-headers-n.nn.n-nn
$  sudo dpkg --purge linux-image-extra-n.nn.n-nn-generic 
$  sudo dpkg --purge linux-image-n.nn.n-nn-generic 
This should free up lots of space, but I check again with df -h. Then run:
$ sudo apt-get -f install
If the amount of space it needs is less than what’s available according to df -h, then I go ahead and finish the install. To be safe, I also do:
$ sudo apt-get update
If there’s not enough space, and I have more old versions of the kernel installed, I just repeat the above dpkg until I have enough space to finish the install.
The above is the happy path. If I didn’t have two versions older than the current running kernel, I would try to remove the partially installed packages. Looking again at the output of:
$ dpkg --list | grep linux-image
if the newest version there is newer than the kernel currently running, then I would try the above dpkg commands to remove the partially installed packages. Some of them won’t work, of course, since the package isn’t installed, But once all the installed packages are removed, presumably there would more space and I could try:
$ sudo apt-get -f install
The reason I want to have two versions older than the current version is in case for some reason the current kernel doesn’t work, I can go back to the previous version. This is a cautious approach. If I’m really stuck, I would remove all versions except the current version. I’d probably make sure that I could boot the current kernel first. I haven’t had to do this and I hope I never do, but…

Monday, 26 May 2014

IT Lottery

So today I had to participate in one of the many little rituals of an enterprise IT project manager: Get someone signed up to my project time charge code.

I submitted the usual paperwork like I've done a number of times before, but this time I was told that the role I selected wasn't a role in the timesheet software, and neither was the individual's official job title. The individual is an employee of the client's company.

I felt somewhat smug that I was able to completely ignore the absurdity that an employee's job title isn't acceptable to the company's timesheet software. But then I realized what I was being asked to do: I had been told two titles that weren't acceptable to the timesheet system, but I had not been told what would be acceptable. I guess I'm supposed to randomly guess until I get the right answer.

Definitely a Dilbert moment. But then I asked myself, "Why would someone respond to me this way?" The person I was dealing with is a very nice, dedicated worker. They weren't just trying to make my life difficult.

I think it's because, in the enterprise IT world, there's no upside to providing service. An IT manager has too many demands, and not enough people to meet the demands. In addition, the path to promotion for the manager is through more responsibility, and the way to get that is by having more staff and a bigger budget. If you provide good service for the same personnel level, you're not meeting your boss's needs.

Also, there's no upside to taking responsibility. If you take responsibility, you can be blamed if, sometimes, you don't achieve the desired result. Better to leave all decisions up to someone else, and don't give them any help, in case they blame you if your help turns out not to be helpful.

This culture so permeates our business that it's not absurd to just tell someone, "You got it wrong," without giving so much as a hint as to what the right answer might be.

No wonder people have such low expectations of IT.

Monday, 12 May 2014

Gender Imbalance in IT -- It Wasn't Always Like This

I was in a meeting a couple of weeks ago at a client's office. This client still uses an IBM mainframe, and the meeting was about some mainframe activities for my project. I realized that four of the seven people in the room were women.

I think the whole mainframe department has slightly more men than women, but it's much closer to gender parity than most IT gatherings.

Then I thought back to my graduating class in Computer Science at the University of Saskatchewan in 1980. We were also close to gender parity.

This certainly shows that there's nothing preventing women from getting into and being successful in IT. What it shows is that, during decades when women were increasing in numbers in most professions, things were actually going in the other direction in IT -- women were being driven out of the field.

Sunday, 20 April 2014

Mounting Windows Shares with Nautilus

From time to time I need to access a Windows share on a domain (not is a local work group) via Nautilus (the LinuxMint file browser I use). It always takes me too long to figure out the trick...

The trick: The domain has to be specified in upper case (or perhaps it's simply case sensitive and it depends how the domain administrator has specified the domain). So in Nautilus, I do File-> Connect to Server..., then fill in the appropriate values in the dialogue that appears, with domain in upper case.

To debug issues when connecting to Windows shares, I open a Terminal window and type:

smbclient -U domain\\windows_user_name //windows_server_name/share

and follow the prompts.

If that works, I try:

gvfs-mount "smb://domain;windows_user_name@windows_server_name/share"

and follow the prompts. (The quotes are required because the semi-colon is a special character to the Terminal program.) I have seen cases where the mount takes a minute or so to connect, so I have to be patient.

If the above works, I unmount the drive with:

gvfs-mount -u "smb://domain;windows_user_name@windows_server_name/share"

Thursday, 27 February 2014

Relocating Another Data Centre

I recently took part in another data centre relocation project. I was one of a number of project managers moving some of the servers in a 1,300 server data centre. I moved about 200, and decommissioned another 50. I was directly planning and executing moves, so my role was different from on my previous project. It was good to experience a move from another position.

The project was successful in the end. I have to say that there were a number of lessons learned, which goes to prove that no many how many times you do something, there's always something more to learn.

Unlike my previous experiences, there were three major organizations working together on this relocation: the customer and two IT service providers to the customer. All organizations had good, dedicated, capable people, but we all had, at a minimum, a couple of reporting paths. That in itself was enough to add complication and effort to the project.

The senior project manager identified this right from the start and he made lots of good tries to compensate and mitigate for it. We did a number of sessions to get everyone on the same page with respect to methodology. Our core team acted as a cohesive team and we all adhered to the methodology. And in fact, across the project I think it's safe to say that the front line people did as much as they could to push toward to project goals.

Despite our best efforts, we all, across the three organizations, had to devote significant efforts to satisfy our own organization's needs. It's worth noting that much of this is simply necessary -- organizational governance is a big issue in the modern economy, and appearing to have management control is an business reality.

So if you're planning a relocation, take a look at the organizational structures that will be involved, and take them into account when planning your data centre relocation project.

Friday, 24 January 2014

Time Zone in Rails

There’s pretty good info out there about using time zones in Rails, and Rails itself does a lot of the heavy lifting. The Railscast pretty much covers it. It’s only missing a discussion of using Javascript to figure out the client browser’s time zone.

Time Zone from the Browser

To get the time zone from the browser, use the detect_timezone_rails gem. The instructions give you what you need to know to set up a form with an input field that will return the time zone that the browser figured out. That would work perfectly if you were implementing a traditional web site sign-up/sign-in form.
However, I needed to do something different. Since I’m using third party identity providers (Google, Twitter, Facebook, etc.) via the excellent Omniauth gems, I needed to be able to put the time zone as a parameter on the URL of the identity provider’s authorization request. Omniauth arranges for that parameter to come back from the identity provider, so it’s available to my app’s controller when I set up the session.
To add the parameter, I added this jQuery script to the head of the welcome page:
<script type="text/javascript">
          .each(function() {
            this.href = this.href + "?time_zone=" +
This added the time zone, appropriately escaped, to the URL for the identity provider (the href of the <a> elements). This worked because I had set each of the links to the identity providers to have class="time_zone", like this:
<div class="idp">
  <%= link_to image_tag("sign-in-with-twitter-link.png", alt: "Twitter"), 
    class: "time_zone" %></div>
In the controller, I did this (along with all the other logging in stuff):
if env["omniauth.params"] &&
  tz = Rack::Utils.unescape(env["omniauth.params"]["time_zone"])
  if user.time_zone.blank? 
    user.time_zone = tz!
    flash.notice = "Your time zone has been set to #{user.time_zone}." +
      " If this is wrong," +
      " please click #{view_context.link_to('here', edit_user_path(user))}" +
      " to change your profile."
  elsif user.time_zone != tz
    flash.notice = "It appears you are now in the #{tz} time zone. " +
      "Please click #{view_context.link_to(edit_user_path(user), 'here')}" +
      " if you want to change your time zone."
  logger.error("#{} (id: #{}) logged in with no time zone from browser.")
Of course, you may want to do something different in your controller.

Testing Time Zones

However you get your time zones, you need to be testing your app to see how it works with different time zones. YAML, at least for a Rails fixture, interprets something that looks like a date or time as UTC. So by default, that’s what you’re testing with. But that might not be the best thing.
I had read that a good trick for testing is to pick a time zone that isn’t the one your computer is in. Finding such a time zone might be hard if you have contributors around the world. I like the Samoa time zone for testing: Far away from UTC, not too many people living in the time zone, and it has DST.
If you want a particular time zone in your fixtures, you have to use ERB. For example, in my fixtures I might put this:
created_at: <%= Time.find_zone('Samoa').parse('2014-01-30T12:59:43.1') %>
And in the test files, something like this:
test "routines layout" do = 'Samoa'
  correct_hash = {
    routines(:routine_index_one)=> {, 01, 30)=> [


I found a few gotchas that I hadn’t seen mentioned elsewhere:
  • Rails applies the time zone magic when it queries the database, so if you change your time zone after you retrieve the data, then you have to force a requery, or the cached times will still be in the model. Shouldn’t be a problem when running tests, but is when using the console to figure things out
  • You can’t use database functions to turn times into dates, as these won’t use the time zone. No group by to_date(...) or anything like that

Tuesday, 7 January 2014

Self-Referential, Polymorphic, STI, Decorated, Many-to-Many Relationship in Rails 4


I wanted to model connections à la connections in LinkedIn or Facebook in a Rails application. This means a many-to-many association between instances of the same class. That caused me some grief trying to get it hooked up right because you can’t rely on Rails to figure everything out.
The other trick in my application is that the people involved in the connections might be users who have registered to use the application, or they might be people created by a registered user, but who aren’t registered to user the application.
Concretely, and hopefully more clearly, I have “users”, who have registered, and I have people who can be involved connections. In my app the people who aren’t registered users are “patients”.
In the course of trying to get this all to work I stumbled across three approaches to this type of problem:
  1. Polymorphic classes
  2. Single Table Inheritance (STI)
  3. Decorator pattern
The combination of the many-to-many combined with the two classes took a lot of work to get straight. The Rails Guides were a great starting point, but I find that specifying Rails associations can be tricky if it’s not completely straightforward, and especially when you start chaining them together.
In the end, I decided to go with the Decorator pattern. But I’ll start with the one I threw out first: Polymorphic.


I got pretty far with polymorphic associations, but I couldn’t figure out how I was going to get a list of all people (patients and users) connected to another person. I could either get all patients or all users from the methods that the Rails associations gave me, but not a list of all together.
I realized in writing the preamble above that I probably should have realized that what I was trying to model wasn’t really a polymorphic situation. Polymorphic in the examples I saw was used to connect an object to another object from any one of a number of unrelated classes. Of course, hindsight is 20/20.
This post convinced me that trying to get a list of all people wasn’t going to come naturally from a polymorphic approach, so I stopped pursuing it.

Single Table Inheritance

I got fired up about single table inheritance (STI) as I was reading about how to make the polymorphic approach work. A good brief write up is here: The Railscast is here: (sorry, it’s a pro Railscast so it’s behind a paywall).
Others say I shouldn’t do STI. People say it can cause problems. One problem is if the type of an object will change, and change because of user input, it’s hard to handle. The view and controller are fixed to a certain object, so you can’t change the object type based on user input.
So here’s the code. First, create the models:
rails g model person name:string type:string provider:string uid:string
rails g model link person_a:references person_b:references b_is:string
class Person < ActiveRecord::Base
  has_many :links, foreign_key: "person_a_id"
  has_many :people, :through => :links, :source => :person_b
  scope :patients, -> { where(type: "Patient") }
  scope :users, -> { where(type: "User") }
user.rb (obviously there will be functionality here, but this is what I needed to get the associations to work):
class User < Person
patient.rb (as with user.rb, functionality will come later):
class Patient < Person
class Link < ActiveRecord::Base
  belongs_to :person_a, class_name: "Person"
  belongs_to :person_b, class_name: "Person"
It was a little hard to get the associations to work. The key to making the has_many :links,... in person.rb work was the , class_name: "Person" on the association in link.rb.
With the above, I can do things like:
person = Person.find(1).first
That’s all pretty sweet, and I really considered using this approach. In fact, I may return to it. There’s a lot left to do with my application. However, I’m pretty sure that I will need to deal with cases like a registered user corresponding to multiple patients (e.g. people get created under different names). Eventually I need a way to consolidate them.


In the end, perhaps the simplest was the best. I just decorated a person with an instance of a user when the person is a registered user. (This allows multiple people for a user, which might be useful for consolidating duplicate people.)
Here’s what I did:
Generate the models:
rails g model link person_a:references person_b:references b_is:string
rails g model person user:references name:string
rails g model user uid:string name:string provider:string
require 'person_helper'

class Person < ActiveRecord::Base
  belongs_to :user
  has_many :links, foreign_key: :person_a_id
  has_many :people, through: :links, source: :person_b

  include PersonHelper
I thought the person model should have has_one instead of belongs_to, but that would put the foreign key in the wrong model.
require 'person_helper'

class User < ActiveRecord::Base
  has_many :identities, class_name: "Person"
  has_many :links, through: :identities
  has_many :people, through: :links, :source => :person_b

  include PersonHelper
module PersonHelper
  def users { |person| ! person.user_id.nil? }

  def patients { |person| person.user_id.nil? }
class Link < ActiveRecord::Base
  belongs_to :person_a, class_name: "Person"
  belongs_to :person_b, class_name: "Person"
With the above, I can do things like:
person = Person.find(1).first
user = User.find(2).first
Again, sweet. Same number of files at the STI version. Instead of subclassing, common functionality is handled by a mixin module.


Another thing people don’t seem to like about STI is that it’s easy to end up with a big table full of all sorts of columns used only in a few places. Most modern database management systems aren’t going to waste a significant amount of space for unused columns, so I’m not sure what the problem is.
However, it got me thinking if there isn’t a way in Rails to have more than one table under a model. Or more to the point, could you have a table for the base model class, and a different table for each of the subclasses, and have Rails manage all the saving a retrieving.
I’m sure I’m not the first person to think of this. But I’m not going to go looking for it right now.

Other Resources

Rails 4 guides on associations: and migrations:
Ryan Bates’ Railscast on self-referential associations:, and on polymorphic associations: