Flaky RSpec Test Investigation
Last time I was investigating how to find out which test or tests are dependent, making another test fail by just running before it. Boiled down to a small group of tests I am now digging into: why this happens.
I have found a command to run a small set of tests which will let my affected test fail:
$ bin/rspec --format documentation --seed 58021 spec/factories_spec.rb spec/decorators/participation_decorator_spec.rb
Step 1: Reducing test scope
The factories_spec.rb
is a spec that validates or factories created. I have a couple of factories listed and the given seed tests 2 of them before ParticipationDecorator
is being tested and several others afterwards. With a simple if
I excluded all but one, the evil one letting the test fail when it runs first, from being run at all. By that I have a neat test result with 2 simple specs, one guaranteed to run first while the second runs last and fails because of the first.
If I would have more specs inside participation_decorator_spec.rb
I would also exclude everything except of the affected test – but currently I don’t.
Step 2: Realize The Actual Behavior
Or in other words: what exactly fails in the affected test?
That’s important to know when investigating because you have to know in which direction to look at.
Failures:
1) ParticipationDecorator#winner_icon when participant wins returns the trophy icon
Failure/Error: h.fa_icon("trophy") if object.winner
NoMethodError:
undefined method fa_icon for #<#<Class:0x007fcb13a4e670>:0x007fcb1385b390>
and the code
def winner_icon
h.fa_icon("trophy") if object.winner
end
As I mentioned before there’s another spec failing together with this one. The problem there was that my h.link_to
code inside the decorator suddenly includes the http://localhost:3000/ in local links which it hasn’t done before. Since I was checking the links, the test started to fail.
Let’s see how we can join these two problems.
Step 3: Investigate The Evil Spec
The first thing is quite easy: since it’s a factory spec it builds a factory object and validates it. I already reduced it to one special factory being tested, so I’m checking its attributes.
FactoryGirl.define do
factory :post do
user
title { generate :title }
teaser { generate :text }
body { generate :text }
...
end
end
It turns out that the :post
factory has a user
linked and commenting it will let my second test pass, so it’s obviously something to do with the :user
factory or model.
When I started commenting out attributes of the :user
factory to find out which one may be the problem I realized that every time the validation spec failed (because I removed a mandatory attribute), the affected test passed.
First I though I found the evil attribute, but unluckily this wasn’t the case. So it either has something to do with a validation that comes after those of the attributes I removed or with some after hook.
One interesting aspect with this is, that even when calling FactoryGirl.build(:post)
, the user
reference will be create
d! That’s why I didn’t focus on the hooks at first.
For the test I could simplify my test scenario by
* adding a let(:user) { create(:user) }
on top of the affected test
* run only this spec without dependency to some other
Having said that, I can let my tests pass by “simply” changing the factory to
FactoryGirl.define do
factory :post do
user { build(:user) }
...
before(:create) do |post, evaluator|
post.user.save
end
end
end
This ensures that a build
is done on the referenced user
when the post
is built
, and a create
is performed with the user
(by saving it) when the post
is created. Any other ideas? I couldn’t find anything in Factory Girl.
However: this eliminates the symtoms for now, but it doesn’t get rid of the actual problem – and will come back as soon as a user
object is needed to be created
in combination with a decorator spec.
Step 4: Find The Root Cause
Starting with my reduced test set up, i. e. adding create(:user)
to the decorator spec, I continue the path.
To cut the long story short here – after playing around with my user model line by line:
My user model uses devise
with a lot of additions and removing :confirmable
will let all specs pass. Adding it back again makes the tests with special helper functionality fail.
A quick look into devise reveals that it may send an email and a quick search on Google now brings this to light: Decorator is unable to access helper methods after sending an ActionMailer email..
The quick workaround to this is to add
FactoryFirl.define do
factory :user do
...
confirmed_at { DateTime.current }
end
end
and hope that they’ll fix it soon – who knows what other problems exist.
No Comments