Sharing some examples of flaky tests from my experience.

Some of these are trivial, others slightly complex. Examples use Ruby on Rails but the ideas and patterns are language agnostic.

Array ordering

expect(user.locations).to eq([location_1, location_2, location_3])

Records being fetched from database by default do not have any order (unless you have a default scope or some explicit order applied):

irb(main)> User.last.locations

SELECT  "locations".* FROM "locations" WHERE "locations"."user_id" = $1

But eq expects array elements to be in exact same order [location_1, location_2, location_3], which makes this test flaky. If ordering is not expected, use match_array instead (also available with alias contain_exactly):

expect(user.locations).to match_array([location_1, location_2, location_3])

Similarly, sometimes we want to check if a particular element "exists", but we do this:

expect(user.locations.first.name).to eq('Mumbai')

(Assuming locations is not ordered), a better way is to test presence instead:

mumbai = user.locations.find { |location| location.name == 'Mumbai' }
# Or if locations is a relation: user.locations.where(name: 'Mumbai')
expect(mumbai).to be_present

Validation errors

Activerecord runs all validations and collects them in arrays.

expect(user.errors.messages[:name][0]).to eq("must be filled")

Although validations run in the order they are defined, with growing codebase, custom validations and refactorings it is better not to rely on something irrelevant such as order of validations.

A better way to make this test resilient is to check if error exists.

expect(user.errors.messages[:name]).to include("must be filled")

Sidenote: Use shoulda-matchers so you don't have to test trivial validations explicitly.

Switching stuff

Refer my previous blog for understanding Sidekiq's testing modes.

Your tests may be order dependent:

# Fake is sidekiq's default mode
# Change to execute background Job inline for this one test
before { Sidekiq::Testing.inline! }
# Execute code and test

# If any future test (even in another file) expects fake mode, it will fail.

Always make sure to set the mode back or use blocks.

# Default fake mode

Sidekiq::Testing.inline! do
  # Execute code and test
end

# Back to fake mode
# Future tests expecting fake mode not affected

Bonus: If you have multiple tests use an around hook:

around(:each) do |example|
  Sidekiq::Testing.inline! do
    example.run
  end
end

Same applies when switching locale and timezones. Prefer using blocks:

# default locale :en
I18n.with_locale(:hi) do
  I18n.locale #=> :hi
  # Execute code and test
end
# default locale back to :en
# Future tests expecting default locale not affected

# default timezone
Time.use_zone(user.time_zone) do
  Time.zone.name #=> Same as user.time_zone
  # Execute code and test
end
# back to the default timezone
# Future tests expecting default timezone not affected

Implicit cast

Our project uses uuid primary keys on all models. We are so used to seeing uuid everywhere that we forget there is this one legacy User model which still uses integer id as primary key.

If id is an integer and not a uuid, guess what? This following test can randomly fail:

context 'when user does not exist' do
  # Like all other models we set UUID
  before { params[:id] = SecureRandom.uuid }

  it 'returns user not found error' do
    expect { UserService.new(params).call }.to raise_error
  end
end

class UserService
  def initialize(params)
    # But `id` is an integer - so this may actually find a user with uuid too!
    @user = User.find(params[:id])
  end
  # more code...
end

Run the following line multiple times in Rails console and look at the SQL queries.

# users table has integer primary key
User.find_by(id: SecureRandom.uuid)

You can find integer values for id. ActiveRecord will cast uuid to integer while constructing SQL query (because the underlying column is integer) and there could be chances of previously created user's id matching resulting in random failures.

Make sure to use data that matches database types to avoid implicit casting leading to unexpected behaviour:

before { params[:id] = 0 }