How much flakiness do you tolerate in end to end tests? (programming.dev)

submitted 1 year ago* (last edited 1 year ago) by kersplort@programming.dev to c/experienced_devs@programming.dev

22 comments fedilink hide all child comments

End to end and smoke tests give a really valuable angle on what the app is doing and can warn you about failures before they happen. However, because they're working with a live app and a live database over a live network, they can introduce a lot of flakiness. Beyond just changes to the app, different data in the environment or other issues can cause a smoke test failure.

How do you handle the inherent flakiness of testing against a live app?

When do you run smokes? On every phoenix branch? Pre-prod? Prod only?

Who fixes the issues that the smokes find?

you are viewing a single comment's thread
view the rest of the comments

[-] agressivelyPassive@feddit.de 4 points 1 year ago

Shouldn't you find out, why the smoke tests keep failing? It's not a good sign, if you can't even guarantee stability in a controlled environment.

[-] kersplort@programming.dev 2 points 1 year ago

It's not a fully controlled environment, that is the point of smokes.

[-] agressivelyPassive@feddit.de 1 points 1 year ago

How do you not fully control the environment your PRs are tested in?

[-] kersplort@programming.dev 1 points 1 year ago

dependencies.

this post was submitted on 30 Aug 2023

34 points (100.0% liked)

Experienced Devs

3870 readers

2 users here now

A community for discussion amongst professional software developers.

Posts should be relevant to those well into their careers.

For those looking to break into the industry, are hustling for their first job, or have just started their career and are looking for advice, check out:

Logo base by Delapouite under CC BY 3.0 with modifications to add a gradient

founded 1 year ago

MODERATORS

snowe@programming.dev

jmk1ng@programming.dev

drewsiferr@programming.dev