TestRail

Regression testing

Comments: TestRail was implemented to do the record taking for manual regression testing and planning.

Pros:

Great product allowing us to keep history of our testing and specifically regression testing of our software development team

Cons:

Find some reporting can be difficult, but in general it is a great tool for managing our regression testing test cases, and ensuring that they have all been run.

Testcase Management

Comments: We enjoy using this tool because it takes little time to train new members and easy of using. SImple and faster.

Pros:

Ease of use. Simple and intuitive flow for the workflows that can be customized based on your work.

Cons:

Traceability is based on how you configure. Little confusing if you try to explain to users not familiar with the tool.

Alternatives Considered: Zephyr

Reasons for Choosing TestRail: Ease of use and less maintenance work. No need to worry about managing the infrastructure and secure.

Switched From: Rally Platform

Used to be great, now it is horrible

Comments: I've been told their team was decimated after it was bought out. They have only 20 people to handle 11,000 customers. A product that is supposed to be for QA, is not QA'd before releases. If you are considering this product my advice is to run in another direction. You will be disappointed if you invest in TestRail.

Pros:

I loved it before it was sold. It was solid, easy to use, and made our work flow simple and efficient. That is no longer the case

Cons:

Since it was bought out, it is now horrific. Test data disappears or is inaccessible. Test runs disappear. EVERY single release is extremely buggy. And knowing that, they still continue to push the release in waves to other customers instead of fixing high priority issues first. Tickets are opened and we get the canned "we're working on it" but rarely are bugs fixed correctly the first time (if at all). We have had months of work disappear with a single release. We've had things break where we were unable to access our tests for days or even weeks at a time. I went from convincing my entire company to join in to use this product, to now only having 3 of us left who will touch it. It's just sad how far it has fallen.

Alternatives Considered: PractiTest and AWS X-Ray

Reasons for Choosing TestRail: We needed something cheaper and easier to use

Reasons for Switching to TestRail: At the time it was the best fit for our needs (now it stinks). It was easy to use and we could purchase only the number of licenses that we needed.

Everything we needed at half the cost

Comments: We are better able to track our software testing and coverage with this tool.

Pros:

I love the way they setup billing, every month you it automatically detects how many active users you have and you pay for that instead of being roped into a set amount of licenses. This has been incredibly helpful as we've gone through hiring, layoffs, and bringing on offshore contracted workers. Great flexibility! Another thing I like a lot are the UI Scripts, where you can customize the instance adding in or removing features and functionality. They are very powerful and super useful for us.

Cons:

It bugs me that every time you close a test run the test case id numbers have a huge spike, a case we created today had the id number of over 11 million even though we have less than 50,000 cases total in our library. I also do not like the way you add and remove tests from a run. It is a bit awkward and hard to see exactly what tests you're adding and removing.

If you are executing manual test cases on a large scale then Test Rail is for you

Comments: Having using various test management suites TestRail does excel in that it was built to run manual test cases and display results in an easy glance for the whole team. Linking to defects is simple, generating a report when the test run is complete. I feel like reusing regression suites could be improved, it does what it is meant to however you have to repeat several steps each time you want to generate a regression suite.
Some areas for improvement would be a test case type for API tests, I am sure everyone needs this but struggles to find a way to record this data.

Pros:

I think report generation is the strong suit of Test rail, the percentage pie chart of passed, failed blocked cases at a glance make daily reporting for scrum stand up meetings ideal. Integration with Jira is excellent, although some room for improvement.

Cons:

Trying to do something simple like import test case steps from a .csv is not so easy. You can import a suite but that is not ideal. Trying to duplicate cases can be tedious, let's say you have a lot of cases, trying to copy paste them in to various projects is a lot of hassle, it is surprising they have not addressed issues like this after so many years.