Testing your software is really important. When time and personnel are abundant, you may not give testing too much thought. Maybe there is a separate team that focuses on integration testing, or UI Automation, or writing the Unit Tests. The goal is then clear: the teams simply spend all their time creating tests. However, when resources become constrained and everybody is doing a little bit of everything, the issue gets more complicated. Every hour counts, and so a decision must be made to add new features, write new tests, or maybe do some maintenance work. If you are going to test, then what kind of test should you create? Do you automate the UI? Do you treat the code as a black box and test at the service API? Do you write Unit Tests for every logical branch in your code? Is Test-Driven Development (TDD) really worth the time investment? None of these questions have clear answers, but we will explore what has worked for us and what hasn’t in this discussion.
Great topic! I’ve gone back and forth on this over the past few years, having experienced lots of the extreme cases on either side. I agree that there’s no “correct” answer. I’ve recently run into people who fall on the side of “write Unit Tests if you have time”, which everyone knows you’ll pretty much never have. If you believe in Unit Tests, you should be writing them. After all, if you think you can know whether a feature is finished or not without tests, why have tests at all?
From my TDD background, I was once on the side of “Unit Test everything”, but that ended up resulting mostly in a bunch of interaction tests that were tedious to maintain and of questionable value. More recently I want to lean more towards the “black box” approach you mention above, and just stub out the endpoints, but want to find a way to do it where the tests are still of a small and maintainable size.
Also, the only thing worse than no tests is bad tests. Maybe a point to discuss a bit later on. How about you? Where are you falling these days on the spectrum of testing?
Before I describe where my opinion falls on testing, details about my work environment are important. First, if my code breaks it will not kill anybody! Second, our software is used internally, so the time it takes to hot fix bugs is much smaller than if we had a shrink wrapped product.
I have tried to follow TDD strictly and had issues with it. The main problem is that unless you have a crystal clear understanding about what needs to be coded, it can be difficult to write the test. What should the name of the test be? What should the name of the supporting classes and methods be? How will everything interact? These are all design questions, and I have had much more success trying to think through some of them without a keyboard in my hands. TDD can give you a false sense of security, and if you got anything seriously wrong in the design and have to refactor heavily, then you will have a lot more code to re-work now. Tests are great when you are making changes within a single method being tested, but if logic and data structures have to change, you’re in for a lot of unneeded work.
If you are not following TDD and are just as bad at design, then you are going to have different problems. It will be impossible to ever come close to any estimates made. From my experience, programmers underestimate their work nearly all the time. This will guarantee that you will never have time to write tests after the fact. So what you are left with is a strategy that makes work take way longer than it needs to. And another method that will get work done but leave no time for enhancing your test suites.
My current testing strategy isn’t too complicated. By default, we always architect the code to use Dependency Injection. Whether or not you are testing, it’s nice to have all your dependencies in the constructor. With this in place, your setup to add Unit Tests in a more piecemeal fashion. I will add tests whenever I don't feel totally comfortable with what the code is doing. Even if I feel confident, I will still try to write a few tests, just to keep myself in check with reality. I weave this throughout my code writing, not all before, and not all after.
The other type of test I like to write is an integration test after the code is pretty solid. It’s more of an extra verification than it is a test to drive the development from. For example, we have a complicated algorithm that generates a data structure as its output. The test will run the algorithm from inputs stored in a locked down test database. It will then load the result into memory and serialize it to JSON. We use Approval Tests to verify the resulting JSON against the JSON we have previously approved. This is usually just the result of the algorithm after it’s gone through Developer and QA and verification. If anything changes in the output, the test will fail and open the results in a diff tool which requires a manual inspection. One of these tests can easily be worth many Unit Tests, and the effort to create it is small.
The one time I do follow TDD is when fixing bugs. One thing I like about this approach is that the bugs are bound to be in the more overlooked code. Writing tests to expose the bugs makes sure the architecture is maintainable and more resistant to future bugs.
Yes, designing perfectly up front is impossible. TDD expects that you will be doing constant refactoring because you don’t fully understand the problem up front. The less you understand up front, the more you have to rework your test suite later as you learn more. It can be very time-consuming. But TDD can be a great way to incorporate dependencies that you aren’t as familiar with. It’s much faster to try something and run a test to see if it worked then it is to rebuild, deploy, and smoke test your software.
I agree with your points about Dependency Injection (DI). Not doing DI at all is baffling to me these days. It simplifies testing and makes your code more composable, at virtually no cost. Writing Unit Tests as a way to learn more about the code is a good approach too. Then, not only do you learn about it, but it’s some form of documentation that wasn’t there before. And it ties nicely into refactoring, as you said, because if you’re working on some code that isn’t tested, you should write a test to expose the bug, then refactor the code as you fix it to make the code more readable and testable. Having a bugfix tied to a line of code and a series of tests adds a great deal of comfort to both developers and business people.
I think Unit Tests in some form are necessary, but TDD isn’t a requirement. I love the idea that legacy code is“any code that doesn’t have tests.” One of the defining features of Legacy Code is the fear of changing it, and that’s something that tests should help you with. Tests (whether they’re Unit Tests or not) should prove that your system behaves as expected, so once you write tests around your legacy code, you can refactor it and be assured that you didn’t break anything as long as those tests still pass. And much as we may not prefer it, the vast majority of us work on legacy codebases, so we should do whatever we can to make this as tolerable and relaxed as possible.
It sounds like we agree that it’s bad to put no design effort into the code up front and rely on refactoring to save the day. Especially when it can be made more expensive to do once tests are created. I am right on board with testing unfamiliar code. I also like to write tests around code that I am not totally comfortable with. Even when writing new code, it can be much easier to create a test than to go through a bunch of manual steps in the UI to re-create the scenario.
While I have seen some businesses operate without tests, it always comes at a performance and maintainability cost. I like a mixture of Unit and Integration tests to maximize my output because I tend to be very time-constrained so it’s all about efficiency. I don’t really believe you need to strive for super high code coverage, especially if it’s at the cost of new features. This ratio needs to be balanced. Some code is just so simple that it really doesn’t need tests. Every time another person reads it there will be a level of verification.
I agree; the biggest challenge to doing testing as thoroughly as we like is dealing with time constraints. I'm glad you mentioned code coverage too. People (mostly managers) love to see coverage percentages increasing over time, but it can easily be a false sense of security. As we said earlier, bad tests are worse than no tests. It's easy to have a Unit Test that covers a path through the code, but it's not a useful test and could end up always passing, even when crucial business logic changes in the function it tests. It makes me wonder whenever I see those tests: did whoever wrote this really put their best effort into writing the test? Or were they sloppy because they were under pressure to get the code out the door and increase test coverage? I love these opportunities to train people on testing. Actually, that's a path I am planning to go down when I'm next interviewing people. “What's wrong with this test and how could you fix it?“
I feel like we may have reached a good stopping point. Being time constrained can be a good thing because it forces you to really think about what’s actually needed. Also, writing tests does have its costs; it’s not all benefits. It’s important to always look at both sides. We have hit on some others ideas that could be full topics in themselves; like, what type of information is useful to communicate to others outside of the immediate development team; and what are some good ways to conduct interviews to try and get as clear a picture as possible about the candidate.
I think that’s about all I can think of on the topic for now as well. Basically, be thoughtful and careful with testing, and don’t just do things because some rituals tell you that you have to. It sounds so obvious when we summarize like that. I agree that there’s good stuff in here for future topics. Interviewing is one of the trickiest subjects and close to a lot of people’s hearts, so that would be a really good one. Thanks for a great discussion!
