I’m writing this from my hotel room in Stockholm, after attending and conferring at Eurostar 2016. I’ve thoroughly enjoyed it and was able to meet so many different and passionate testers from places I didn’t even know had testers.
But this experience was somewhat spoiled by something completely different.
Here’s a few things that get me all riled up: Pokémon that run away and injustice. (amongst other things)
Today I’m complaining about injustice.
The Context
This injustice comes from a source that I had hoped valued integrity and transparency as much as it marketed to be: We-Are-Testers.com. (WAT)
They provide a service, like many other crowd testing companies, that links a customer with specific needs to a number of testers who are willing to jump at the opportunity to test something new AND make a few extra bucks while doing it.
I welcome this idea as it offers me the chance to do some extra testing, but also get some pocket-money to spend.
This mission, I had two days time to test an application on an Iphone and come up with linguistic issues in it.
Pretty easy right? Especially because all the Dutch text I had to review seemed to be copy pasted from a Google translate service.
However, some constraints hindered me: Only two days of testing were foreseen and the first day I spent an hour trying to install the app, only to find out the configuration of the install thingy didn’t work for me. An admin from WAT had to correct this for me.
The second day I was able to test for two hours before the deadline.
I logged 18 bugs, which would amount to maximum 144 EUR.
All required fields were filled in, as per agreement. Steps, expected outcome and actual outcome were all given as well as any other required fields. (How else would I be able to submit the bug?!)
Later, the next day, mails came flying in. INCOMPLETE, INCOMPLETE, INCOMPLETE.
Wait a second, there seems to be something wrong…
I hadn’t included screenshots.
Ok, I agree that a screenshot is pretty handy to have in most bugs.
HOWEVER, in this context, with two hours time to find as many bugs as possible, making, uploading, downloading, linking,… screenshots would heavily cut into my bug-finding time.
Steps, the name of the screen where the bug was found, the actual text and what it should become had to be more than enough for meager text issues.
In any case, for the three hours of work I wouldn’t see a penny.
And me being me, I kind of want to make a problem out of that.
Gathering evidence
The Discussion
I contacted the moderators and spokespeople from WAT, who are generally really nice guys and girls, to ask them to look into my situation.
Argument 1: Their website’s “Payment Policy” doesn’t mention Screenshots are a requisite for payout. It specifically says attachment only if relevant.
Argument 2: The mission’s “How to report a bug” doesn’t mention Screenshots are a requisite for a complete bug.
Argument 3: If Screenshots are a necessity for this project, please please please make the input field for screenshots a required field?
This is a clear miscommunication, so I gave them time to find out how they’d handle my situation.
Today I heard that I wouldn’t get a dime for my efforts, that it was unfortunate but that it was virtually my own fault.
My point is not whether screenshots should or shouldn’t be added to bugs.
My point is that WAT is telling me I should not be paid because I didn’t adhere to a rule that I can’t find anywhere in their Terms of Agreement, Payment Policy, FAQ or Mission Description.
They did offer four extra hours to insert the screenshots. At noon. While I’m at work. Thanks a lot!
They argue that the devs need screenshots, but frankly, that’s not my problem.
WAT provides a service to the devs. I provide a service to WAT.
WAT should carry the burden of handling communication errors on their part,
And I shouldn’t be the one to suffer from the gaps they leave.
I’m pretty sure those bugs will be fixed in the next version. But regardless of that, not paying people because of rules applied by a third party seems kind of illegal to me.
If you don’t agree, let me know.
Isle of IT 
Hi Berenvd
As a tester I disagree with you and the fact that you didn’t cover all the relevant bases. As a test lead, I expect a tester to ask all the questions, cover all the bases, make sure that there is no uncertainty in the required outcome. There should never have been anything left to chance and I believe in screenshots, especially when bug reporting, giving not only exact reproducible steps but a screenshot as well.
From a service provider point of view I also cannot completely side with WAT as this this should have been part of their T’s and C’s. They could have acted in good faith by compensating you at least half of the expected amount, as you had done the work, although not to the required expectation, but the expectation wasn’t clarified.
Sorry, but just how I see it.
regards
John Gordon
LikeLike
Hi John,
Thank you for your reply. It takes courage to express disagreement and I can certainly respect that. There’s nothing to be sorry about.
First of all, I would clarify that “ask all the questions, cover all the bases and make sure there’s no uncertainty, not leaving anything to chance” are all very dangerous expectations you set for yourself and your testers.
Be wary about (and with) such phrases. Especially under time constraints.
Secondly, I’m not arguing that screenshots don’t add value or anything and I usually add as many screenshots (+annotations) as is pragmatically interesting.
In this context, however, the mission was to find as many issues as possible and describe them in a good enough way.
Issues, being spelling errors and grammatical errors.
Good enough, meaning clear enough for them to be fixed.
If you’ve ever reviewed a paper or a document, you realize that screenshots of mistakes feel like overkill. Finding and correcting them in any other source shouldn’t be too much more of a hassle.
But I might be wrong about that and have no problem admitting that.
In any case, the whole screenshot thing aside, my point is that I made a decision/assumption about not including something that wasn’t required.
That *could* be interpreted as neglecting by my part.
In a normal situation, there’d be a discussion and enough time for me to add the extra information.
I would argue that these bases should be covered by the service.
While I’m working as a crowd tester, I’m not a consultant. I’m a tester that looks for certain information and report that information back in the format the service asks me to. Which is what I did.
Yet, the service and their client both will benefit from that work and the only one who isn’t, is me.
I’m making this pretty black and white, I acknowledge that.
Had they made any effort in reconciling and as you propose paid half, this blog post wouldn’t be here.
I’m more than happy to further clarify any questions you might have and/or I’m open to having further discussions.
Kind regards,
Beren
LikeLike
Hi Beren
I take all your reply for the value it adds to discussion, as I believe discussion is always important and can only add value.
However, from your reply I would highlight the following, “In any case, the whole screenshot thing aside, my point is that I made a decision/assumption about not including something that wasn’t required.” As testers I believe we should never make assumptions and always try confirmation where possible. As you mentioned there were time constraints and you chose what seemed to be the logical and or in your opinion not a requirement, and well and true not clearly stated as a requirement but however at the end of the day it left you having done the work but not being compensated as this non obvious requirement wasn’t covered.
I would concur with you that screenshots can be overkill and not really at all necessary but then we need to clarify this upfront with the dev team.
The above said, I nonetheless enjoy your articles and I always like to engage in healthy conversation as we can all learn a little from each other.
best regards
John Gordon
LikeLiked by 1 person
Hey berenvd,
It really sucks that you had such a bad experience. On my case, I retired from all those type of platforms, at the end, they want you to do a top quality job, in a short amount of time. It is really sad, that they did not state that screenshots are required and at the end mark them as incomplete.
You say you are putting things Black and white, but at the end they did the same thing to you.
Hope you have a better experience next time.
LikeLike
Hi Priamo,
Thank you for your support!
I wasn’t sure how this blogpost would be regarded by the larger community.
But I felt I had to share it none the less.
I like that I’m seeing other people speak up about it.
Working for a few extra bucks is nice, working for free is extra sour.
Good luck!
Kind regards,
Beren
LikeLike
I’ve had to many bad experience on those sites, first from crappy requirement documents, environment or steps to install the application out of date etc.
As mentioned before, I prefer no working, then working for free.
At the end, they still got their bugs, and you had $0.
LikeLike
Hi Beren, such is life and you’ve come across some less professionally acting individuals. Give them a chance to learn and improve. 🙂
As I’ve been working with and researching Crowdtesting providers intensively in the past, here are the ones I definitely recommend: http://www.test.io and http://www.testbirds.com
Take a look also at http://www.applause.com – they are market leader and that’s why their size can get in your way (a huge crowd means tough competition). So far I’ve been unlucky with them (from a customer perspective) – I did an exploratory testing as well as a scripted testing pilot project but that was still back in the days when they were called uTest and just trying to crack into the European Market.
LikeLike
Hi Beren,
this is a really bad experience you describe here. I also had some issues while testing these kind of platforms, you will find my feedback here with WAT, tester.io, utest and others: https://www.lyontesting.fr/en/crowd-testing-opportunity-or-scam-for-testers.
I think you (or your colleague) were very unlucky and it’s a shame that WAT didn’t support you and fix the issue (they did when I experienced it in a similar situation).
It’s true that screenshots are most of the time requested, but of course only if this is a clear requirement. I sometimes reported issues without screenshot and it was ok. It really should be ok in this case when you just need to file issues for internationalization issues. What’s the point in recording and uploading a screenshot of a text? The customer just need to search and replace (or sed it).
I think that maybe it was not clear enough between the customer and WAT, and the customer simply declined the issues. The schedule are always very short with these projects.
Did you have the opportunity to check the results a few weeks later? Might be interesting… 😉
Stéphane
LikeLike