A Series of Tubes

I previously posted that the eastbay express article along with a follow up article ran about Yelp.  They articles claimed that business owners were extorted by Yelp.  The CEO of yelp responded on his Blog (twice), but did not respond to the second article which did not have anonymous sources.  I posted a 1 star review of yelp on yelp.com.  Within yelp itself, the CEO of yelp, Jeremy Stoppleman, responded to my 1 star review and to the allegations made in the east bay express stories and we have been going back and forth a bit today.  I figured I would post the exchange below since it is interesting.  I will always remember this day as the day my career as a hard hitting interviewer took off.

Update: this post keeps getting updated so I’ll just post the date of the last change: (4/13/09)

For formatting purposes I’ll put my replies in orange and removed the signatures:

We already thoroughly responded.
http://officialblog.ye…

My problem with the initial story wasn’t the anonymous sources, there are verifiable claims you can check for yourself.

Jeremy

East Bay Express has an update to their initial story (please see http://www.eastbayexpr…).  In it they did not use anonymous sources.  Since this update from east bay express clears up your major concern with the initial article, can you respond to the updated claims from east bay express regarding the named sources that yelp participates in extortion and regularly removes postings.  Also saying that the algorithm for postings is proprietary decreases the confidence that users have that yelp is being fair and open it its business practices and only helps further the claims made by names sources in the east bay express’ updated article.

The crux of my rebuttal was not that she used anonymous sources (that was a side point). Please look at my two posts, she claims extortion (title:Yelp and the Business of Extortion 2.0), asserting we take money for removing negative reviews. Anyone can validate for themselves that this is not the case (simply browse a handful of advertisers and look at their reviews, not hard to find a negative like everybody else).

Some businesses clearly don’t like Yelp and fear the change Yelp and sites like it represent. The problem with her story is that the allegations are testable and obviously untrue (note they are not supported or echoed by reputable journalists who looked deeper).

http://officialblog.ye…
http://officialblog.ye…

Even vocal Yelp critics concede Yelp is both good for local businesses and “is the future”
http://officialblog.ye…

I agree having a secret algorithm causes suspicion, but its something required to keep the site useful and is the same problem Google has. If we did not have the aglo the site would be useless for consumers (why don’t people trust CitySearch reviews?).

Thanks for replying so quickly.

You said that “simply browse a handful of advertisers and look at their reviews, not hard to find a negative like everybody else”. I am a scientist and just looking at the reviews of advertisers is not the test that you would run in order to show foul play. The test would be to show that before and after signing up for the advertising program, the displayed reviews on the business did not change. One would need to see when an advertiser was first contacted about the advertising program with a cached version of the review page and compare that to one soon after signing up. Then you would need to collect data from those who were contacted about the advertising program, but declined yelp’s offer to join. If there is no statistically significant difference in the quality of the top displayed reviews for these two categories then you will have proven no wrongdoing.

Of course in order for the data to be trustworthy, an independent source would have to run this test and the list of companies in the sample size would have to be random (not selected by eastbayexpress or yelp). Such a test would also allow yelp to keep its comment and review filters a secret while clearing its name of any allegations.

Just looking at the comments for paid advertisers is anecdotal and inconclusive. Until a study, such as the one I’ve outlined above, is conducted, yelp will remain in a constant he said, she said with eastbay express over the question of extortion.

As for the specific allegations made in the second eastbay express story, I’ll stick with the named sources in the second article since the anonymous sources cannot be verified, you noted that they are not supported by reputable journalists that looked deeper. Unfortunately a google search for “yelp extortion untrue” or “yelp advertising” on google does not yield these articles. Perhaps you could forward or link to the articles written by the more reputable journalists that refute her claim so that I see things from their perspective.

Again thanks so much for your fast responses.

The numerous articles are listed here at the top of my post (a link I previously sent):
http://officialblog.ye…

As a scientist I’m sure you agree that actual data is far more important than claims, theories, or accusations. If you read through our reviews of Yelp on Yelp you’ll also see the occasional 1 star review by an advertiser who is upset that we didn’t remove a review. So it goes…

—————————–

I have yet to send a response because Jeremy keeps linking to the same blog posting about the original article and is not addressing the updated article’s claims.  I will however look through the links on the top of the blog post from Jeremy from these more reputable reporters along with my analysis.  I will be looking for reporters “looked deeper” directly :

Inc:  This is not an investigative report at all.  The writer is armchair commentating on something he has not done the legwork on.  The crux of the argument from this post is that yelp did not use named sources (something the second east bay express article did).  Hopefully this article is not indicative of the rest of the links.

NYTimes:  The article only briefly mentions the claims of extortion.  There is no depth to that portion of the reporting and the only conclusion that can be drawn with regards to the claims made by east bay express is: ‘“If there’s no clarity about that process at all, it exacerbates the suspicion,” said Eric Goldman, a professor at Santa Clara University School of Law and the former general counsel of Epinion.’  This article did not dig deeper into the claims of extortion….strike 2

SJ Mercury News:  This one is awful awful awful.  Someone take away this reporter’s license to claim any authority on anything.  All he does is interview Jeremy and then not challenge a thing he says.  It would have been harder for the reporter to have done less work or to have kissed more ass.

Financial Times:  This reporter basically has a summary of what happened.  The east bay express claimed yelp extorts businesses, the CEO of the company denies it.  I guess I should reiterate what I’m looking for.  Jeremy said “the allegations are testable and obviously untrue (note they are not supported or echoed by reputable journalists who looked deeper)”.  I’m looking for someone who looked deeper into these allegations.  So far I am hugely disappointed since the articles I was told would contain evidence that directly contradicts the east bay express story instead have summaries or quotes of Jeremy as the only evidence presented.

SFWeekly:  This is finally a more substantial article (weighing in at 5 pages).  My summary-  Page 1: a florist got a bad review and went through hoops with the reviewer to get it changed (unrelated to the allegations of extortion). Page 2: A history of yelp and a quick bio of Jeremy.  Page 3:  This is where the allegation is mentioned.  The reporter simply parrots the same thing that Jeremy said to me which is “look at the sponsor’s reviews.  They have bad reviews just like everyone else.”  If you look at my response above, I kind of pointed out that the argument holds less water than a strainer.  Pages 4 and 5 are mostly fluff or things unrelated to the charges of extortion.  So far this article comes the closest to refuting the claim made by east bay express.  However the argument against the east bay express story is actually a quote of Jeremy himself.  Where is the independent evidence to refute the extortion claim.  I don’t need yet another quote of Jeremy claiming it’s not true.

Greg Sterling (local analyst):  Greg is analyzing the NYtimes article from above in a blog post.  The conclusion he draws, without doing any reporting himself, is that “This article, however, should put to rest the allegations of review manipulation to reward or punish small business advertisers (or prospects) at Yelp”.  Sorry Greg but I did not get that out of the NYTimes piece.

Huffington Post:  Well the bias is stated in the article: “First, let me put a couple of things out in the open. I am far from unbiased here. I have a clear conflict of interest in that my own company RateItAll, also offers a local business rating service“.  Actually I really agree with one portion of this analysis: “While allowing businesses to pay to remove negative reviews might improve short term revenue, it wouldn’t be long before Yelp’s credibility with consumers was shot.”  Unfortunately it looks less and less like yelp opted for the short term revenue while getting puff pieces published with Jeremy quoted in them.  Also this article does not have any reporting work that would add anything to the conversation, it just has commentary from an openly biased individual.

Digital Media (CNet):  This is a good summary of what has happened to yelp.  The article quotes officials from yelp but also quotes a reviewer saying” “I have switched to Yahoo Local for reading reviews,” Estelle said. “I don’t trust Yelp’s tagline anymore, ‘real people, real reviews.’”  This article simply states the he said she said arguments without figuring out who is right.

The Social (CNet):  This is not a report but simply an interview of Jeremy.  I’m noticing a trend and this is getting tiring.

AppScout (PCMag Network): This article basically says that the eastbay express article did not cite its sources, something Jeremy complained about and something eastbay express did not do in its second article on yelp.

—————

Jeez that took a while.  I guess the take away message from Jeremy is that he thinks that if he repeats the “We don’t extort people” line enough times in enough “articles” that everyone will agree.  Unforunately the repitition did not prove his case and made me pissed off that he thought I was that stupid.  I’m not changing my 1 star review of yelp.

Update: Ok I sent one more reply back.  Here it is below

As a scientist I do agree that actual data is far more important than claims, theories, or accusations. Unfortunately an occasional one star review of an advertiser’s business is not data that can be used to weigh in on whether yelp modified the reviews of that business around the time that the business joined yelp as an advertiser. Like I said before:

” One would need to see when an advertiser was first contacted about the advertising program with a cached version of the review page and compare that to one soon after signing up. Then you would need to collect data from those who were contacted about the advertising program, but declined yelp’s offer to join. If there is no statistically significant difference in the quality of the top displayed reviews for these two categories then you will have proven no wrongdoing.”

The above data would weigh in definitively one way or the other on the claims made by east bay express.

I noticed that in the articles linked in your blog posting the reporters did not directly refute the claims made by east bay express. If a refutation of the extortion claims in the article appeared, it was usually a quote of you or a representative of yelp. I was wondering if there is an article where the reporter is concluding that extortion does not occur at yelp instead of simply quoting you as saying so.

Thanks so much.

“Indeed, a look at any sponsored review page, which will likely feature at least a few negative reviews, would seem to discredit that.” – SFWeekly

“I’m sorry but that ain’t enough. Yelp listings, like Google search results, change regularly. Sometimes negative reviews move up, sometimes they move down. A simple correlation does not mean that anything nefarious is going on. It just means that businesses that get bad reviews generally get angry about those bad reviews. It also doesn’t seem totally crazy that one of Yelp’s 200 employees might happen to dislike an establishment that also got a sales call.

So I’m not convinced there’s anything untoward happening.” -Inc

The Inc article was in response to the first east bay express article.  The main thrust of the complaint in the article is the lack of named sources.  If you were to continue the Inc quote:

“So I’m not convinced there’s anything untoward happening. I’ve always found Yelp’s reviews pretty accurate. But let’s see if more people start coming out of the woodwork with real examples of abuse.”

The second east bay express article, with it’s numerous named sources, would seem to meet the criteria laid out by the Inc reporter.

An expanded version of The San Francisco Weekly article is as follows:

“Stoppelman denies accusations in recent press reports by several business owners who say Yelp sales reps told them they could pay to rearrange reviews to bury or even remove negative ones as “a rumor that’s taken on a life of its own.” Indeed, a look at any sponsored review page, which will likely feature at least a few negative reviews, would seem to discredit that. The order of reviews on any given page is weighted by its recentness; how many users have voted it “funny,” “useful,” or “cool”; and being an appropriate length, Stoppelman says.”

As you can see the reporter appears to be quoting you.  Regardless, pointing out the existence of negative reviews on a sponsor’s page is not the correct test to see if there was user review manipulation around the time that businesses were contacted by yelp regarding your advertising program.  I would love it if you would address the east bay express articles claim, that the reviews of businesses were altered soon after being contacted by yelp, head on.  I hate to sound like a broken record here but the true test to discredit the actual accusation made by the east bay express is:

” One would need to see when an advertiser was first contacted about the advertising program with a cached version of the review page and compare that to one soon after signing up.  Then you would need to collect data from those who were contacted about the advertising program, but declined yelp’s offer to join.  If there is no statistically significant difference in the quality of the top displayed reviews for these two categories then you will have proven no wrongdoing.”

A fair analysis from a trusted source that went through the above steps would be the definitive and best method of demonstrating that yelp did nothing wrong, and continues to do nothing wrong.  Would yelp be open to performing such a study and releasing the results to the public?

From the NYtimes article:
“If there’s no clarity about that process at all, it exacerbates the suspicion,” said Eric Goldman, a professor at Santa Clara University School of Law and the former general counsel of Epinions, another review site.

I couldn’t agree more.  Thanks for the prompt responses.

8 comments