Posted by 
To paraphrase J. Cole, Facebook messes up on its math one time; shame on Facebook. Facebook messes up on its math a second time; shame on those of us blindly trusting Facebook’s math. Facebook messes up on its math a third time; time to make a list of all the times Facebook messed up on its math.
On threeseparateoccasions since September, Facebook has disclosed more than a handful of errors related to its own measurements that brands, publishers and others use to evaluate the performance of their Facebook ad campaigns, Facebook-native videos, Facebook-only Instant Articles and Facebook Pages, as well as their off-Facebook mobile apps and websites.
To help myself and anyone else keep track of these errors, here’s an itemized list of each error Facebook has announced to date, a (hopefully) normal-speak explanation of what the error was, whether Facebook has corrected it or plans to, how it has/will correct the error and what impact the error had on the measurements Facebook reported.
Average Watch Time of Pages’ Videos
Error: Facebook calculated the average time people spent watching a video by dividing the cumulative time spent watching it by the number of views. But it only counted views that lasted at least three seconds, which set an artificial minimum for the measurement.
Corrected? Yes.
How was it corrected? Facebook has created two different replacements for the faulty average watch time metric. The first, called “average watch time,” takes the total amount of time that people watched a video and divides it by the number of times that video was played, either automatically or after being clicked. The second, called “unique average watch time,” takes the longest single instance that an individual watched a video and adds that duration — and only that duration — to the video’s total watch time, counts one play toward the total number of plays and divides total watch time by total number of plays.
Impact: The faulty metric overstated videos’ average watch time by between 60 percent and 80 percent.
Pages’ Organic Reach: Part 1
Error: When gauging the total number of people who saw at least one of a Page’s posts organically over a seven-day or 28-day period, Facebook didn’t de-duplicate that audience when reporting a Page’s organic reach in its Page Insights dashboard. In other words, if someone saw a Page’s post on Monday, and then another post from that Page on Tuesday, Facebook counted that person as two people.
Corrected? Yes.
How was it corrected? Facebook now counts individuals exposed to a Page’s posts only one time per seven-day or 28-day period.
Impact: After the correction, Pages’ seven-day organic reach counts were expected to drop by 33 percent, on average, and their 28-day organic reach counts were expected to drop by by 55 percent, on average, according to Facebook.
Pages’ Organic Reach: Part 2
Error: When measuring Page posts’ organic reach, Facebook didn’t consider whether those organic posts actually appeared on someone’s screen.
Corrected? Not yet, but sometime “in the coming months,” according to Facebook.
How will it be corrected? As with its ads, Facebook will no longer count a post as having reached someone until it appears on their screen.
Impact: Once the correction rolls out, Facebook expects Pages’ organic reach counts to drop by 20 percent, on average.
Video Ads’ Completion Rate (more accurately “Times the Video Ad’s Last Frame Was Played”)
Error: Sometimes Facebook streams a video ad for “a fraction of a second” longer or shorter than the video’s actual length because the video’s visual and audio don’t line up, leading it to mis-time when the end of a video has been watched.
Corrected? Yes.
How was it corrected? Facebook has updated “how we read the video length,” according to the company. It’s worth pointing out that this metric isn’t truly a measure of a video ad’s completion rate. It’s a measure of how many times the end of a video was viewed, including times when someone skipped to the end of the video and only played that part (which would be a way to juice the measurement).
Impact: Post-correction, Facebook thinks advertisers will see a 35 percent increase in the number of “video watches at 100%,” which is not necessarily the same thing as completed views.
Average Time Spent on Instant Articles
Error: Facebook was supposed to calculate the amount of time people spent reading an Instant Article by dividing the total time an Instant Article’s entire audience spent reading by the total number of times the Instant Article was opened. But it didn’t. Instead it took an average from a histogram it made plotting time spent.
Corrected? Yes.
How was it corrected? Facebook now calculates average time spent the way it was supposed to calculate it all along.
Impact: Prior to the correction, the average time spent measured on Instant Articles was, on average, seven to eight percent higher than it should have been, according to Facebook.
Referral Traffic to Websites, Mobile Apps
Error: The “referrals” metric in Facebook’s Analytics for Apps — an analytics tool Facebook offers for third-party mobile apps and websites — was supposed to report how many times people clicked on a Facebook post to open a business’s mobile app or visit its website. It was supposed to report only that number. But it didn’t. Given that those posts linking to a business’s mobile app or site could also include a Facebook-native photo or video, Facebook incorrectly counted clicks on those photos or videos as referral clicks, even though these clicks didn’t send anyone to the business’s app or site but only enlarged the photo or video.
Corrected? Not yet.
How will it be corrected? Presumably, Facebook will limit its count to times when people clicked a post and actually opened the corresponding app or site, but the company has only said that it is “working to fix this.”
Impact: Facebook incorrectly counted, on average, 30 percent of the referrals it reported to mobile apps and websites using its Analytics for Apps tool, according to the company.
Like, Share and Comment Counts for Off-Facebook Links
Error: Facebook’s Graph API can report the total number of engagements (shares, likes and comments) a link received on or off Facebook, including when people paste the link into a Facebook status update and when they click the Facebook like button embedded on the link’s actual web page. Facebook’s mobile search results can report a subset of the Graph API total, specifically the number of shares and comments on Facebook. But sometimes the mobile search number is larger than the Graph API number, which should never be the case.
Corrected? No.
Why hasn’t it been corrected? Facebook hasn’t figured out yet why the two numbers don’t correspond.
Impact: Unclear. For one NBCNews.com link I examined late last month, the mobile search number was 71 percent larger than the Graph API number.
Reaction Breakdowns of Pages’ Live Videos
Error: Facebook’s Page Insights dashboard breaks down the number of “reactions” a Page’s Live video generated by whether the reactions happened while people were watching the live stream — “reactions on post” — or whether they were received on others’ posts that included the Live video — “reaction on shares” — such as when someone shares, or Facebook-retweeted, a Page’s Live video post to their friends and those friends clicked the like/reaction button on the friend’s post after the live stream had already ended. The “reactions on post” number was supposed to only measure one reaction per individual live viewer, and it did. But people may react multiple times to a live video, such as by clicking “haha” each time the stream makes them laugh. Facebook incorrectly included those extra reactions to the “reactions on shares” figure, even though they weren’t reactions on shared posts.
Corrected? Yes (or at least soon).
How is it being corrected? Starting in mid-December, Facebook will report the total number of reactions generated during the live stream within the “reactions on post” metric.
Impact: The corrections will result in a 500 percent average increase in the number of “reactions on post” and a 25 percent average decrease in the number of “reactions on shares,” according to Facebook.
