Flirt Detectors and Robot Moderation on Patook
Published 8/11/2018

Patook is an app for making strictly platonic friends.
One of the main differentiating features of Patook, and probably the main reason Patook is still being used exclusively for friends (rather than slowly turning into a dating app), is the flirt detector ("FD"). This post talks about the FD and touches a bit on some of the other detectors (collectively "Robot Moderation") present in Patook. Please note this is an informal blog post that may not be up to date with the Terms of Service - Patook is governed by the Terms of Service and Privacy Policy and they always supersede anything mentioned in this post.

What is the FD?
The FD is a novel and unique system, whose purpose is to try to identify users who are suspected of flirting, and ban, hide, or block them from the site. Flirting and any other non-platonic behavior on Patook are strictly prohibited.

How does it work?
The FD algorithms are secret, unique, and proprietary. However, they monitor nearly all users' activity and messages and build a model that provides us with the likelihood of a user being on the app for non-platonic reasons.

As an example (which may or may not apply to the Patook FD), a user could receive a red flag for sending a message like "you're hot", and another red flag for showing too much skin on their main profile picture

A married 50 year old man messaging exclusively 18-year old women might get a flag.

A user who is getting hit on and encourages the sender (rather than reporting/blocking) might get a flag.

A new user who receives multiple "behaving creepily" reports from established users might also get flagged.

The FD is trained on millions of messages and billions of activity data points and is a work in progress, but takes up the biggest chunk of development. It has reached the point where there are hundreds of thousands of rules, so most the training for it is done at an aggregate level (i.e. we don't usually fix problems with it one at a time, but thousands at a time).

What is the vision and drive behind Patook's FD?
We have the vision of Patook being a community where people can approach one another in a strictly platonic way, without having to worry about this person wanting to have sex with them, or that person trying to enlist them into their pyramid/MLM scheme, or whatever. Just a bunch of friends with no ulterior motives. So we work very hard to build and refine the algorithms, even if they sometimes end up being controversial and way too strict. From what we've seen, a good chunk of people will only approach others for ulterior motives, even after the dozens of warnings - and we do not hesitate to ban them.

What does "strictly platonic friends" mean?
On Patook, strictly platonic friends are friends with whom you don't share any romantic, physical or sexual desire or connection.

Why have a FD? Why not just let users report users who are flirting?
Because there is a large chunk of "flirters" or bad apples in the general population and they quickly overwhelm any effort to report them. In addition, they tend to be far more aggressive in messaging than people looking for genuine friendships.
If you open your inbox and half the messages are "send nudes", "nice boobs" or "join my MLM scheme", you're most likely not gonna bother reporting them individually. You'll probably just leave the community, which will give even bigger reign to those who seek to abuse it. If you're married/in a relationship, you'll probably be even more sensitive to receiving such messages. In certain countries it almost even seems like the concept of approaching someone of a different gender platonically is non-existent, so the FD has to work overtime to avoid an inbox completely full of inappropriate comments. With the FD intercepting the majority of those messages, and banning many of those users before their interaction reaches you, it becomes more manageable to just report say one out of twenty messages.

In addition, having an FD helps protect recipients who might otherwise be too uncomfortable or even intimidated to report someone. The sender usually ends up getting angry at the FD (and the app - check out our 1-star play store reviews) and taking it out on us, rather than on the recipient who would otherwise have had to report them.

There are hundreds (if not thousands) of dating apps on the market and Patook is extremely explicit about the fact that it's for strictly platonic friends only, so as a policy we have zero tolerance for anyone who is suspected of flirting (or other ulterior motives).
If you are on the fence about this, we invite you to try other "friend" apps instead. In our experience they've mostly either leaned heavily into pure dating apps, or have decided to completely ban certain genders (e.g. no men allowed, or no talking between men and women, etc.).

Can I look for people for "friends or more", or "friends first and let's see what happens"?
Absolutely not. Patook is not the place for it and the "or more" counts as an ulterior motive and tarnishes the "strictly platonic" aspect of the app. We consider it to be deceitful and manipulative for users to use a strictly platonic app to try to bridge their way into more. Users need to feel confident that when they are being approached by others, it is not someone trying to sneak their way into their hearts, pants, or pockets. The FD can help detect things like this as well and we have banned thousands of users that thought they would step into the "or more" phase.

If you are not able to keep your emotions platonic, then please restrict your interactions to genders you are not attracted to.
Otherwise, please use a dating app specialized for this purpose instead.
The FD will not hesitate to interrupt your conversation right in the middle and completely ban both users.

Why can't I just turn off the FD?
Similar answers to the above. If we start allowing users to turn off the FD, we believe Patook will snowball into a dating app. If you really want this option, please try some of the other apps out there; many of them have dual friend/dating purposes.
If the FD is misbehaving or incorrectly flagging things as flirty when they're not, we'd like to fix it.
If it is not, then we might assume you're turning it off so you can flirt.

Why is there no human appeal process for the FD?
There used to be one but we removed it for a couple of reasons:
1- There's just too many users that get flagged as flirty/creepy/MLMy/etc. We cannot deal with every one manually. Patook has millions of messages and billions of actions every month and bad apples make up a good chunk of those. We review random samples every once in a while to make sure it hasn't gone completely out of control.
2- Many of those people are unaware that what they're doing is creepy or inappropriate and we don't want to be in the business of explaining why something is creepy. Let's give a couple of examples, modified for anonymity. Bob is a 50 year old man that liked telling 20-year old women that their bodies are hot and that if he was younger [...]. Bob was quickly banned. Bob appealed with the very common appeal argument: "I wasn't flirting, I was just complimenting them.". We have to reject the appeal since we "think" Bob is being flirty, and this person is essentially chasing away many users. Bob certainly doesn't think what he did was creepy and goes on all social media to complain about how he was unfairly banned.

Alice was lured into an pyramid scheme (not through Patook) and wants to share the revolutionary news with half the Patook users and get them to join. Alice gets banned. Alice appeals with the common "it's not a pyramid scheme" response. Alice thinks she was banned unfairly and we are preventing her potential Patook friends from becoming rich.
3- Flirting (and other ulterior motives) is subjective and different people have different opinions as to what it might mean. So is "creepiness"; the same actions and messages might be construed as creepy or not based on how you look. So we believe it's more fair to give full authority to the robotic FD output and then only modify it in aggregate for many users if we think it's overtriggering, rather than making exceptions for individual users.

By joining the app, you agree that your continued usage is contingent upon the FD's and Robot Moderation's approval.

How about human moderation?
We do have some volunteer human moderators that can review content at their discretion. In general, all public content can be reviewed by moderators, even if they are not Patook staff (see Privacy Policy).
We often offer moderator status to Patook users who have a history of being strict on flirters/spammers (e.g. they report a bunch of people, or call them out whenever they're approached inappropriately).

In addition, the community itself moderates users. If you're getting reported multiple times, or e.g. more than half the people you've talked to have blocked you, you likely won't be on the site very long. There's also options to report based on "intuition", which also are counted.
Private messages and all user activity can be reviewed by Patook staff at any time for any reason they deem appropriate, and without notification.

However, as mentioned in the previous point, there is far more that gets flagged as flirty/creepy than we will ever have moderators/staff for.

Does the FD only do bans?
It can take any action it deems appropriate and essentially has the same privileges as moderators. You can check the terms of service through the in-app link or link at the bottom of the page. It can block messages, hide messages, block users, hide users, etc. and it can do so without notifying them.

Are you saying all bans are fair?
Unfortunately not. The FD is just a robot algorithm and can make mistakes. We re-train it with nearly every update to account for new data and new user behavior. If you were on the app during Patook's beta launch, you'll remember that it was flagging things as flirty left and right. It's come a long way since then and we still plan on making it go much further (hopefully to where there are minimal false flirt detections).

In addition, the FD is tuned to be extremely sensitive. If it learns someone was hit on and evaded detection it will try to overcompensate in the next iteration.

Does the FD catch everything?
No. It is a novel and experimental detector (so are all the other detectors). There are absolutely no guarantees on its accuracy, and in fact it's pretty terrible in many non-English languages. FYI it helps if you add the languages you're speaking in your profile.

Someone hit on me!
We're sorry this happened. Please use the "..." button and report the conversation as flirty or creepy. If confirmed, the user will be banned.
In addition, if the user sent you an inappropriate picture (e.g. "dick pic") within the app, please report it as well. We may opt to take legal action against them - before being given permissions to send pictures, users agree to a 1,000$ fine (donated to charity) for sending inappropriate pictures.

Why are many obscenities intercepted or censored?
In general, we err on the side of being conservative. Most obscenities are blocked unless they are used in a context which the FD has seen many times and knows to not be flirty. The word "fuck" for example is generally blocked. However, if it's in the context of e.g. "what the fuck", then it might not be. As the FD gets trained with more and more info, it learns which context which obscene words are generally acceptable in.

Will the FD always notify users they were banned?
No. It used to but now rarely does. Mostly because people who were desperate enough to repeatedly try to flirt just kept trying to rephrase until they evaded it. "you're hot", "ure h0t", "ur ht". It can now silent- or soft-ban users. This was even worse with scammers. Per the ToS, no notification needs to be sent; however, they do get sent occasionally at its discretion.

Do you ever review FD flags?
Per the ToS, the Patook team can review anything (including private communications). In more practical terms, we review a few FD flags at our discretion - for example, if the FD tells us it wants to ban a user who's been in good standing on the app for a year, we might suspect something went wrong. Or if it tells us that he suspects someone is flirting but it's not entirely sure and wants human review, we might give input. Or if a particularly bad user manages to evade it, we might try and figure out what went wrong. These are all a few examples and do not constitute a complete list.

And when retraining the FD, we have access to all the stuff that was marked as "flirty", and the stuff that was reported by users (that we failed to mark as flirty) and we do overall reviews.

Are there other detectors?
There's many more, the majority of which are secret. But for example we do try to detect NSFW pictures, scams, spam, fraud, invalid accounts, duplicate accounts, etc. They all more or less have similar shortcomings to the FD. By continuing to use this app, you agree that all your activity/messages may be monitored by them and potentially escalated to human moderators without notice.

Does the FD or other detectors interact with police directly?
Per the ToS and privacy policy, we may file police reports on any suspected illegal activity at our discretion. In certain limited cases, the Robot Moderation system may file automated police/NCMEC reports without human staff review.

However, automated police reports are very rare and are reserved for cases where it would be unacceptable to ask Patook staff or moderators for verification - for example, where exploitation of minors is suspected, we will defer verification to the NCMEC.

"The FD should do this or that"
We've worked on the FD for years and have a decent understanding of what kind of abuse it is exposed to, the main categories of "bad users" it has to deal with, its strengths/weaknesses, as well as many plans to improve it over the next few years. Also, per the terms of service, we do not accept unsolicited ideas.

We understand this isn't for everyone. If you do not like the app, the flirt detector, or the community it has helped create, you can delete your account at any time by going to Me -> General Settings -> Delete Account.

Terms of service are HERE

The above article is Copyrighted (C) by Patook LLC. Patook (R) is a registered trademark of Patook LLC.