Principles of Profiling Users with AI
Two days ago I read a Washington Post article about 3 Californian AI startups that profile users based on opaque AI algorithms. Their products calculate a quality score for people in different domains without explaining how it is calculated
The main problem with these and similar systems is that they use machine learning, in particular deep learning as a black box. Their algorithm gives back a score claiming high numerical accuracy without explaining how it has been calculated.
Our own Condor software is doing similar things, showing a bird’s eye view of the communication patterns of organizations based on their E-mail, or social media archives. There is one key difference though, we apply the “Google Maps Privacy Principle”: aggregated information is shown to all users, the individual information is only shown to the affected individual. The principle is derived from Google Maps, which becomes truly useful by aggregating the location information of Android users with location tracking turned on and iPhone users with Google Maps turned on through dynamically tracking their smartphone location. But the only individual who knows her/his own personal location is the owner of the phone. Google Maps therefore aggregates global information and returns individually useful information to the individual user.
This approach is what we are trying to pursue in our own work:
(1) Show aggregated information to the public, and individual information only to the affected individual.
There are however applications where the individual user has to be identified to others. These applications can be split into two categories.
(2) The application needs to identify the user, and the user gets a benefit from being identified, for example as a “rock star” employee, most collaborative employee, etc.
(3) The application needs to identify the user, and the user has a disadvantage from being identified, for example as a potential security risk, low performer, etc.
The applications from predictim, HireVue, and Fama are clearly in category (3). Users are convicted by a machine learning algorithm without knowing why. The algorithm operates as a black box. While arguments can be made for category (2) applications to run in such a mode – the user gets a pleasant surprise, even if s/he does not know why, this is clearly not acceptable for category (3) systems. At the very least does the user need to know why s/he has been convicted.
As I assume that more and more systems in category (3) will be built, for instance by law enforcement, I envision the need for an impartial authority, which can be public or private, to check and certify the accuracy of these AI-based prediction systems.
A second point which sets Condor apart from other AI-based prediction systems is the transparency of its algorithms. The scoring algorithms applying the "seven honest signals of collaboration" and a list of predefined "virtual tribes" is documented in great detail in over 150 academic papers and five books. This is quite different from e.g. predictim's case studies, which predict the past, without disclosing how it is done. There is no guarantee that the training data of the past will still be valid to detect future criminals.
It is still early days, so please tell me what you think, I would love to hear your opinion
- Predictim.com calculates a “risk rating” of babysitters based on their social media activities.
- HireVue analyzes tone, word choice, and facial movement of job candidates to predict their skill on the job.
- Fama does employee screening on social media and internal HR data to prevent what they call “brand risk” such as sexual harassment, bullying, or insider threats of employees.
The main problem with these and similar systems is that they use machine learning, in particular deep learning as a black box. Their algorithm gives back a score claiming high numerical accuracy without explaining how it has been calculated.
Our own Condor software is doing similar things, showing a bird’s eye view of the communication patterns of organizations based on their E-mail, or social media archives. There is one key difference though, we apply the “Google Maps Privacy Principle”: aggregated information is shown to all users, the individual information is only shown to the affected individual. The principle is derived from Google Maps, which becomes truly useful by aggregating the location information of Android users with location tracking turned on and iPhone users with Google Maps turned on through dynamically tracking their smartphone location. But the only individual who knows her/his own personal location is the owner of the phone. Google Maps therefore aggregates global information and returns individually useful information to the individual user.
This approach is what we are trying to pursue in our own work:
(1) Show aggregated information to the public, and individual information only to the affected individual.
There are however applications where the individual user has to be identified to others. These applications can be split into two categories.
(2) The application needs to identify the user, and the user gets a benefit from being identified, for example as a “rock star” employee, most collaborative employee, etc.
(3) The application needs to identify the user, and the user has a disadvantage from being identified, for example as a potential security risk, low performer, etc.
The applications from predictim, HireVue, and Fama are clearly in category (3). Users are convicted by a machine learning algorithm without knowing why. The algorithm operates as a black box. While arguments can be made for category (2) applications to run in such a mode – the user gets a pleasant surprise, even if s/he does not know why, this is clearly not acceptable for category (3) systems. At the very least does the user need to know why s/he has been convicted.
As I assume that more and more systems in category (3) will be built, for instance by law enforcement, I envision the need for an impartial authority, which can be public or private, to check and certify the accuracy of these AI-based prediction systems.
A second point which sets Condor apart from other AI-based prediction systems is the transparency of its algorithms. The scoring algorithms applying the "seven honest signals of collaboration" and a list of predefined "virtual tribes" is documented in great detail in over 150 academic papers and five books. This is quite different from e.g. predictim's case studies, which predict the past, without disclosing how it is done. There is no guarantee that the training data of the past will still be valid to detect future criminals.
It is still early days, so please tell me what you think, I would love to hear your opinion
Comments
Post a Comment