Contents
AI Cards: Use Case
Protify: an AI-Based Student Proctoring System
Cite as: Delaram Golpayegani, Isabelle Hupont, Cecilia Panigutti, Harshvardhan J. Pandit, Sven Schade, Declan O’Sullivan, and Dave Lewis. "AI Cards: Towards an Applied Framework for Machine-Readable AI and Risk Documentation Inspired by the EU AI Act" In Annual Privacy Forum, pp. 48-72. Cham: Springer Nature Switzerland, 2024.
Copyright © 2024 the document editors/authors. This work is available under the Creative Commons Attribution 4.0 International Public License; additional terms may apply
Protify Description
Proctify is intended to suspicious behaviour during online exams by analysing facial behaviour from a student's facial video captured throughout the exam using a webcam.
Prior to this, students have explicitly consented to be recorded during the exam and informed that they must be alone in the room.
The system incorporates a graphic interface displaying an analysis of the student's face including the head pose, gaze direction, and face landmarks' positions.
This extracted information is then provided as an input to SusBehavedModel, which has been trained in-house by the system's provider using SusBehavedDataset,
to determine whether the student is displaying suspicious behaviour, e.g. looking away from the screen, leaving the room, or a third person detected in the room.
Detection of suspicious behaviour raises an alarm in the interface to inform and let the human oversight actors, e.g. human instructors, take appropriate actions, e.g. communicating with the student.
Protify Risks
Proctify's provider has implemented an AI risk management system, as a part of its overall AI management system that is in conformity with ISO/IEC 42001.
Throughout the risk management process, the risks and impacts of the system are identified and assessed, including the following:
These events have the potential to negatively impact students' mental health, future career, and their rights to dignity and non-discrimination .
Some of the measures applied to address the system's risks and impacts are:
AI Cards for Proctify
Proctify's Machine-Readable Specification
The machine-readable specification of AI Cards relies on the AI Risk Ontology (AIRO) and Vocabulary of AI Risks (VAIR). This machine-readable specification, represented below, can also be accessed here.
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix dcterms: <http://purl.org/dc/terms#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix freq: <http://purl.org/cld/freq/> .
@prefix airo: <https://w3id.org/airo#> .
@prefix vair: <https://w3id.org/vair#> .
@prefix dpv: <https://w3id.org/dpv#> .
@prefix dqv: <http://www.w3.org/ns/dqv#> .
@prefix dcat: <https://www.w3.org/TR/vocab-dcat-3/> .
@prefix ex: <https://example.com/proctify#> .
@base <https://example.com/proctify#> .
ex:proctify
a airo:AISystem ;
airo:hasVersion ex:v_1.0.2 ;
dcterms:date "2023-09-11"^^xsd:date ;
airo:hasModality vair:Software ;
airo:usesTechnique vair:DeepLearning ;
airo:hasInput ex:facial_video ;
airo:producesOutput ex:suspicious_behaviour_alarm ;
airo:isProvidedBy ex:AIEduX ;
airo:isDevelopedBy ex:AIEduX ;
airo:hasComponent ex:facial_analysis_toolkit,
ex:susbehaved_dataset ;
airo:hasModel ex:susbehaved_model ;
airo:isAppliedWithinDomain vair:Education ;
airo:hasPurpose vair:DetectingProhibitedBehaviourDuringTest,
ex:facial_behaviour_analysis,
ex:video_analysis ;
airo:hasCapability ex:facial_recognition ;
airo:isDeployedBy ex:university ;
#airo:hasUseInstruction ;
#airo:hasDeploymentInstruction ;
airo:hasAutomationLevel vair:PartialAutomation ;
airo:hasAISubject ex:student,
ex:other_occupant ;
airo:hasAIUser ex:instructor ;
airo:hasRisk ex:inaccuracy_risk_for_darker_skin ;
airo:hasRiskControl ex:bias_testing ;
dqv:hasQualityMeasurement ex:alarm_precision_measurement, ex:alarm_recall_measurement, ex:alarm_f_score_measurement ;
airo:hasPreDeterminedChange ex:change_of_model ;
airo:compliesWithRegulation ex:EU_GDPR ,
ex:Irish_Data_Protection_Act ;
airo:conformsToStandard vair:ISOIEC42001-2023 ,
vair:ISOIEC27001-2022 ;
airo:followsCodeOfConduct ex:use_of_AI_and_data_in_teaching_and_learning ;
dpv:hasProcessing ex:processing_1 .
ex:EU_GDPR dpv:hasJurisdiction ex:EU .
ex:Irish_Data_Protection_Act dpv:hasJurisdiction ex:Ireland .
ex:facial_analysis_toolkit
a airo:AIComponent ;
airo:hasVersion ex:v_3.3.2 ;
airo:isProvidedBy ex:FACE_research_group ;
airo:hasPurpose ex:extracting_facial_landmark,
ex:extracting_gaze_direction,
ex:extracting_head_pose ;
airo:hasDocumentation .
ex:AIEdux a airo:AIProvider .
ex:susbehaved_model
a airo:AIModel ;
airo:hasVersion ex:v_1.1.2 ;
airo:hasPurpose ex:detecting_suspicious_behaviour,
ex:raising_alarm ;
airo:hasDocumentation .
ex:susbehaved_dataset
a dcat:Dataset ;
airo:hasVersion ex:v_2.0.1 ;
airo:hasPurpose ex:train_model ;
airo:hasDocumentation .
ex:student
a airo:AISubject,
vair:NaturalPerson,
dpv:DataSubject ;
airo:hasHumanInvolvement vair:IntendedInvolvement, vair:ActiveInvolvement , vair:InformedInvolvement ;
airo:hasControlOverAIOutput vair:ChallengeOutput .
ex:other_occupant
a airo:AISubject,
vair:NaturalPerson ;
airo:hasHumanInvolvement vair:UnintendedInvolvement, vair:PassiveInvolvement, vair:UninformedInvolvement ;
airo:hasControlOverAIOutput vair:CannotOptOutOfOutput .
ex:instructor
a airo:AIUser ,
vair:NaturalPerson ;
airo:hasHumanInvolvement vair:IntendedInvolvement, vair:ActiveInvolvement , vair:InformedInvolvement ;
airo:hasControlOverAIOutput vair:CorrectOutput .
ex:processing_1
a dpv:Processing ;
dpv:hasData ex:facial_data ;
dpv:hasLegalBasis dpv:InformedConsent ;
dpv:hasDataSubject ex:student .
ex:facial_data a dpv:SensitivePersonalData ;
dpv:hasDataSource ex:user_input .
ex:inaccuracy_risk_for_darker_skin
a airo:Risk ;
airo:hasLikelihood "Low" ;
airo:hasConsequence ex:raise_of_false_alarms_for_darker_skin .
ex:unrepresentative_dataset
a airo:RiskSource ;
airo:isRiskSourceFor ex:inaccuracy_risk_for_darker_skin ;
airo:hasLikelihood "Medium" .
ex:testing_dataset_for_representativeness
a airo:RiskControl ;
airo:modifiesRiskConcept ex:unrepresentative_dataset .
ex:raise_of_false_alarms_for_darker_skin
a airo:Consequence ;
airo:hasImpact ex:bias_against_students_with_darker_skin ;
airo:hasLikelihood "Low" ;
airo:hasSeverity "Medium" .
ex:bias_against_students_with_darker_skin
a airo:Impact ;
airo:hasImpactOnArea vair:RightToNondiscrimination ;
airo:hasImpactOnEntity ex:student ;
airo:hasLikelihood "Low" ;
airo:hasSeverity "Very_High" .
ex:bias_testing
a airo:RiskControl , vair:TestingMeasure ;
airo:eliminatesRiskConcept ex:bias_against_students_with_darker_skin .
ex:alarm_precision_measurement
a dqv:QualityMeasurement ;
dqv:computedOn ex:proctify ;
dqv:isMeasurementOf ex:alarm_precision ;
dqv:value "98"^^xsd:double .
ex:alarm_precision
a dqv:Metric ;
dqv:expectedDataType xsd:double ;
dqv:inDimension ex:accuracy ;
airo:hasDocumentation .
ex:alarm_recall_measurement
a dqv:QualityMeasurement ;
dqv:computedOn ex:proctify ;
dqv:isMeasurementOf ex:alarm_recall ;
dqv:value "90"^^xsd:double .
ex:alarm_reccall
a dqv:Metric ;
dqv:expectedDataType xsd:double ;
dqv:inDimension ex:accuracy ;
airo:hasDocumentation .
ex:alarm_f_score_measurement
a dqv:QualityMeasurement ;
dqv:computedOn ex:proctify ;
dqv:isMeasurementOf ex:alarm_f_score ;
dqv:value "93"^^xsd:double .
ex:alarm_f_score
a dqv:Metric ;
dqv:expectedDataType xsd:double ;
dqv:inDimension ex:accuracy ;
airo:hasDocumentation .
ex:change_of_model
a airo:Change ;
airo:hasChangedEntity ex:susbehaved_model ;
airo:hasPurpose ex:enhance_fairness ;
airo:hasFrequency freq:bimonthly .
Acknowledgements
The views expressed in this article are purely those of the authors and may not, under any circumstances, be regarded as an official position of the European Commission.
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 813497 (PROTECT ITN), as part of the ADAPT SFI Centre for Digital Media Technology is funded by Science Foundation Ireland through the SFI Research Centres Programme and is co-funded under the European Regional Development Fund (ERDF) through Grant#13/RC/2106_P2