Zeynep Akata

Selected Publications

2020

• Semantically Tied Paired Cycle Consistency for Any-Shot Sketch-based Image Retrieval
  Anjan Dutta, Zeynep Akata
  International Journal of Computer Vision (IJCV), accepted in June 2020
• e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations
  Virginie Do, Oana-Maria Camburu, Zeynep Akata, Thomas Lukasiewicz
  CVPR 2020 Workshop on Fair, Data-Efficient and Trusted Computer Vision
• Learning Robust Representations via Multi-View Information Bottleneck
  Marco Federici, Anjan Dutta, Patrick Forre, Nate Kushmann, Zeynep Akata
  International Conference on Learning Representations (ICLR) 2020
• Evaluating Weakly Supervised Object Localization Methods Right
  Junsuk Choe, Seong Joon Oh, Seungho Lee,  Sanghyuk Chun, Zeynep Akata, Hyunjung Shim
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2020
• Manipulating Attributes of Natural Scenes via Hallucination
  Levent Karacan, Zeynep Akata, Aykut Erdem, Erkut Erdem
  ACM SIGGRAPH 2020 (invited paper)

2019

• Modeling Conceptual Understanding in Image Reference Games
  Rodolfo Corona, Stephan Alaniz, Zeynep Akata
  Neural Information Processing Systems, NeurIPS 2019 (Spotlight)
• Combining Generative and Discriminative Models for Hybrid Inference
  Victor Garcia Satorras, Zeynep Akata, Max Welling
  Neural Information Processing Systems, NeurIPS 2019 (Spotlight)
• Manipulating Attributes of Natural Scenes via Hallucination
  Levent Karacan, Zeynep Akata, Aykut Erdem, Erkut Erdem
  ACM Transactions on Graphics, TOG 2019 (full paper)
• Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval
  Anjan Dutta, Zeynep Akata
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019
• f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning
  Yongqin Xian, Saurabh Sharma, Bernt Schiele, Zeynep Akata
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019
• Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
  Edgar Schoenfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, Zeynep Akata
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019
• SPNet: Semantic Projection Network for Zero-Label and Few-Label Semantic Segmentation
  Yongqin Xian, Subharata Choudhury, Yang He, Bernt Schiele, Zeynep Akata
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019
• Zero-Shot Learning – A Comprehensive Evaluation of the Good, the Bad and the Ugly
  Yongqin Xian, Christoph Lampert, Bernt Schiele, Zeynep Akata
  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2019

2018

• Grounding Visual Explanations
  Lisa Anne Hendricks, Ronghang Hu, Trevor Darrell, Zeynep Akata
  European Conference of Computer Vision, ECCV 2018
• Textual Explanations for Self-Driving Vehicles
  Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, Zeynep Akata
  European Conference of Computer Vision, ECCV 2018
• Multimodal Explanations: Justifying Decisions and Pointing to the Evidence
  Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, Marcus Rohrbach
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018 (Spotlight Presentation)
• Feature Generating Networks for Zero-Shot Learning
  Yongqin Xian, Tobias Lorenz, Bernt Schiele, Zeynep Akata
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018

2017

• Gaze Embeddings for Zero-Shot Image Classification
  Nour Karessli, Zeynep Akata, Bernt Schiele, Andreas Bulling
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017 (Spotlight Presentation)
• Zero-Shot Learning – The Good, the Bad and the Ugly
  Yongqin Xian, Bernt Schiele, Zeynep Akata
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017
• Exploiting saliency for object segmentation from image level labels
  Seong Joon Oh, Rodrigo Benenson, Anna Khoreva, Zeynep Akata, Mario Fritz, Bernt Schiele
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017

2016

• Learning What and Where to Draw
  Scott Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele and Honglak Lee
  Neural Information Processing Systems, NIPS 2016 (Oral Presentation)
• Generating Visual Explanations
  Lisa Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schiele and Trevor Darrell
  European Conference of Computer Vision, ECCV 2016
• Generative Adversarial Text to Image Synthesis
  Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Honglak Lee and Bernt Schiele
  International Conference of Machine Learning, ICML 2016 (Oral Presentation)
• Learning Deep Representations of Fine-Grained Visual Descriptions
  Scott Reed, Zeynep Akata, Honglak Lee and Bernt Schiele
  IEEE Conference of Computer Vision and Patter Recognition, CVPR 2016 (Spotlight Presentation)
• Multi-Cue Zero-Shot Learning with Strong Supervision
  Zeynep Akata, Mateusz Malinowski, Mario Fritz and Bernt Schiele
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 (Spotlight Presentation)
• Latent Embeddings for Zero-Shot Classification
  Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein and Bernt Schiele
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 (Spotlight Presentation)
• Label Embeddings for Image Classification
  Zeynep Akata, Florent Perronnin, Zaid Harchaoui, Cordelia Schmid
  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), Vol:38, No:7, July 2016

2015 and Earlier

• Evaluation of Output Embeddings for Image Classification
  Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee and Bernt Schiele
  IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015
• Label-Embedding for Attribute-Based Classification,
  Zeynep Akata, Florent Perronnin, Zaid Harchaoui and Cordelia Schmid,
  IEEE Computer Vision and Pattern Recognition (CVPR) 2013
• Good Practice in Large-Scale Learning for Image Classification,
  Zeynep Akata, Florent Perronnin, Zaid Harchaoui and Cordelia Schmid,
  IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2014

Research

Clearly explaining a rationale for a classification decision to an end-user can be as important as the decision itself. As decision makers, humans can justify their decisions with natural language and point to the evidence in the visual world which led to their decisions. In contrast, artificially intelligent systems are frequently seen as opaque and are unable to explain their decisions. This is particularly concerning as ultimately such systems fail in building trust with human users. 

Explanations are valuable because they enable users to adapt themselves to the situations that are about to arise while allowing users to attain a stable environment and have the possibility to control it. Explanations in the medical domain can help patients identify and monitor the abnormal behaviour of their ailment. In the domain of self-driving vehicles they can warn the user of some critical state and collaborate with her to prevent a wrong decision. In the domain of satellite imagery, an explanatory monitoring system justifying the evidence of a future hurricane can save millions of lives. Hence, a learning machine that a user can trust and easily operate need to be fashioned with the ability of explanation. 

While deep neural networks lead to impressive successes, e.g. they can now reliably identify 1000 object classes, argue about their interactions through natural language, answer questions about their attributes through interactive dialogues, integrated interpretability is still in its early stages. In other words, we do not know why these deep learning based visual classifications systems work when they are accurate and why they do not work when they make mistakes. Enabling such transparency requires the interplay of different modalities such as images and text, whereas current deep networks are designed as a combination of different tools each optimising a different learning objective with extremely weak and uninterpretable communication channels. However, deep neural networks draw their power from their ability to process large amounts of data in an end-to-end manner through a feedback loop with forward and backward processing. Although interventions on the feedback loop have been implemented by removing neurons and back propagating gradients, a generalizable multi-purpose interpretability is still far from reach. 

Apart from the lack of an integrated interpretability module, deep neural networks require a large amount of labeled training data to reach reliable conclusions. In particular, they need to be trained for every possible situation using labeled data. For instance, the system needs to observe the driver’s behavior at the red light to be able to learn to stop at red light both in a sunny and rainy weather, both in daylight and in night, both in fog and in snow, and so on. This causes a significant overhead in labelling every possible situation. Hence, our aim is to build an explainable machine learning system that can learn the meaning of “red light” and use this knowledge to identify many other related situations, e.g. although red light may look different in darkness vs daylight, the most important aspect in such a situation is to identify that the vehicle needs to stop. In other words, we would like to transfer the explainable behaviour of a decision maker to novel situations.

In summary, we propose an end-to-end trainable decision maker operating in sparse data regime with an integrated interpretability module. Our main research direction to build such a system is two folds: learning representations with weak supervision and generating multimodal explanations of classification decisions. 

Teaching

• Leren (Introduction to Machine Learning), 2017 (BSc AI, Y2, P2)
• Leren (Introduction to Machine Learning), 2018 (BSc AI, Y2, P2)
• Philosophy of AI (Guest Lecture), March 2018
• Computer Vision 2 (Guest Lecture), May 2018
• Back2Basic 2018, Annual Computer Science Event, Amsterdam, Invited Speaker
• DSSV 2018, Conference on Data Science, Statistics and Visualization, Invited speaker
• BMVC 2018, British Machine Vision Conference, Invited Speaker

Students

• 2019-Now Lennart van Goten @ KTH Stockholm (with Prof. Kevin Smith) 
• 2018-Now Marco Federici @ University of Amsterdam (with Dr. Nate Kushman, Dr. Patrick Forre)
• 2017-Now Stephan Alaniz @ University of Tübingen and Max Planck Institute for Informatics
• 2016-Now Yongqin Xian @ Max Planck Institute for Informatics (with Prof. Bernt Schiele)
• 2017-2019 Victor Garcia @ University of Amsterdam (with Prof. Max Welling)
• 2017-2019 Sadaf Gulshad @ University of Amsterdam (with Prof. Arnold Smeulders)
• 2018-2019 Maartje ter Hoeve @ University of Amsterdam (with Prof. Maarten de Rijke)
• 2018-2019 Rodolfo Corona @ UC Berkeley (with Prof. Trevor Darrell)