My master project at UCL. Active learning tries to solve a practical problem for machine learning, which is to create a good model with only limited labelling budget. In this project, we exploit useful properties from semi-supervised generative model and use them in active learning. Our experiments in the half-moon and MNIST dataset show that by using semi-supervised generative model with simple acquisition function such as predictive entropy, we are able to improve the performance of active learning. Further experiments on our proposed acquisition functions expose interesting challenges in using data density provided by the model, which can be a valuable pointer for future active learning research.
You Need Only Uncertain Answers: Data Efficient Multilingual Question Answering
Zhihao Lyu, Danier Duolikun, Bowei Dai, Yuan Yao, Pasquale Minervini, Tim Z. Xiao, and Yarin Gal
ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning, 2020
I am an advisor for this project. Data scarcity is a major barrier for multilingual question answering: current systems work well with languages such as English where data is affluent, but face challenges with small corpora. As data labelling is expensive, previous works have resorted to pre-tuning systems on larger multilingual corpora followed by fine-tuning on the smaller ones. Instead of curating and labelling large corpora, we demonstrate a data efficient multi-lingual question answering system which only selects uncertain questions for labelling, reducing labelling efforts and costs. To realise this Bayesian active learning framework, we develop methodology to quantify uncertainty in several state-of-art attention-based Transfer question answering models. We then propose an uncertainty measure based on the variance of BLEU scores, and computed via Monte Carlo Dropout, to detect out-of-distribution questions. We finish by showing the effectiveness of our uncertainty measures in various out-of-distribution question answering settings.
Wat zei je? Detecting Out-of-Distribution Translations with Variational Transformers
My master project at Oxford. In the project, we detect out-of-training-distribution sentences in Neural Machine Translation using the Bayesian Deep Learning equivalent of Transformer models. For this we develop a new measure of uncertainty designed specifically for long sequences of discrete random variables – i.e. words in the output sentence. Our new measure of uncertainty solves a major intractability in the naive application of existing approaches on long sentences. We use our new measure on a Transformer model trained with dropout approximate inference. On the task of German-English translation using WMT13 and Europarl, we show that with dropout uncertainty our measure is able to identify when Dutch source sentences, sentences which use the same word types as German, are given to the model instead of German.
A partial reconfiguration controller for Altera Stratix V FPGAs
Zhenzhong Xiao, Dirk Koch, and Mikel Lujan
In International Conference on Field Programmable Logic and Applications (FPL), 2016