Hi Lobsters! I’m one of the authors of this resource.
Adversarial machine learning is a relatively new but rapidly developing field.
It’s easy to see why people are excited about this research area: ML systems
are being increasingly deployed in the real world, and yet, they’re very easy
to fool with maliciously perturbed inputs. There have been dozens of proposed
attacks and hundreds of proposed defenses against malicious inputs to machine
learning systems. To help researchers keep up with developments in this field,
we created this community-run reference for state-of-the-art adversarial
example defenses.
Unlike most subfields of ML, security is a negative goal: the goal is to
produce a machine learning system that can’t be fooled. Showing that a system
can’t be fooled is really hard.
Measuring progress in traditional machine learning can be done through a
monotonically increasing objective: if a paper increases accuracy on a given
benchmark from 94% to 95%, progress has been made. Future papers may improve on
the benchmark, but accuracy will not decrease. In contrast, measuring progress
in adversarial machine learning is exceptionally difficult. By definition, the
metric used to measure accuracy on a given defense is success on the best
attack (that respects the threat model), which may not exist at the time of
publication. This is why future third-party analyses of defense techniques are
important.
robust-ml.org lists current defenses along with analyses of the defenses,
making it easy to get a complete picture of which techniques have been shown to
be broken and which techniques currently seems to be working.
“Adversarial machine learning is a relatively new”
I haven’t gotten into this topic yet. The descriptions of what it’s about are pretty exciting given outsiders have worried about the security of ML approaches. Far as new, I wonder if you all would count work like Danny Hillis’ use of adversarial co-evolution? In his work on sorting algorithms, he kept changing the tests to be harder to break the algorithms. They were like parasites in the metaphors. The results over his prior method without co-evolution were pretty impressive.
Hillis’ stuff was always one of my favorite stories in that space. I guess I’m just curious if that kind of thing was any inspiration to your field, if you all classify it as a technique in your field, and/or if the field still uses methods like that?I’m also curious if there’s been any general-purpose methods so far in the new research that you think can get interesting results on cheap hardware. What should I tell people at smaller, local colleges to look into that they could do on their desktops or otherwise on a budget?
Hillis’s work on adversarial co-evolution seems more similar to Generative
Adversarial Networks than adversarial examples / robustness / machine learning
security. Some subset of ML researchers group together GANs and adversarial
examples under the label “Adversarial ML”, but many other researchers think of
them as distinct research areas.
I’m not sure if Hillis’s work / similar efforts were an inspiration for
GAN-based methods. I don’t think it was an inspiration for research related to
ML security.
What’s neat about this research area, especially on the attack side, is that
you don’t need that much compute. For example, all the work I’ve done on attacks can be done with a single high-end GPU, and reproducing some of the
results on a slightly smaller scale can even be done on a laptop CPU (e.g. see
this blog
post).
Hi Lobsters! I’m one of the authors of this resource.
Adversarial machine learning is a relatively new but rapidly developing field. It’s easy to see why people are excited about this research area: ML systems are being increasingly deployed in the real world, and yet, they’re very easy to fool with maliciously perturbed inputs. There have been dozens of proposed attacks and hundreds of proposed defenses against malicious inputs to machine learning systems. To help researchers keep up with developments in this field, we created this community-run reference for state-of-the-art adversarial example defenses.
Unlike most subfields of ML, security is a negative goal: the goal is to produce a machine learning system that can’t be fooled. Showing that a system can’t be fooled is really hard.
Measuring progress in traditional machine learning can be done through a monotonically increasing objective: if a paper increases accuracy on a given benchmark from 94% to 95%, progress has been made. Future papers may improve on the benchmark, but accuracy will not decrease. In contrast, measuring progress in adversarial machine learning is exceptionally difficult. By definition, the metric used to measure accuracy on a given defense is success on the best attack (that respects the threat model), which may not exist at the time of publication. This is why future third-party analyses of defense techniques are important.
robust-ml.org lists current defenses along with analyses of the defenses, making it easy to get a complete picture of which techniques have been shown to be broken and which techniques currently seems to be working.
Cool work! Thanks for posting it.
“Adversarial machine learning is a relatively new”
I haven’t gotten into this topic yet. The descriptions of what it’s about are pretty exciting given outsiders have worried about the security of ML approaches. Far as new, I wonder if you all would count work like Danny Hillis’ use of adversarial co-evolution? In his work on sorting algorithms, he kept changing the tests to be harder to break the algorithms. They were like parasites in the metaphors. The results over his prior method without co-evolution were pretty impressive.
Hillis’ stuff was always one of my favorite stories in that space. I guess I’m just curious if that kind of thing was any inspiration to your field, if you all classify it as a technique in your field, and/or if the field still uses methods like that?I’m also curious if there’s been any general-purpose methods so far in the new research that you think can get interesting results on cheap hardware. What should I tell people at smaller, local colleges to look into that they could do on their desktops or otherwise on a budget?
Hillis’s work on adversarial co-evolution seems more similar to Generative Adversarial Networks than adversarial examples / robustness / machine learning security. Some subset of ML researchers group together GANs and adversarial examples under the label “Adversarial ML”, but many other researchers think of them as distinct research areas.
I’m not sure if Hillis’s work / similar efforts were an inspiration for GAN-based methods. I don’t think it was an inspiration for research related to ML security.
What’s neat about this research area, especially on the attack side, is that you don’t need that much compute. For example, all the work I’ve done on attacks can be done with a single high-end GPU, and reproducing some of the results on a slightly smaller scale can even be done on a laptop CPU (e.g. see this blog post).
That’s neat. Good that one can get results on a budget. I’ll keep the link saved for any students that might be interested.