…any derivative or gradient, of any function you can program, or of any program that computes a function…

Well, technically automatic differentiation doesn’t applies to “any-any” function but to any function you specify as a series of compositions (and similar operations) of math functions with known derivative. sin(1+x^2) works. A recursively specified factorial wouldn’t work out of the box (but could work if you used gamma or whatever).

It is one of those “look how simple and beautiful this is - here’s a long page of caveats” things.

A recursively specified factorial wouldnâ€™t work out of the box

Sure it would.

import Numeric.AD
fac :: (Num a, Eq a) => a -> a
fac = go 1 where
go acc 0 = acc
go acc n = go (n * acc) (n - 1)
dfac :: (Num a, Eq a) => a -> a
dfac = diff fac
fac' :: (Eq a, Num a) => a -> a
fac' 0 = 1
fac' n = n * fac' (n - 1)
dfac' :: (Eq a, Num a) => a -> a
dfac' = diff fac'
fac'' :: (Enum a, Num a) => a -> a
fac'' n = product [1..n]
dfac'' :: (Enum a, Num a) => a -> a
dfac'' = diff fac''

Just for kicks: this is the most abstract factorial definition I could come up with, and it’s still perfectly amenable to AD.

{-# LANGUAGE DeriveFunctor #-}
{-# LANGUAGE FlexibleInstances #-}
{-# LANGUAGE FunctionalDependencies #-}
{-# LANGUAGE MultiParamTypeClasses #-}
import Control.Arrow ((&&&))
import Numeric.AD
newtype Fix f = Fix (f (Fix f))
class Functor f => Fixpoint f t | t -> f where
inF :: f t -> t
outF :: t -> f t
para :: Fixpoint f t => (f (a, t) -> a) -> t -> a
para alg = alg . fmap (para alg &&& id) . outF
data NatF r =
ZeroF
| SuccF r
deriving Functor
instance (Ord a, Num a) => Fixpoint NatF a where
inF ZeroF = 0
inF (SuccF n) = n + 1
outF n
| n > 0 = SuccF (n - 1)
| otherwise = ZeroF
type Nat = Fix NatF
fac :: (Ord a, Num a) => a -> a
fac = para alg where
alg ZeroF = 1
alg (SuccF (f, n)) = f * (n + 1)
dfac :: (Ord a, Num a) => a -> a
dfac = diff fac

What function does it result in? My naive interpretation of

fac' :: (Eq a, Num a) => a -> a
fac' 0 = 1
fac' n = n * fac' (n - 1)

is that it gives the derivative of n! as n! and I don’t think that’s right. at least it wouldn’t conform to the gamma function [1], which I think is the only analytic function taking n! for it’s positive values.

Well, technically automatic differentiation doesn’t applies to “any-any” function but to any function you specify as a series of compositions (and similar operations) of math functions with known derivative. sin(1+x^2) works. A recursively specified factorial wouldn’t work out of the box (but could work if you used gamma or whatever).

It is one of those “look how simple and beautiful this is - here’s a long page of caveats” things.

Sure it would.

Just for kicks: this is the most abstract factorial definition I could come up with, and it’s still perfectly amenable to AD.

Can you explain how this computes the derivative?

What function does it result in? My naive interpretation of

is that it gives the derivative of n! as n! and I don’t think that’s right. at least it wouldn’t conform to the gamma function [1], which I think is the only analytic function taking n! for it’s positive values.

Maybe I’m missing something.

[1] https://en.wikipedia.org/wiki/Gamma_function