1. 14
  1.  

  2. 1

    …any derivative or gradient, of any function you can program, or of any program that computes a function…

    Well, technically automatic differentiation doesn’t applies to “any-any” function but to any function you specify as a series of compositions (and similar operations) of math functions with known derivative. sin(1+x^2) works. A recursively specified factorial wouldn’t work out of the box (but could work if you used gamma or whatever).

    It is one of those “look how simple and beautiful this is - here’s a long page of caveats” things.

    1. 2

      A recursively specified factorial wouldn’t work out of the box

      Sure it would.

      import Numeric.AD
      
      fac :: (Num a, Eq a) => a -> a
      fac = go 1 where
        go acc 0 = acc
        go acc n = go (n * acc) (n - 1)
      
      dfac :: (Num a, Eq a) => a -> a
      dfac = diff fac
      
      fac' :: (Eq a, Num a) => a -> a
      fac' 0 = 1
      fac' n = n * fac' (n - 1)
      
      dfac' :: (Eq a, Num a) => a -> a
      dfac' = diff fac'
      
      fac'' :: (Enum a, Num a) => a -> a
      fac'' n = product [1..n]
      
      dfac'' :: (Enum a, Num a) => a -> a
      dfac'' = diff fac''
      
      1. 1

        Just for kicks: this is the most abstract factorial definition I could come up with, and it’s still perfectly amenable to AD.

        {-# LANGUAGE DeriveFunctor #-}
        {-# LANGUAGE FlexibleInstances #-}
        {-# LANGUAGE FunctionalDependencies #-}
        {-# LANGUAGE MultiParamTypeClasses #-}
        
        import Control.Arrow ((&&&))
        import Numeric.AD
        
        newtype Fix f = Fix (f (Fix f))
        
        class Functor f => Fixpoint f t | t -> f where
          inF  :: f t -> t
          outF :: t -> f t
        
        para :: Fixpoint f t => (f (a, t) -> a) -> t -> a
        para alg = alg . fmap (para alg &&& id) . outF
        
        data NatF r =
            ZeroF
          | SuccF r
          deriving Functor
        
        instance (Ord a, Num a) => Fixpoint NatF a where
          inF ZeroF = 0
          inF (SuccF n) = n + 1
          outF n
            | n > 0     = SuccF (n - 1)
            | otherwise = ZeroF
        
        type Nat = Fix NatF
        
        fac :: (Ord a, Num a) => a -> a
        fac = para alg where
          alg ZeroF = 1
          alg (SuccF (f, n)) = f * (n + 1)
        
        dfac :: (Ord a, Num a) => a -> a
        dfac = diff fac
        
        1. 1

          Can you explain how this computes the derivative?

          What function does it result in? My naive interpretation of

          fac' :: (Eq a, Num a) => a -> a
          fac' 0 = 1
          fac' n = n * fac' (n - 1)
          

          is that it gives the derivative of n! as n! and I don’t think that’s right. at least it wouldn’t conform to the gamma function [1], which I think is the only analytic function taking n! for it’s positive values.

          Maybe I’m missing something.

          [1] https://en.wikipedia.org/wiki/Gamma_function