I am working on building progressively larger neocortical models with NUPIC and doing research into deep learning.
Two things I have learnt this week from research and experimentation:
- With deep networks you don’t have to necessarily feed all of the data into the bottom of the network, in fact it’s better if you dont. I noticed this when I was reading papers about dissection of the human neocortex, layers 4 and 5 connect almost entirely vertically where as layers 1,2,3 connect horizontally more. This makes sense because you can present the neocortex with different spatial patterns for higher / lower SDR resolution - dont mix your data types in the same SDR, encode them differently and feed them into the higher parts of the network. Example: one might encode an SDR to an image using black / white pixels and then set another SDR to “dog” and “not dog” for classification purposes. Feed the classification SDR to the top of the network and the raw data to the bottom … this gave me stability much faster because the classification was not “mixed” up with the data.
Deeper networks mean more context. This sounds like a ‘duh’ that’s obvious, but if your neocortical models are performing badly, try adding 1 more layer, the number of connections increases exponentially
I am working on building progressively larger neocortical models with NUPIC and doing research into deep learning.
Two things I have learnt this week from research and experimentation: - With deep networks you don’t have to necessarily feed all of the data into the bottom of the network, in fact it’s better if you dont. I noticed this when I was reading papers about dissection of the human neocortex, layers 4 and 5 connect almost entirely vertically where as layers 1,2,3 connect horizontally more. This makes sense because you can present the neocortex with different spatial patterns for higher / lower SDR resolution - dont mix your data types in the same SDR, encode them differently and feed them into the higher parts of the network. Example: one might encode an SDR to an image using black / white pixels and then set another SDR to “dog” and “not dog” for classification purposes. Feed the classification SDR to the top of the network and the raw data to the bottom … this gave me stability much faster because the classification was not “mixed” up with the data.