Saturday, December 24, 2016

Python how to swap two values in one line of code - tuple unpacking

 A common problem we need to solve is how to swap two values
 For example if we have to variables x and y, 
 we might want to swap the
 the values.  

 x = 123
 y = 456
 print('Start values:  a = {} b = {} '.format(x, y))

 temp = x
 x = y
 y = temp
 print('Finish values: a = {} b = {} '.format(x, y))


 A better, more succinct method is to use the cool 'tuple unpacking'

 x = 123
 y = 456
 print('Start values:  a = {} b = {} '.format(x, y))

 x, y = y, x
 print('Finish values: a = {} b = {} '.format(x, y))

Sunday, November 27, 2016

Best way to transfer files to and from Raspberry Pi using Mac

I've been messing about with scp and ftp and now have stumped across a really happy solution for my Mac.

As I use Pycharm to develop my python files it is handy to have a method that uses finder to manage the files. Hard-core folks might be happiest with emacs and scp etc.

So, on the Raspberry Pi. run:
apt-get install netatalk

Then on the Mac in a terminal:

open afp://192.168.1.100 
replace the above ip address with that of your Raspberry Pi.


This will install Apple talk protocol onto the Pi which means that the Pi can appear in the Finder.



Sunday, October 2, 2016

If n people are tested positive for an illness with x probability of accuracy, how confident are we that you are sick?

Yet another way of explaining Bayes's conditional probability.

I found this from Hilary Mason, who in turn heard it from Jake Hofman and Chris Wiggins.

If there are 10,000 people.
1% are sick.
The test has a 99% confidence of accuracy
Given a positive result, what is the probability that you are sick?

 So,
10% of 10,000 is 100 - 100 people test positive. 99% accuracy means that we expect 99 people to be sick and not the whole 100.

But also, of the remaining 10,000 - 100 people, 99% will have tested incorrectly so 1% of that group will be ill. 1% of 9900 = 99.

So in total, we have 99 sick people testing positive
And 99 healthy people being positive.

So, if you test positive you have a 50% chance of being sick.

Sunday, September 11, 2016

How to use my own text in NLTK?

The best explanation I have found of this is in the 3 hour presentation by Benjamin Bengfort.

https://youtu.be/itKNpCPHq3I?list=PLOiJc_waA85o9HpyjRnfsK8slfYRmVICF&t=3220

Click on the link above and it should take you to 53:40 in the video where he talks about how this.

Friday, September 9, 2016

Resources for learning NLTK

Some great resources:

The incredible video series from Harrison.
https://www.youtube.com/watch?v=FLZvOKSCkxY&index=1&list=PLQVvvaa0QuDf2JswnfiGkliBInZnIC4HL
This take you through the theory of NLP with NLTK.

And from the same nice chap,
https://pythonprogramming.net/data-analysis-tutorials/

Another very good lecture:
https://www.youtube.com/watch?v=itKNpCPHq3I

District Data Labs exercises from a workshop.
https://github.com/DistrictDataLabs/intro-to-nltk

A talk on product categorisation
https://www.youtube.com/watch?v=Xg8UtTgziZE

Some useful NLP Python libraries

NLTK is the first port of call for me. It is the core for all NLP stuff. Even production work.

Text blob https://textblob.readthedocs.io/en/dev/
This uses NLTK under the hood and is a useful API for Nltk.

Pattern http://www.clips.ua.ac.be/pattern This is not Python3 yet. It is a web inning module.

Gensimhttp://radimrehurek.com/gensim/ Topic modelling. Analyse plain-text documents for semantic structure. Good for unsupervised or topic modelling.

SciKitLearn
Useful for supervised learning and Classifiers.

MITIEhttps://github.com/mit-nlp/MITIE Library for information extraction. Written in C++ but callable from R, Python, C, ...

SpaCy
A newish project with a good future. Vector models for text only.

I have left out the screen scraping ones like Beautiful Soup and readability (https://pypi.python.org/pypi/readability-lxml )

There are wrappers for the Stanford CoreNLP and also for he Berkley Parser. I don't think that these parsers are free, though I might be wrong. I've not used them. I remember one of them being free for research purposes only.


Sunday, February 7, 2016

Tim Minchin - Tony the fish, to mankind.

https://www.youtube.com/watch?v=UR_Fp09NC_0

And imagine what Tony would think. Standing there on his brand new feet, on brink of the beginnings of mankind as we know it. If he could look forward, just a few, short, hundreds of millions of years, to see one of his decedents, an Israeli Jew by the name of Jesús, having a nail hammered through his feet. (The very feet that Tony provided him with), as a punishment for having a sort of schizophrenic discourse with a god who was created by man to explain the existence of feet in absence of the knowledge of the existence of Tony.

Neural Networks and Deep Learning online book

http://neuralnetworksanddeeplearning.com/index.html

I've recently got into the topic of deep learning having been interested in Neural Nets for some years. Here's a good resource. It's a free book that you can donate if you wish.