Matplotlib.pyplot not working on Mac in virtualenv

I had a very curious problem where python would not find pyplot in matplotlib.

I was using:

• python3
• virtualenv
• matplotlib

To create the virtual environment I used: 

virtualenv -p python3 <myproject name>

However, when I used:

python3 -m venv <myproject name>

It worked ok.

An explanation was found here:
https://matplotlib.org/faq/osx_framework.html

How to update a console line without writing a new line in python3

import time
import sys
print('Update the same line in console output')

sys.stdout.write('Doing task a\r')
sys.stdout.flush()
time.sleep(1)
sys.stdout.write('Doing task b\r')
sys.stdout.flush()
time.sleep(1)
sys.stdout.write('Doing task c\n')
time.sleep(1)
print('done')

Python how to swap two values in one line of code - tuple unpacking

 A common problem we need to solve is how to swap two values

 For example if we have to variables x and y, 

 we might want to swap the

 the values.  

 x = 123

 y = 456

 print('Start values:  a = {} b = {} '.format(x, y))

 temp = x

 x = y

 y = temp

 print('Finish values: a = {} b = {} '.format(x, y))

 A better, more succinct method is to use the cool 'tuple unpacking'

 x = 123

 y = 456

 print('Start values:  a = {} b = {} '.format(x, y))

 x, y = y, x

 print('Finish values: a = {} b = {} '.format(x, y))

How to use my own text in NLTK?

The best explanation I have found of this is in the 3 hour presentation by Benjamin Bengfort.

https://youtu.be/itKNpCPHq3I?list=PLOiJc_waA85o9HpyjRnfsK8slfYRmVICF&t=3220

Click on the link above and it should take you to 53:40 in the video where he talks about how this.

Resources for learning NLTK

Some great resources:

The incredible video series from Harrison.
https://www.youtube.com/watch?v=FLZvOKSCkxY&index=1&list=PLQVvvaa0QuDf2JswnfiGkliBInZnIC4HL
This take you through the theory of NLP with NLTK.

And from the same nice chap,
https://pythonprogramming.net/data-analysis-tutorials/

Another very good lecture:
https://www.youtube.com/watch?v=itKNpCPHq3I

District Data Labs exercises from a workshop.
https://github.com/DistrictDataLabs/intro-to-nltk

A talk on product categorisation
https://www.youtube.com/watch?v=Xg8UtTgziZE

Some useful NLP Python libraries

NLTK is the first port of call for me. It is the core for all NLP stuff. Even production work.

Text blob https://textblob.readthedocs.io/en/dev/
This uses NLTK under the hood and is a useful API for Nltk.

Pattern http://www.clips.ua.ac.be/pattern This is not Python3 yet. It is a web inning module.

Gensim - http://radimrehurek.com/gensim/ Topic modelling. Analyse plain-text documents for semantic structure. Good for unsupervised or topic modelling.

SciKitLearn
Useful for supervised learning and Classifiers.

MITIE - https://github.com/mit-nlp/MITIE Library for information extraction. Written in C++ but callable from R, Python, C, ...

SpaCy
A newish project with a good future. Vector models for text only.

I have left out the screen scraping ones like Beautiful Soup and readability (https://pypi.python.org/pypi/readability-lxml )

There are wrappers for the Stanford CoreNLP and also for he Berkley Parser. I don't think that these parsers are free, though I might be wrong. I've not used them. I remember one of them being free for research purposes only.

Python Virtualenv on OSX

Install virtualenv

Johns-MBP:~ johnroberts$ pip install virtualenv

Johns-MBP:~ johnroberts$ mkdir -p ~/Virtualenvs

Johns-MBP:~ johnroberts$ mkdir -p ~/Projects

Johns-MBP:~ johnroberts$ cd ~/Virtualenvs/

Create the virtual environments

Johns-MBP:~ johnroberts$ virtualenv foobar

New python executable in foobar/bin/python2.7

Also creating executable in foobar/bin/python

Installing setuptools, pip, wheel...done.

Johns-MBP:~ johnroberts$ virtualenv -p python3 foobar-py3

Running virtualenv with interpreter /usr/local/bin/python3

Using base prefix '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5'

New python executable in foobar-py3/bin/python3.5

Also creating executable in foobar-py3/bin/python

Installing setuptools, pip, wheel...done.

This should get Virtualenv in place.

Activate an environment

We made the Python environment, now we need to activate it.

Johns-MBP:~ johnroberts$ source foobar/bin/activate

(foobar)Johns-MBP:~ johnroberts$ 

Notice the "(foobar)" which shows that we are now in the environment.

What version of python are we using

We can check which version of python

(foobar)Johns-MBP:~ johnroberts$ which python

/Users/johnroberts/foobar/bin/python

This shows that we are using python from our environment.

(foobar)Johns-MBP:~ johnroberts$ python --version

Python 2.7.10

We can also check pip

(foobar)Johns-MBP:~ johnroberts$ which pip

/Users/johnroberts/foobar/bin/pip

Use pip to see what packages are here

(foobar)Johns-MBP:~ johnroberts$ pip list

pip (7.1.2)

setuptools (18.2)

wheel (0.24.0)

If I did pip list outside of this environment I'd get a lot of packages (I won't list them here)

Install the packages for this environment

(foobar)Johns-MBP:~ johnroberts$ pip install numpy

Collecting numpy

  Downloading numpy-1.10.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (3.7MB)

    100% |████████████████████████████████| 3.7MB 143kB/s 

Installing collected packages: numpy

Successfully installed numpy-1.10.2

Cool as a cucumber.

List of Dependencies

Make a list of dependencies for this project and view it using cat

(foobar)Johns-MBP:~ johnroberts$ pip freeze --local > requirements.txt

(foobar)Johns-MBP:~ johnroberts$ cat requirements.txt 

numpy==1.10.2

wheel==0.24.0

The list of dependencies is very useful for recreating other environments using pip. Create the environment and then use pip and the requirements.txt file to install the dependencies.

    pip install -r requirements.txt

Get out of the environment - deactivate

We just type deactivate

(foobar)Johns-MBP:~ johnroberts$ deactivate

Johns-MBP:~ johnroberts$ 

Notice that "(foobar)" has been removed. We are no longer in that environment.

Remove the Virtual Environment
Once deactivated, we can remove:

Johns-MBP:~ johnroberts$ rm -rf foobar/

Simple.

Latest Python on El Capitan

El Capitan OSX comes with Python 2.7 but you might like to get the latest version and ensure you keep it updated.

I use Homebrew which was installed before I updated Python so I won't write how to install it.
Find http://brew.sh

GCC

You will need the latest GCC compiler. For that you can instal Xcode. If you instal a fresh version on Xcode you might need to run the following to install the command line tools:

xcode-select --install

Python 2.7

Run the following:
brew install python

pip all be installed by brew

Python 3

Run the following:
 brew install python3

Virtualenv

If you're running two versions of Python on the same system you may benefit from looking at virtualenv

Installing numpy and scipy for OSX - using Homebrew

The place I found for instructions on how best to do this, and it has been edited recently, is
https://joernhees.de/blog/2014/02/25/scientific-python-on-mac-os-x-10-9-with-homebrew/

I have copied the script from Jörn's page to here for my personal reference. But all credit is Jörn's 
I did need X11 installed and I thought this was possible via Homebrew but I had a problem so I resorted to downloading it from Quartz page. It took a while. It used to be distributed by Apple but no longer.

Notice that some files are downloaded from pip and not brew.

Also, I installed the brew install Caskroom/cask/mactex so that I could run matpltlib with latex fonts and formulas. This does take a while to download. Not sure why.

Possible errors when installing pip pyquery and lxml

You might find compile errors fatal error: 'libxml/xmlversion.h' file not found

You can google solutions to this and this link has a few approaches to solve the problem:

http://stackoverflow.com/questions/19548011/cannot-install-lxml-on-mac-os-x-10-9

# install PIL, imagemagick, graphviz and other
# image generating stuff
brew install libtiff libjpeg webp little-cms2
pip install Pillow
brew install imagemagick --with-fftw --with-librsvg --with-x11
brew install graphviz --with-librsvg --with-x11
brew install cairo
brew install py2cairo # this will ask you to download xquartz and install it
brew install qt pyqt

# install virtualenv, nose (unittests & doctests on steroids)
pip install virtualenv
pip install nose

# install numpy and scipy
# there are two ways to install numpy and scipy now: via pip or via brew.
# PICK ONE, i prefer pip for proper virtualenv support and more up-to-date versions.
pip install numpy
pip install scipy
# OR:
# (if you want to run numpy and scipy with openblas also remove comments below:)
#brew install openblas
brew install numpy # --with-openblas
brew install scipy # --with-openblas

# test the numpy & scipy install
python -c 'import numpy ; numpy.test();'
python -c 'import scipy ; scipy.test();'

# some cool python libs (if you don't know them, look them up)
# matplotlib: generate plots
# pandas: time series stuff
# nltk: natural language toolkit
# sympy: symbolic maths in python
# q: fancy debugging output
# snakeviz: cool visualization of profiling output (aka what's taking so long?)
#brew install Caskroom/cask/mactex  # if you want to install matplotlib with tex support and don't have mactex installed already
brew install matplotlib --with-cairo --with-tex  # cairo: png ps pdf svg filetypes, tex: tex fonts & formula in plots
pip install pandas
pip install nltk
pip install sympy
pip install q
pip install snakeviz

# ipython with parallel and notebook support
brew install zmq
pip install ipython[all]

# html stuff (parsing)
pip install html5lib cssselect pyquery lxml BeautifulSoup

# webapps / apis (choose what you like)
pip install Flask Django tornado

# semantic web stuff: rdf & sparql
pip install rdflib SPARQLWrapper

# graphs (graph metrics, social network analysis, layouting)
pip install networkx
brew install graph-tool

# maintenance: updating pip libs
pip install pip-tools  # you'll then have a pip-review command, see Updating section below

Machine Learning in Python

I was looking for info on Stocastic Gradient Decent algorithm and saw this:

Scikit

http://scikit-learn.org/stable/index.html

There is a good 45 min video http://scikit-learn.org/stable/presentations.html (the skit-learn video by Jake Vanderplas) where the url he talks about is wrong I think he means http://www.astroml.org/sklearn_tutorial/

I've since been searching around and found the following that look interesting too:

PyBrain

http://www.pybrain.org

mlpy

http://mlpy.sourceforge.net
Seems pretty extensive

milk

http://pythonhosted.org/milk/

Still early days

Calling Prolog from Python

This is my shopping list of things to investigate. It's a notes page of where I am at the moment in the process of connecting Python and Prolog. I hope it might help someone who is trying to do the same thing, or perhaps thinking about trying to do the same thing. If you have any ideas then it would be great to hear from you. At the moment I am thinking that programming in C++ and using SWI-Prolog's C++ interface might be the answer.

I've been looking into this lately and have collected these  things to look at.
I can program in Prolog and like it. I'm not great at it but I love the way of thinking. I have been using Python also, not that I particularly like Python, but it's so useful and appears in so many places. (yes I've been plating with the Raspberry Pi). So, can I get the Pi to use prolog as it's decision making engine?

pyswip

https://code.google.com/p/pyswip/
This is a bridge between SWI-Prolog and Python and it seems to be the first place to visit. It works on Linux and Win32 but not sure about OSX

I looked at this first and thought that it had stopped being developed. I'm not sure. I think there was an update in December 2012.

There seems to be a nice post on someone's expereince from 2011 here: http://ryepdx.com/2011/09/prolog-in-python-pt-1/

Picstus

This is an interface between Sicstus Prolog and Python. I probably wont look at this as I am using SWI-Prolog for the time being. Sicstus costs about 165 Euros and I'm not that professional.
http://www.biolab.si/picstus/picstus.html

A Prolog Interpreter in Python

This is very interesting stuff. Excelent stuff. http://wwwold.stups.uni-duesseldorf.de/thesis/Bolz2007-Bachelorarbeit.pdf but probably it's not going to help me.
[UPDATE 2016] The above link no longer works. Try this http://stups.hhu.de/w/A_Prolog_Interpreter_in_Python

There is also some work by Chris Meyers http://www.openbookproject.net/py4fun/ a few links in this page. In summary though it's too slow for a real world solution.

SWIG

"SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages." So We could wrap Prolog with Swig.

http://www.swig.org/

Probably not what I want to do as a bit complex.

Pwig

This is a swig extension for Python.

http://pwig.sourceforge.net
The last activity was in 2004.
Can download from here:
http://sourceforge.net/projects/pwig/files/pwig/
SWIG is at version 2.0 but PWIG required SWIG 1.3.23 which I couldn't see. It might be there somewhere.

PyKE

Now this is interesting. http://pyke.sourceforge.net/ 

It introduces a form of logic programming into Python. There is an interesting paper at http://pyke.sourceforge.net/PyCon2008-paper.html 

It has an inference engine but the syntax is new. I suppose it would do but it's a shame not to improve my skills in Prolog. That said though, the skill in prolog is the thinking not the syntax.

PyProlog

A Python extension embedding SWI-Prolog. This was last updated in 2001 so is a tad out of date with very little or no information.

http://pyprolog.sourceforge.net/

Pyrolog

I'm not sure about this one. A prolog interpreter written in RPython.
https://bitbucket.org/cfbolz/pyrolog

PyLog

http://wiki.python.org/moin/PyLog

Well I'm not sure about this. It's a first order logic library. It also

Knowrob

KnowRob is a knowledge processing system that combines knowledge representation and reasoning methods with techniques for acquiring knowledge and for grounding the knowledge in a physical system and can serve as a common semantic framework for integrating information from different sources.

Too big for my needs here but really cool.

http://www.knowrob.org/knowrob

So now what do I think?

Well I'm beginning to think I should look at coding in C or C++.
My task is for the Raspberry Pi and it's a shame to slow it up by getting into C again after so many years.  Python wold be such a fast and rapid development. Maybe I create a C interface to my Prolog. Then use Python for all other motor controlling, sensor controlling. Then, when it's a success and become a millionaire, I can refactor all the code from Python to C for the fun of it.
This is a disappointment. I'll do some more digging and write my thoughts here.

Is numpy included on the Raspberry Pi

Yes it is.
My Raspberry came with Raspbian Wheezy installed.

At the command line I started python,

then entered:

>>> from numpy import *
>>> a = arange(15).reshape(3, 5)
>>> a

 Which gave:

array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])
>>> 

How to select the Python.3 interpreter in Eclipse on a Mac

This foxed me for a bit.
I have Pydev installed, and Python 2.n and all is well. But I needed Python 3.3

Install 3.3

So go to Python: http://www.python.org/download/ and download the installer.

Then install as usual on a Mac.

Now check that it's in place.
There should be a folder in Applications/Python 3.3

There should also
Macintosh HD - Library/Frameworks/Python.framework/Versions/3.3

Select the interpreter.

So now, select the interpreter in Pydev.

1. You go to Eclipse > Preferences > PyDev > Interpreter - Python
2. From here you, Select New, 
3. in Interpreter Name you enter python3 and in Interpreter Executable you enter /usr/local/bin/python3 or /usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/bin/python3.3.
4.  OK