Error starting Rabbitmq on MacOS

After having problems with my Homebrew I had to reinstall a few things. A month or so later I found I had forgotten to reinstall RabbitMq so I did the usual:

brew install rabbitmq

I went to:

cd /usr/local/sbin

Then I started rabbitmq:

./rabbitmq-server 

which resulted in an error:

JMBP:sbin johnroberts$ ./rabbitmq-server 

  ##  ##

  ##  ##      RabbitMQ 3.7.14. Copyright (C) 2007-2019 Pivotal Software, Inc.

  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/

  ######  ##

  ##########  Logs: /usr/local/var/log/rabbitmq/rabbit@localhost.log

                    /usr/local/var/log/rabbitmq/rabbit@localhost_upgrade.log

              Starting broker...

{"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_exchange,[name,type,durable,auto_delete,internal,arguments,scratches,policy,operator_policy,decorators,options],[name,type,durable,auto_delete,internal,arguments]}]},{rabbit,start,[normal,[]]}}}"}

Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{schema_integrity_check_failed,[{table_attributes_mismatch,rabbit_exchange,[name,type,durable,auto_delete,internal,ar

Crash dump is being written to: /usr/local/var/log/rabbitmq/erl_crash.dump...done

Looking at $PATH

echo $PATH

I noticed rabbitmq was not in there so I added it.

Still no good!

I then noticed that the error referred to a table and that the error log file referred to a database directory:

 database dir   : /usr/local/var/lib/rabbitmq/mnesia/rabbit@localhost

I had a hunch that removing rabbitmq the first time probably didn't remove the database directory and the new install was probably incompatible with what was in there.

So I uninstalled rabbitmq using

 brew uninstall rabbitmq

and noticed the db directory was still there.
So I removed the directory.

Then,
Reinstalled rabbitmq.
Started it.
And jolly good news.

  ##  ##

  ##  ##      RabbitMQ 3.7.14. Copyright (C) 2007-2019 Pivotal Software, Inc.

  ##########  Licensed under the MPL.  See https://www.rabbitmq.com/

  ######  ##

  ##########  Logs: /usr/local/var/log/rabbitmq/rabbit@localhost.log

                    /usr/local/var/log/rabbitmq/rabbit@localhost_upgrade.log

              Starting broker...

 completed with 7 plugins.

thanks

Comprehending Monads by Philp Wadler. Probably the first time Monads could be used to structure programs

If you're interested in the origins and why's of the Functional Programming idea then this is an interesting read.  In the 1960's monads were invented by Category theorists. In the 1970's Functional programmers invented list comprehensions. This paper shows how the two come together.

https://ncatlab.org/nlab/files/WadlerMonads.pdf

Why Functional Programming Matters, John Hughes The University, Glasgow

A really interesting paper by John Hughes though really quite old now.

As software becomes more and more complex, it is more and more important to structure it well. Well-structured software is easy to write and to debug, and provides a collection of modules that can be reused to reduce future programming costs. In this paper we show that two fea- tures of functional languages in particular, higher-order functions and lazy evaluation, can contribute significantly to modularity. As examples, we manipulate lists and trees, program several numerical algorithms, and implement the alpha-beta heuristic (an algorithm from Artificial Intelligence used in game-playing programs). We conclude that since modularity is the key to successful programming, functional programming offers important advantages for software development.

https://www.cs.kent.ac.uk/people/staff/dat/miranda/whyfp90.pdf

Matplotlib.pyplot not working on Mac in virtualenv

I had a very curious problem where python would not find pyplot in matplotlib.

I was using:

• python3
• virtualenv
• matplotlib

To create the virtual environment I used: 

virtualenv -p python3 <myproject name>

However, when I used:

python3 -m venv <myproject name>

It worked ok.

An explanation was found here:
https://matplotlib.org/faq/osx_framework.html

How to update a console line without writing a new line in python3

import time
import sys
print('Update the same line in console output')

sys.stdout.write('Doing task a\r')
sys.stdout.flush()
time.sleep(1)
sys.stdout.write('Doing task b\r')
sys.stdout.flush()
time.sleep(1)
sys.stdout.write('Doing task c\n')
time.sleep(1)
print('done')

Python how to swap two values in one line of code - tuple unpacking

 A common problem we need to solve is how to swap two values

 For example if we have to variables x and y, 

 we might want to swap the

 the values.  

 x = 123

 y = 456

 print('Start values:  a = {} b = {} '.format(x, y))

 temp = x

 x = y

 y = temp

 print('Finish values: a = {} b = {} '.format(x, y))

 A better, more succinct method is to use the cool 'tuple unpacking'

 x = 123

 y = 456

 print('Start values:  a = {} b = {} '.format(x, y))

 x, y = y, x

 print('Finish values: a = {} b = {} '.format(x, y))

Best way to transfer files to and from Raspberry Pi using Mac

I've been messing about with scp and ftp and now have stumped across a really happy solution for my Mac.

As I use Pycharm to develop my python files it is handy to have a method that uses finder to manage the files. Hard-core folks might be happiest with emacs and scp etc.

So, on the Raspberry Pi. run:

apt-get install netatalk

Then on the Mac in a terminal:

open afp://192.168.1.100 

replace the above ip address with that of your Raspberry Pi.

This will install Apple talk protocol onto the Pi which means that the Pi can appear in the Finder.

If n people are tested positive for an illness with x probability of accuracy, how confident are we that you are sick?

Yet another way of explaining Bayes's conditional probability.

I found this from Hilary Mason, who in turn heard it from Jake Hofman and Chris Wiggins.

If there are 10,000 people.
1% are sick.
The test has a 99% confidence of accuracy
Given a positive result, what is the probability that you are sick?

 So,
10% of 10,000 is 100 - 100 people test positive. 99% accuracy means that we expect 99 people to be sick and not the whole 100.

But also, of the remaining 10,000 - 100 people, 99% will have tested incorrectly so 1% of that group will be ill. 1% of 9900 = 99.

So in total, we have 99 sick people testing positive
And 99 healthy people being positive.

So, if you test positive you have a 50% chance of being sick.

How to use my own text in NLTK?

The best explanation I have found of this is in the 3 hour presentation by Benjamin Bengfort.

https://youtu.be/itKNpCPHq3I?list=PLOiJc_waA85o9HpyjRnfsK8slfYRmVICF&t=3220

Click on the link above and it should take you to 53:40 in the video where he talks about how this.

Resources for learning NLTK

Some great resources:

The incredible video series from Harrison.
https://www.youtube.com/watch?v=FLZvOKSCkxY&index=1&list=PLQVvvaa0QuDf2JswnfiGkliBInZnIC4HL
This take you through the theory of NLP with NLTK.

And from the same nice chap,
https://pythonprogramming.net/data-analysis-tutorials/

Another very good lecture:
https://www.youtube.com/watch?v=itKNpCPHq3I

District Data Labs exercises from a workshop.
https://github.com/DistrictDataLabs/intro-to-nltk

A talk on product categorisation
https://www.youtube.com/watch?v=Xg8UtTgziZE

Some useful NLP Python libraries

NLTK is the first port of call for me. It is the core for all NLP stuff. Even production work.

Text blob https://textblob.readthedocs.io/en/dev/
This uses NLTK under the hood and is a useful API for Nltk.

Pattern http://www.clips.ua.ac.be/pattern This is not Python3 yet. It is a web inning module.

Gensim - http://radimrehurek.com/gensim/ Topic modelling. Analyse plain-text documents for semantic structure. Good for unsupervised or topic modelling.

SciKitLearn
Useful for supervised learning and Classifiers.

MITIE - https://github.com/mit-nlp/MITIE Library for information extraction. Written in C++ but callable from R, Python, C, ...

SpaCy
A newish project with a good future. Vector models for text only.

I have left out the screen scraping ones like Beautiful Soup and readability (https://pypi.python.org/pypi/readability-lxml )

There are wrappers for the Stanford CoreNLP and also for he Berkley Parser. I don't think that these parsers are free, though I might be wrong. I've not used them. I remember one of them being free for research purposes only.

Tim Minchin - Tony the fish, to mankind.

https://www.youtube.com/watch?v=UR_Fp09NC_0

And imagine what Tony would think. Standing there on his brand new feet, on brink of the beginnings of mankind as we know it. If he could look forward, just a few, short, hundreds of millions of years, to see one of his decedents, an Israeli Jew by the name of Jesús, having a nail hammered through his feet. (The very feet that Tony provided him with), as a punishment for having a sort of schizophrenic discourse with a god who was created by man to explain the existence of feet in absence of the knowledge of the existence of Tony.

Neural Networks and Deep Learning online book

http://neuralnetworksanddeeplearning.com/index.html

I've recently got into the topic of deep learning having been interested in Neural Nets for some years. Here's a good resource. It's a free book that you can donate if you wish.

How to get π Pi in OSX (use option-p)

It can be really hard to find the Pi character in the Special Character search in OSX. It it buried in the Greek alphabet in the Maths Symbols.

Option-P will print it.

⌥ - P

How to view the Special characters in Xcode. How to get Maths characters

I'm not sure if it has recently changed, but in Xcode when you select Edit -> Emoji & Symbols you get (I think by default) this smaller Special Chars tool.
To get access to the more familiar view with a search select the icon in the top right.



The view I am more familiar with with a search and math category.

Closures syntax simplification in Swift

I think the process of simplification for the closure syntax is quite interesting

This is an initial way of passing a function as an argument, sometime called a delegate function. The example is a simply multiplication of two numbers.

func doSomething(operation: (Double, Double) -> Double) {
    theReturnValue = operation( 2, 3)
}

func multiply(op1: Double, op2: Double) -> Double {
    return op1 * op2
}
// The main calling of the method
DoSomething(multiply)

The first step that Swift makes to simplify this is to make the multiply function an inline function, a lambda or Closure. We place the code inline, use the "in" keyword and move the curly brace.

func doSomething(operation: (Double, Double) -> Double) {
    theReturnValue = operation( 2, 3)
}

func multiply(op1: Double, op2: Double) -> Double {
    return op1 * op2
}// The main calling of the method
DoSomething( { op1: Double, op2: Double) -> Double in
   return op1 * op2
}) 


But is tidied up further because Swift, like C# in this situation, is very good with type inference.  So we can remove the type declaration of the arguments and return type because it already knows them.

func doSomething(operation: (Double, Double) -> Double) {
    theReturnValue = operation( 2, 3)
}
// The main calling of the method
DoSomething({ (op1:, op2:) -> in
    return op1 * op2
}) 

Now we can tidy this up a little, by putting the return on the same line as the declaration
, but also remove the return keyword because we are returning an expression and it knows we are going to return a Double.

func doSomething(operation: (Double, Double) -> Double) {
    theReturnValue = operation( 2, 3)
}
// The main calling of the method
DoSomething({ (op1:, op2:) -> in op1 * op2 })


Cool. But Swift doesn't need you to name the arguments as it will use $0 and $1 for the first two args if names are not provided. So we can write:


func doSomething(operation: (Double, Double) -> Double) {
    theReturnValue = operation( 2, 3)
}
// The main calling of the method
DoSomething({ $0 * $1 })


One last thing. The last argument can be moved outside of the parenthesis.

func doSomething(operation: (Double, Double) -> Double) {
    theReturnValue = operation( 2, 3)
}
// The main calling of the method
DoSomething() { $0 * $1 }

Other arguments would go inside the parenthesis of DoSomething(...), but in this case there are no other args so we can remove the parenthesis entirely.

func doSomething(operation: (Double, Double) -> Double) {
    theReturnValue = operation( 2, 3)
}
// The main calling of the method
DoSomething { $0 * $1 }

Quite interesting.

How to remove backups from Time Machine

We have a few Macs in the house and a few Time Machine devices setup. As a number of Macs share a Time Machine disk I have found that one Macs' backups can monopolise the disk through one reason or another which prevents another mac from doing a backup because of lack of space.
I could have managed this better by partitioning the disk and that's something I'll do in future, but for the time being I wanted to remove some old unwanted backups to free up some space.
This can take a very long time.

To remove a specific backup I used Time Machine Utility:

sudo tmutil delete /Volumes/drive_name/Backups.backupdb/mac_name/YYYY-MM-DD-hhmmss

I compacted the remaining sparse file by using

sudo hdiutil compact [path and name of sparsefile]


You can list the backups by using:

tmutil list backups

Unavailable

If you see the time machine is unavailable then you probably have it mounted and this will prevent you from deleting files. Unmount (eject) and try again.

tmutil docs

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man8/tmutil.8.html

hdiutil docs

https://developer.apple.com/library/mac/documentation/Darwin/Reference/ManPages/man1/hdiutil.1.html

Python Virtualenv on OSX

Install virtualenv

Johns-MBP:~ johnroberts$ pip install virtualenv

Johns-MBP:~ johnroberts$ mkdir -p ~/Virtualenvs

Johns-MBP:~ johnroberts$ mkdir -p ~/Projects

Johns-MBP:~ johnroberts$ cd ~/Virtualenvs/

Create the virtual environments

Johns-MBP:~ johnroberts$ virtualenv foobar

New python executable in foobar/bin/python2.7

Also creating executable in foobar/bin/python

Installing setuptools, pip, wheel...done.

Johns-MBP:~ johnroberts$ virtualenv -p python3 foobar-py3

Running virtualenv with interpreter /usr/local/bin/python3

Using base prefix '/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5'

New python executable in foobar-py3/bin/python3.5

Also creating executable in foobar-py3/bin/python

Installing setuptools, pip, wheel...done.

This should get Virtualenv in place.

Activate an environment

We made the Python environment, now we need to activate it.

Johns-MBP:~ johnroberts$ source foobar/bin/activate

(foobar)Johns-MBP:~ johnroberts$ 

Notice the "(foobar)" which shows that we are now in the environment.

What version of python are we using

We can check which version of python

(foobar)Johns-MBP:~ johnroberts$ which python

/Users/johnroberts/foobar/bin/python

This shows that we are using python from our environment.

(foobar)Johns-MBP:~ johnroberts$ python --version

Python 2.7.10

We can also check pip

(foobar)Johns-MBP:~ johnroberts$ which pip

/Users/johnroberts/foobar/bin/pip

Use pip to see what packages are here

(foobar)Johns-MBP:~ johnroberts$ pip list

pip (7.1.2)

setuptools (18.2)

wheel (0.24.0)

If I did pip list outside of this environment I'd get a lot of packages (I won't list them here)

Install the packages for this environment

(foobar)Johns-MBP:~ johnroberts$ pip install numpy

Collecting numpy

  Downloading numpy-1.10.2-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (3.7MB)

    100% |████████████████████████████████| 3.7MB 143kB/s 

Installing collected packages: numpy

Successfully installed numpy-1.10.2

Cool as a cucumber.

List of Dependencies

Make a list of dependencies for this project and view it using cat

(foobar)Johns-MBP:~ johnroberts$ pip freeze --local > requirements.txt

(foobar)Johns-MBP:~ johnroberts$ cat requirements.txt 

numpy==1.10.2

wheel==0.24.0

The list of dependencies is very useful for recreating other environments using pip. Create the environment and then use pip and the requirements.txt file to install the dependencies.

    pip install -r requirements.txt

Get out of the environment - deactivate

We just type deactivate

(foobar)Johns-MBP:~ johnroberts$ deactivate

Johns-MBP:~ johnroberts$ 

Notice that "(foobar)" has been removed. We are no longer in that environment.

Remove the Virtual Environment
Once deactivated, we can remove:

Johns-MBP:~ johnroberts$ rm -rf foobar/

Simple.

Latest Python on El Capitan

El Capitan OSX comes with Python 2.7 but you might like to get the latest version and ensure you keep it updated.

I use Homebrew which was installed before I updated Python so I won't write how to install it.
Find http://brew.sh

GCC

You will need the latest GCC compiler. For that you can instal Xcode. If you instal a fresh version on Xcode you might need to run the following to install the command line tools:

xcode-select --install

Python 2.7

Run the following:
brew install python

pip all be installed by brew

Python 3

Run the following:
 brew install python3

Virtualenv

If you're running two versions of Python on the same system you may benefit from looking at virtualenv

Installing numpy and scipy for OSX - using Homebrew

The place I found for instructions on how best to do this, and it has been edited recently, is
https://joernhees.de/blog/2014/02/25/scientific-python-on-mac-os-x-10-9-with-homebrew/

I have copied the script from Jörn's page to here for my personal reference. But all credit is Jörn's 
I did need X11 installed and I thought this was possible via Homebrew but I had a problem so I resorted to downloading it from Quartz page. It took a while. It used to be distributed by Apple but no longer.

Notice that some files are downloaded from pip and not brew.

Also, I installed the brew install Caskroom/cask/mactex so that I could run matpltlib with latex fonts and formulas. This does take a while to download. Not sure why.

Possible errors when installing pip pyquery and lxml

You might find compile errors fatal error: 'libxml/xmlversion.h' file not found

You can google solutions to this and this link has a few approaches to solve the problem:

http://stackoverflow.com/questions/19548011/cannot-install-lxml-on-mac-os-x-10-9

# install PIL, imagemagick, graphviz and other
# image generating stuff
brew install libtiff libjpeg webp little-cms2
pip install Pillow
brew install imagemagick --with-fftw --with-librsvg --with-x11
brew install graphviz --with-librsvg --with-x11
brew install cairo
brew install py2cairo # this will ask you to download xquartz and install it
brew install qt pyqt

# install virtualenv, nose (unittests & doctests on steroids)
pip install virtualenv
pip install nose

# install numpy and scipy
# there are two ways to install numpy and scipy now: via pip or via brew.
# PICK ONE, i prefer pip for proper virtualenv support and more up-to-date versions.
pip install numpy
pip install scipy
# OR:
# (if you want to run numpy and scipy with openblas also remove comments below:)
#brew install openblas
brew install numpy # --with-openblas
brew install scipy # --with-openblas

# test the numpy & scipy install
python -c 'import numpy ; numpy.test();'
python -c 'import scipy ; scipy.test();'

# some cool python libs (if you don't know them, look them up)
# matplotlib: generate plots
# pandas: time series stuff
# nltk: natural language toolkit
# sympy: symbolic maths in python
# q: fancy debugging output
# snakeviz: cool visualization of profiling output (aka what's taking so long?)
#brew install Caskroom/cask/mactex  # if you want to install matplotlib with tex support and don't have mactex installed already
brew install matplotlib --with-cairo --with-tex  # cairo: png ps pdf svg filetypes, tex: tex fonts & formula in plots
pip install pandas
pip install nltk
pip install sympy
pip install q
pip install snakeviz

# ipython with parallel and notebook support
brew install zmq
pip install ipython[all]

# html stuff (parsing)
pip install html5lib cssselect pyquery lxml BeautifulSoup

# webapps / apis (choose what you like)
pip install Flask Django tornado

# semantic web stuff: rdf & sparql
pip install rdflib SPARQLWrapper

# graphs (graph metrics, social network analysis, layouting)
pip install networkx
brew install graph-tool

# maintenance: updating pip libs
pip install pip-tools  # you'll then have a pip-review command, see Updating section below