What SQL Server Perfmon Counters to look at

There are a ton of Perfmon counters to look at when investigating SQL Server problems. This pdf is a great resource for starting out.

http://www.quest.com/backstage/images/promotions/SQLServer-Perfmonance-Poster.pdf

There is a cool spreadsheet too

http://blogs.msdn.com/b/jimmymay/archive/2008/10/15/perfmon-objects-counters-thresholds-utilities-for-sql-server.aspx

SQl Server Diagnostics scripts

Glenn Berry's SQL Server Performance

http://sqlserverperformance.wordpress.com/2013/07/09/sql-server-diagnostic-information-queries-for-july-2013/

Download and go. Pretty darned good.

SQL Server 2005 Waits and Queues

Really, really useful. Everywhere I go the same problems exist. SQL Server slow and no one knows why. It's a useful topic and this is a good part of it.

SQL Server 2005 Waits and Queues - SQL Server Best Practices Article, by Tom Davidson, 2005

It can be downloaded from http://bit.ly/aUh6S


"wait time" - the total time taken for a thread to go from Running, to being "put on the queue", or to being "queued, and then back to Running again.

"resource wait time" - the time a thread is waiting on the Waiter List with the state of Suspended.

"signal wait time" - time waiting for the processor once the resource has become available.

So wait time = resource wait time + signal wait time

When a thread has server time, and then becomes blocked, it is set as Suspended and then placed on the Waiter list.

When it receives a signal to say the resource is available, it is set to Runnable and it is place on the Runnable Queue. It will wait here until it is given to the Processor.

sys.dm_os_waiting_tasksDMV
Will show all threads that are currently suspended.

Machine Learning in Python

I was looking for info on Stocastic Gradient Decent algorithm and saw this:

Scikit

http://scikit-learn.org/stable/index.html

There is a good 45 min video http://scikit-learn.org/stable/presentations.html (the skit-learn video by Jake Vanderplas) where the url he talks about is wrong I think he means http://www.astroml.org/sklearn_tutorial/

I've since been searching around and found the following that look interesting too:

PyBrain

http://www.pybrain.org

mlpy

http://mlpy.sourceforge.net
Seems pretty extensive

milk

http://pythonhosted.org/milk/

Still early days

Calling Prolog from Python

This is my shopping list of things to investigate. It's a notes page of where I am at the moment in the process of connecting Python and Prolog. I hope it might help someone who is trying to do the same thing, or perhaps thinking about trying to do the same thing. If you have any ideas then it would be great to hear from you. At the moment I am thinking that programming in C++ and using SWI-Prolog's C++ interface might be the answer.

I've been looking into this lately and have collected these  things to look at.
I can program in Prolog and like it. I'm not great at it but I love the way of thinking. I have been using Python also, not that I particularly like Python, but it's so useful and appears in so many places. (yes I've been plating with the Raspberry Pi). So, can I get the Pi to use prolog as it's decision making engine?

pyswip

https://code.google.com/p/pyswip/
This is a bridge between SWI-Prolog and Python and it seems to be the first place to visit. It works on Linux and Win32 but not sure about OSX

I looked at this first and thought that it had stopped being developed. I'm not sure. I think there was an update in December 2012.

There seems to be a nice post on someone's expereince from 2011 here: http://ryepdx.com/2011/09/prolog-in-python-pt-1/

Picstus

This is an interface between Sicstus Prolog and Python. I probably wont look at this as I am using SWI-Prolog for the time being. Sicstus costs about 165 Euros and I'm not that professional.
http://www.biolab.si/picstus/picstus.html

A Prolog Interpreter in Python

This is very interesting stuff. Excelent stuff. http://wwwold.stups.uni-duesseldorf.de/thesis/Bolz2007-Bachelorarbeit.pdf but probably it's not going to help me.
[UPDATE 2016] The above link no longer works. Try this http://stups.hhu.de/w/A_Prolog_Interpreter_in_Python

There is also some work by Chris Meyers http://www.openbookproject.net/py4fun/ a few links in this page. In summary though it's too slow for a real world solution.

SWIG

"SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages." So We could wrap Prolog with Swig.

http://www.swig.org/

Probably not what I want to do as a bit complex.

Pwig

This is a swig extension for Python.

http://pwig.sourceforge.net
The last activity was in 2004.
Can download from here:
http://sourceforge.net/projects/pwig/files/pwig/
SWIG is at version 2.0 but PWIG required SWIG 1.3.23 which I couldn't see. It might be there somewhere.

PyKE

Now this is interesting. http://pyke.sourceforge.net/ 

It introduces a form of logic programming into Python. There is an interesting paper at http://pyke.sourceforge.net/PyCon2008-paper.html 

It has an inference engine but the syntax is new. I suppose it would do but it's a shame not to improve my skills in Prolog. That said though, the skill in prolog is the thinking not the syntax.

PyProlog

A Python extension embedding SWI-Prolog. This was last updated in 2001 so is a tad out of date with very little or no information.

http://pyprolog.sourceforge.net/

Pyrolog

I'm not sure about this one. A prolog interpreter written in RPython.
https://bitbucket.org/cfbolz/pyrolog

PyLog

http://wiki.python.org/moin/PyLog

Well I'm not sure about this. It's a first order logic library. It also

Knowrob

KnowRob is a knowledge processing system that combines knowledge representation and reasoning methods with techniques for acquiring knowledge and for grounding the knowledge in a physical system and can serve as a common semantic framework for integrating information from different sources.

Too big for my needs here but really cool.

http://www.knowrob.org/knowrob

So now what do I think?

Well I'm beginning to think I should look at coding in C or C++.
My task is for the Raspberry Pi and it's a shame to slow it up by getting into C again after so many years.  Python wold be such a fast and rapid development. Maybe I create a C interface to my Prolog. Then use Python for all other motor controlling, sensor controlling. Then, when it's a success and become a millionaire, I can refactor all the code from Python to C for the fun of it.
This is a disappointment. I'll do some more digging and write my thoughts here.

Can I instal prolog on the Raspberry Pi

Yep.

sudo apt-get install swi-prolog

Is numpy included on the Raspberry Pi

Yes it is.
My Raspberry came with Raspbian Wheezy installed.

At the command line I started python,

then entered:

>>> from numpy import *
>>> a = arange(15).reshape(3, 5)
>>> a

 Which gave:

array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])
>>> 

How to select the Python.3 interpreter in Eclipse on a Mac

This foxed me for a bit.
I have Pydev installed, and Python 2.n and all is well. But I needed Python 3.3

Install 3.3

So go to Python: http://www.python.org/download/ and download the installer.

Then install as usual on a Mac.

Now check that it's in place.
There should be a folder in Applications/Python 3.3

There should also
Macintosh HD - Library/Frameworks/Python.framework/Versions/3.3

Select the interpreter.

So now, select the interpreter in Pydev.

1. You go to Eclipse > Preferences > PyDev > Interpreter - Python
2. From here you, Select New, 
3. in Interpreter Name you enter python3 and in Interpreter Executable you enter /usr/local/bin/python3 or /usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/bin/python3.3.
4.  OK

Lego Nxt plug connector for Raspberry Pi

I haven't done this yet but I'm getting ready to. That is, connect the Raspberry Pi to the Lego NXT Mindstorm sensors.


I've a circuit schematic, but there's also some good info on how to adapt a regular telephone plug to use a NXT socket.
http://www.philohome.com/nxtplug/nxtplug.htm

You'll need to cut and file a bit.

How to connect the socket on the Pi for a NXT cable.

I haven't actually tested this yet. Maybe next weekend :)
My advice would be to use this as a starting off point and experiment.

How and why to document Business rules (and Use cases)

Business rules are, I think, not given enough consideration in the projects I come up against. I'm a bit of an itinerant member of smallish teams and regularly see teams short on resource and time. Often I hear people speak of "business rules",  and nearly every project I've been on I've not seen anyone documents them.
Perhaps they don't need documenting. I think they do.

In my most recent contract one project had an incredible amount of Business rules. The task of the project was to replace an old system and set of accompanying processes with a new set. Most of the Business rules were to be in an "expert system" to do with qualifications, grades and occupations but there were many outside of that. The expert system was techie and geekie and obtuse but it documented the rules via the application's UI so they were taken care of, almost by accident. The rest were stumbled upon, discussed in emails by people who didn't know for sure, and either forgotten or if we were lucky, a code change was made to support it. Either way the rule was lost either in code or the fog of the project. And this continued for years.

The project team was under-strength and spread over three locations, business analysis was inexperienced and absent for some months, and the project was far more complex than anyone really expected. For my sins I came into the project after two years and immediately struggled. After my first eight months the first major phase was working. Which is ok I suppose. I'm only saying this as an example. It's an example in the general sense and to a lesser or greater degree it is common.

The general picture of the development and project process in this example would be to say that some of the developers wanted a Scrum approach but other developers distrusted this and management certainly distrusted this. So Scrum was replaced by a version of nothing in particular. This is pretty common too. So what could have happened at the beginning to have made it all smoother?
The objective for the process at the beginning of the project was to let the Business knew what they were letting themselves in for as the foundations of the project were being laid and details being discovered.
What is it that we are really going to be building?
What changes is this thing going to make to the business processes, to the people, to the productivity, to the service? Oh and how many people will be needed to get this all done and how long will we have to wait?

The way not to do this is to get inexperienced people to document the requirements and blame them a few years later if it goes titsup. What not to do after this is to get all excited and give it to the developers, stand back, and wait for the first delivery.

The Scrum thing will not work here. What we should do is think about the business processes. What is it that the Business is doing and how will we change that. There are ways to document this and be all professional (BPMN see bpmn.org). But if you are without a BA a quick google can help. Something is better than nothing. A flow chart, a list of processes in MS Word is a great way to frame the problem domain. The Business Domain.

But this post is about BusinessRules. So where do they come in?

For me they come in alongside the Use Cases, or User Stories. Quite often a User Case or Story can be a Business Rule. For example, A Business Rule could be that we send a summary of trades to the Ombudsman within 20 days. This could also be a Story, As Compliance Manager, I must send the summary of trades to the Ombudsman within 20 days of the request.

However, it may be slightly different. Perhaps the story / requirement will be for a configurable date or alerting system.

Also, Business rules may define the data being held on a customer, and when it must be deleted for data-protection regulation purposes.

For me, the huge benefit of use cases is the fact that they describle, in easy terms, how a lump of work is done. A process. A small process. That is the business and that is what out work is going to support. Starting with declarative technical requirements such a the "the Save button must confirm the user is saved in the database before saving the data." is so abstract and difficult to communicate to a non-technical business user in a project meeting. They must be good for the development team a bit later on, maybe in a work unit, a task detail, but not in a 100 page requirements document.

How to start with Business rules

I always create a document with a few sentences explaining why the document exists, and then I put a list of Business Rules, and then Use cases.

The only important rules I apply to the writing of business rules is:

The rule is not dependent on nor refers to any technology or system. The rule should be equally valid if the business was being run using a pencil and paper.

Rules should be grouped into easily understandable and relevant groups. And each Business Rule should have three fields:

1. An ID (can be a number or a letter and a number). The letter can relate to the title of the group of the rules. If  a rule is in a group entitled Refund Business Rules, then number the rules R1, R2, R3...
2. Business rule name, one to 4 words in length
3. A short description of the rule. 

Rules can be things like:

• what bits of data are mandatory, 
• what bits of data are to be removed.
• Trading days,
• Compliance issues.
• Regulatory points.
• Welfare issues
• ...