Note to self - how to do many to many in Entity Framework code-first

I've probably done a handful of EF code-first projects and I always forget how to do the mappings. I always end up looking at old projects and try to figure out what went on. This is a good place to make notes. In my last project I had no idea what I was looking at. Old age has crept up.

I won't go over the whole DbContext Model classes thing,  but in case someone is reading this that isn't familiar, you make a your model classes for your application and they become mapped to your database entities in the Entity Framework ORM world.

So, for example say we have these two model classes amongst our many other model classes:

• WatchingList
• Thing

We would then create our DbContext class by creating a class and implementing from DbContext class. For example:

public class MyProjectDbContext : DbContext
{
   // in here we will map our model objects with DbSet<>
   public DbSet<WatchingList> WatchingLists { get; set;}
   public DbSet<Thing> Things { get; set;}

     ... and so on...
}

The above is where all the magic happens. Look elsewhere for a fuller explanation, but in short when we run our app after having done the above, we can then use LINQ to query the database objects, the database will be created for us if it doesn't already exist.

On another note, if it does already exist then we can configure our project to recreate the database on changes. I'm not going to write about that but it's great do developing though probably not great for production systems :)

Many to Many mapping

Anyway, to do the many to many mappings we'll need a

modelBuilder.Entity<WatchingList>()
      .HasMany((w => w.Things).WithMany()
      .Map(m => m.ToTable("WatchingList_Thing")
      .MapRightKey("ThingId")
      .MapLeftKey("WatchingListId"));

Basically,  ThingId and WatchingListId are the columns in the mapping table and the mapping table will be called  WatchingList_Thing

Abstract base classes

If we have some inheritance with these three model objects say,

• Animal - maybe an abstract base class for the other two concrete classes listed here
• Dog - inherits/extends the Animal class
• Cat - inherits/extends the Animal class

If we have these three model classes declared with DbSet in the usual way we will get three tables with those names. We might want to have the table names in the database slightly different to reflect their relationship. We can do that by also overriding the OnModelCreating() method.

protected override void OnModelCreating(DbModelBuilder modelBuilder)

{

   modelBuilder.Entity<Animal>().ToTable("Animal");

   modelBuilder.Entity<Dog>().ToTable("Animal_Dog");

   modelBuilder.Entity<Cat>().ToTable("Animal_Cat");

}

So we've controlled the naming of tables by EF.

XSRF Cross Site Request Forgery attack - ASP.NET MVC

This is a note on the topic that I've been meaning to write for a very long time. I'm a bit old, been around a bit and have worked with a whole range of people. The interesting thing about the XSRF is that I hardly hear it mentioned in the work place. I hope other folk have a better experience. But we just seem to be getting on with the list of functional requirements in time for the next deadline. Perhaps that why it's not talked about enough but it is probably the most 'popular' security vector being exploited by the nasty people.

A quick intro.

In simple terms:

1. A friendly person A visits our happy site Happy.com and logs in. 
2. Happy.com site authenticates him and gives him a cookie so that every time he visits us we know he's a good guy.
3. Unknowingly, friendly guy A visits a bad site Evil.com.
4. Evil site Evil.com will sneakily return a malicious form tag that will request to our Happy.com site. This malicious code will be allowed to access our server because it is attached to the authentication cookie we gave him and we think that it has already been authenticated.
5. Bad things can happen.

This works because the authentication cookie is being transported with the request. 

The only way to prevent this is to ensure the form is created by the good user. We use an antiForgery token to help here. 

In the View we put the anti forgery helper:

<% using (Html.BeginForm()) {%>

    <% = Html.AntiForgeryToken() %>

In the controller action we use the ValidateAntiForgeryToken attribute:

[ValidateAntiForgeryToken]

[AcceptVerbs(HttpVerbs.Post)]

public ActionResult Edit(int id, 

With this asp.net creates a crypto token that can't be created by the Evil user.

Peter Doolittle talk on "working memory".

Working memory is the part of our consciousness that we are aware of.
1. Allows us to store immediate experiences and a little bit of knowledge
2. Allows us to reach into our long term memory and use it to achieve our current goal

We can remember about 4 things in Working Memory. It has a very limited capacity and we'll forget those things very quickly, seconds, unless we do something with them. We need to process them as they happens.

Working memory Capacity has a long history and is related to some positive effects. People that have relatively high Working memory capacity tend to be good story tellers and tend to be able to reason at high levels.

We need to use Working Memory efficiently to handle the stream of life that comes at us to make sense of it and to achieve our goals. Working memory is really quite small and this talk sheds some light on this interesting architecture. That's how I view it - as an architecture. I think the RAM, cache, disk storage solution in computing is a kind of metaphor for this. But, did our understanding of computer architecture allow us to pin this solution on our mind when actually we are just approximating the solution. Anyway, it's a nice talk.

http://www.ted.com/talks/peter_doolittle_how_your_working_memory_makes_sense_of_the_world.html

External WD 1 TB will not connect (mount) on a Mac

I have had this problem for a while where the external WD drive doesn't want to connect or be visible on the Mac. I thought it was the cable and fiddled about with that but no. Occasionally it would connect but it didn't follow a pattern, or didn't seem to. I am using the drive as a Time Machine backup.

I may not have fixed it entirely but I can now get it to mount and indeed it has mounted correctly ever since - a few times.

I went to the Disk Utility app and the 1 TB WD drive was being found. It was in the left-hand pane as  shown in the image below.

The below image shows the drive in its working state.
In it's error state it was displaying the text: disk1s3

I went to the Terminal and ran the command:

diskutil list

This provided me with the following:

John-Mac-mini:Volumes John$ diskutil list

/dev/disk0

   #:                       TYPE NAME                    SIZE       IDENTIFIER

   0:      GUID_partition_scheme                        *500.1 GB   disk0

   1:                        EFI EFI                     209.7 MB   disk0s1

   2:                  Apple_HFS Macintosh HD            499.2 GB   disk0s2

   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3

/dev/disk1

   #:                       TYPE NAME                    SIZE       IDENTIFIER

   0:     Apple_partition_scheme                        *1.0 TB     disk1

   1:        Apple_partition_map                         32.3 KB    disk1s1

   2:                  Apple_HFS                         1.0 TB     disk1s3

I then mounted the drive:

John$ diskutil mount disk1s3

I noticed immediately that the drive mounted on the desktop and in the DiskUtility app.

Note that the disk identifier may be different for your setup. If you are unsure, look at the disk Utility app where I have drawn the red rectangle.

As I say, I haven't fixed this because at the next power off it may refuse to connect again.

When running distil list again the drive partition is given a name ' My Book Time Machine' as expected. I am not sure if this name is only available after mounting, and therefore it is correctly blank before mounting, or whether that is a part of the problem.

/dev/disk1

   #:                       TYPE NAME                    SIZE       IDENTIFIER

   0:     Apple_partition_scheme                        *1.0 TB     disk1

   1:        Apple_partition_map                         32.3 KB    disk1s1

   2:                  Apple_HFS My Book Time Machine    1.0 TB     disk1s3

The INSERT key on a Mac is '0' on the number pad

Yep. It is.

Useful for a number of things. If you're on a MAC, with Windows running in a Virtual, and you're running vSphere to log into an ESXi Windows VM, ... (take a breath) ... then you'll need to enter a CTRL + ALT + DELETE.

Which is actually, Ctrl + alt + Insert, which is in fact, Ctrl + alt + 0.

(That will make sense if you need to do it, it's just garbled nonsense if you're not trying to do it)

1. You are on a Mac.
2. You have VirtualBox
3. In VirtualBox you have a Windows virtual machine
4. You are running the Vshpere application in the Windows machine.
5. vSphere is connecting to a Windows VM on another computer (Windows #2).
6. You will need to issue the CTRL+ALT+DELETE command to that other Windows (Windows #2) to start it.
7. vSphere will accept CTRL+ALT+INSERT, however because we're on a Mac, we'll need to use CTRL+ALT+0

Reverend Thomas Bayes lived in Tunbridge Wells - now there's a thing

The other night I was in Tunbridge Wells happily wandering along in the dark autumn drizzle. The damp orange street light was skimming off the cold black road surfaces. I was off to see Norman Watt-Roy again. He's great.
Quite by chance I noticed Bayes' plaque on the gate post.

A couple of my earlier posts:

Another very simple way of looking at Bayes theorem

The very best Bayes explanation? Actually I don't think so now. I re-read it a few months back and it was confusing.

Virtual Box error with OSX Maverick

I just knew it would happen. I knew that upgrading to Maverick would trouble my VirtualBox. And so it was.

This is my third post on VirtualBox. I really think I should have learnt something by now. Will I never learn!

The first thing I thought I'd check was the version. The latest is 4.2.18. The version I was running was  4.1.18. So I should upgrade. That fills me with dread also.

First though I thought I'd test the Ubuntu VM I had. It worked. So that meant that VirtualBox will work with Ubuntu on Maverick, but it will not work with Windows 7 on Maverick. I'm not sure if this is good news or bad.

Uninstall the new VirtualBox, unintall the old one

Run the instal package, and when presented with the following window, run the script (see bottom right of the window).

But, you may get this error when running the VirtualBox_Uninstall.tool

So try this if you feel brave. It may be that your OSX is protecting us from Developers not identified by the App Store.
Set the Allow apps... from Anywhere. I would suggest you set this back to something more secure once finished with it.

The run the uninstall again. More errors in the Terminal Windows?

Sigh!

Restart OSX. Do not start VirtualBox. Retry the uninstall.

Note:

• When the script asks you for Yes/No, type Yes.
• When you enter your password during there is no feedback. Type the password in carefully and hit return.

There will be one feedback on the screen that looks like errors. They may not be. I can't remember what they were, something like "...forgotten". You can ignore them.

Once the uninstall script has run, then install and go.

Good luck.

Let me know if this was of any use. Thanks

How to password protect a Pages file on iCloud

It took me awhile to figure out how to do this it so I thought I'd post it.

This isn't special for iCloud usage of course, it's for the Pages file in general which can be stored locally or on iCloud. It's very handy for accessing a sensitive doc from home, work or on the go with the phone.

Anyway, it's very easy. The feature is tucked away in the Inspector. See below.

Drawing class diagrams or flow diagrams in ascii characters

I've always wanted this, and I have often thought about writing it. But I needn't.
This is great.

Drawing flow diagrams, or class diagrams in ascii.

http://www.asciiflow.com/#Draw



+------------+
| Comparison |
+------------+            e.g. Apple     e.g. Pear    e.g. Grape
                          +---------+   +---------+   +---------+
                          | Thing   |   | Thing   |   | Thing   |
                          |         |   |         |   |         |
                          +---------+   +---------+   +---------+

  +---------------+       +----------+  +----------+  +----------+
  |ComparisonPoint|       |Calculated|  |Calculated|  |Calculated|
  +---------------+       | Value    |  | Value    |  | Value    |
                          +----------+  +----------+  +----------+

  +---------------+       +----------+  +----------+  +----------+
  |ComparisonPoint|       |Calculated|  |Calculated|  |Calculated|
  +---------------+       | Value    |  | Value    |  | Value    |
                          +----------+  +----------+  +----------+

  +---------------+       +----------+  +----------+  +----------+
  |ComparisonPoint|       |Calculated|  |Calculated|  |Calculated|
  +---------------+       | Value    |  | Value    |  | Value    |
                          +----------+  +----------+  +----------+

                          +----------+  +----------+  +----------+
                          |Comparison|  |Comparison|  |Comparison|
                          | result   |  | result   |  | result   |
                          +----------+  +----------+  +----------+







         Strategy Pattern
        -----------------


        +-----------------------+        +----------------+
        |      Context          |        |   IStrategy    |
        |-----------------------|        |----------------|
        |                       +-------->                |
        | ctor(IStrategy)       |        |                |
        |                       |        | PerformWork()  |
        +-----------------------+        +---^------------+
                                             |
                                             |                          +
                       +---------------------+----+
                       |                          |
                       |                          |
        +-----------+--+-----+      +-------------+------+
        | ConcreteStrategyA  |      | ConcreteStrategyB  |
        |--------------------|      |--------------------|
        |                    |      |                    |
        | PerformWork()      |      | PerformWork()      |
        +--------------------+      +--------------------+

Install libraries into Clojure

This can be a bit confusing at the beginning, the loading of libraries into the Clojure project.
For this I use Eclipse and the excellent CounterClockwise.

The initial dependencies can be found in the Leiningen dependencies folder, shown below.

This is the initial project.clj file with one single dependency.

Adding the dependencies to the project.clj file

Doing the above will make Lein go and get the libs and place them in the dependencies folder as shown.

Not using Eclipse?

If Lein is installed, you can navigate to the project folder in the command line, terminal, and enter:
lein deps

Six Deadly Sins of Leadership - a comment

This post by Jack Welch and Suzy Welch appeared on Linked in and it struck a note with me. I feel that there is a lot in it that I think I agree with. I'm not sure if I'm entirely in agreement with the list or if it really matters whether I agree or not but there is, deep down in me, a healthy dislike of the mundane, uncreative and uninspired leadership/management here in the UK.

http://www.linkedin.com/today/post/article/20130327154206-86541065-the-six-deadly-sins-of-leadership?trk=eml-mktg-inf-m-top13-0827-p3

The main headings of the article are, the six deadly sins are:

1. Not giving self-confidence its due.
2. Muzzling voice
3. Acting phoney
4. Lacking the guts to differentiate
5. Fixation on results at the expense of values.
6. Skipping the fun part

The trouble with these lists are that they can only dumb-down what is a very serious problem. That's a bit unfair of me as its purpose is to highlight some problems and I guess it works at that. It's generated me to waffle on in this post to no-one in particular, so at least I'm thinking about it. But dumbing-down? Well probably. Or maybe just over-simplifying. Perhaps that's a nicer way to say it.

#1 - Not giving self-confidence its due

"Self-confidence is the lifeblood of success."

I'm not a great manager, or leader in the charismatic way, but I think and learn and challenge. Sometimes when time permits I can look up and innovate. But the above list of six things are not quite right, or in the right order. You do tend to associate precedence to the order of things and, Not giving self-confidence its due, isn't number one.

There are at least two equally important and crucial additional aspects to this. 

One - Not giving confidence or time to the person that lacks confidence. I have seen an environment where a person was not regarded highly for his talents. In fact the prevailing opinion of him was truely low. Behind the scenes, management would talk rather poorly of the person. I knew the person was bright, was hard-working, sharp, but there were issues. The transformation to this person's life, career and prospects are now profoundly improved because I worked at this, pressing  ahead with  support and mentoring. I remember that it took just one year from meeting this person to bring about significant change to him and the team.

Two - not dealing with the over-confident or arrogant.

This is tough and it takes good judgement and self-control. One person's arrogant is another person's genius. You must get it right but you must look at the personal interplay in the team and ensure that personalities are in control and in some sort of harmonious balance and that other potential stars are not being stifled. I've seen very dominant characters suppress others through bullying or sheer confidence. Whatever the method of handling this, it is important.

#2 - Muzzling voice

"Perhaps the most frustrating way that leaders underperform is by over-talking." 

Over-talking is bad. Yes I agree to that. But, over-talking is different to over-driving the discussion to your favour. I talk too much after coffee, but always listen to the input of the team. Collectively we know more and are greater than the sum of our individual voices. So I agree in general with the #2 but I think it's slightly off.

#3 - Acting phoney

"Can you spot a phoney  Of course you can – and so can your people. Indeed, if there is one widespread human capability, it is sniffing out someone who is putting on airs, pretending to be who they’re not, or just keeping their real self hidden. Yet too many leaders spend way too much time creating personas that put a wall between them and their employees. What a waste."

I didn't agree with this at first. I thought it over simplified what I imagined to be the real problem. But now I think I agree. There are a lot of "suits". Even if they don't wear them. Often they are well intentioned and well-ordered. Sometimes they are playing the corporate ladder of survival and progression. Maybe the companies need them. Maybe the company simply needs fewer of them. But there is a style that can be destructive to creativity and motivation. This needs to be understood and managed.

#4 - Lacking the guts to differentiate

"You only have to be in business a few weeks to know that not all investment opportunities are created equal. But some leaders can’t face that reality, and so they sprinkle their resources like cheese on a pizza, a little bit everywhere 

... 

But leaders who don’t differentiate do the most damage when it comes to people. Unwilling to deliver candid, rigorous performance reviews, they give every employee the same kind of bland, mushy, “nice job” sign-off."

Yes I probably understand this and agree. But  headline problem for this list. I'm not sure because I haven't managed many managers.

#5 - Fixation on results at the expense of values.

"Everyone knows that leaders deliver. Oratory and inspiration without results equal…well, a whole lot of nothing. But leaders are committing a real dereliction of duties if all they care about are the numbers."

I like this one. One thought along these lines that I believe is with visual products for example. It is simply not good enough to provide function and expect to differentiate ourselves from the rest. If we can get people to love our product through aesthetics or service then we have values and we have pride.  There are many aspects to this results v values equation that I need to think about. I will waffle on some more in another post.

#6 - Skipping the fun part

"What a lost opportunity. Celebrating makes people feel like winners and creates an atmosphere of recognition and positive energy. Imagine a team winning the World Series without champagne spraying everywhere. You can’t! And yet companies win all the time and let it go without so much as a high-five."

Yep.

I agree in the celebration part but celebration is not the only definition or expression of fun. Before that there is the fun of doing the job. I'm thinking particularly of a development environment here and there has to be a lot of appropriate fun. Interaction in the team, jokes, support, team work, banter - all useful where appropriate and sensitive to those who might feel alienated by certain behaviour.

Success comes daily. Solving a sticky bug, a change to a design, writing a good doc. It's not all big end-of-project celebrations.

How to make a Clojure program executable (Mac or Linux)

1. Create the project

Using Eclipse (and CounterClockwise) create a new project to try this out.
Create the project using the Leiningen option, the default.

For this example I created a project named, getting-data

2. Edit the source file (:gen-class)

In the getting-data.core file, in the src folder:

Add the directive:
 (:gen-class)

I have also added a simple "say hello" function.

(ns getting-data.core

  (:gen-class))

(defn say-hello [n]

  (println "hello, " n))

(defn -main [& args]

 (say-hello "john"))

3. Edit the project.clj file - :main getting-data.core

(defproject getting-data "0.1.0-SNAPSHOT"

  :description "FIXME: write description"

  :url "http://example.com/FIXME"

  :license {:name "Eclipse Public License"

            :url "http://www.eclipse.org/legal/epl-v10.html"}

  :dependencies [[org.clojure/clojure "1.5.1"]]

  :main getting-data.core)

You can see the text  :main getting-data.core has been added.

4. Download Leiningen.

This needs to be done to run the executable from the command line. Counterclockwise instals its own version of Lein but we will need our own.

This is very simple.

Go to github.com/technomancy/leiningen and download the script. You place the script where it can be executed, make it executable and run it. This will install Leiningen.

These are the instructions from the Leiningen page:

1. Make sure you have a Java JDK version 6 or later.
2. Download the script.
3. Place it on your $PATH. (~/bin is a good choice if it is on your path.)
4. Set it to be executable. (chmod 755 ~/bin/lein)

Downloading on the Mac will place the file into the Download folder. use the mv unix comand to move it and rename the file if needs be. You may need to remove a ".txt" extension of the file as you mv it to the ~/bin folder.

5. Navigate to the project in Terminal

This may be similar to this: cd ~/Documents/workspace/getting-data/

Check the files exist:

John-Mac-mini:getting-data John$ ls -l

total 16

-rw-r--r--  1 John  staff  202  7 Sep 19:56 README.md

drwxr-xr-x  3 John  staff  102  7 Sep 19:56 bin

drwxr-xr-x  3 John  staff  102  7 Sep 19:56 doc

-rw-r--r--  1 John  staff  298  8 Sep 13:56 project.clj

drwxr-xr-x  2 John  staff   68  7 Sep 19:56 resources

drwxr-xr-x  3 John  staff  102  7 Sep 19:56 src

drwxr-xr-x  3 John  staff  102  7 Sep 19:56 target

drwxr-xr-x  3 John  staff  102  7 Sep 19:56 test

Now just check the project file is correct:

Enter cat project.clj at the command line.

John-Mac-mini:getting-data John$ cat project.clj

(defproject getting-data "0.1.0-SNAPSHOT"

  :description "FIXME: write description"

  :url "http://example.com/FIXME"

  :license {:name "Eclipse Public License"

            :url "http://www.eclipse.org/legal/epl-v10.html"}

  :dependencies [[org.clojure/clojure "1.5.1"]]

  :main getting-data.core)

Yes that looks like the file.

6. Now run the file using, lein run

John-Mac-mini:getting-data John$ lein run

hello,  john

This doesn't create the executable, it simply runs the project. We can leave this step out if we wish.

7. Now we can create the executable Jar

Use, lein uberjar

This will contain all the dependencies and will be placed in the ./target folder.

John-Mac-mini:getting-data John$ lein uberjar

Warning: specified :main without including it in :aot. 

Implicit AOT of :main will be removed in Leiningen 3.0.0. 

If you only need AOT for your uberjar, consider adding :aot :all into your

:uberjar profile instead.

Compiling getting-data.core

Created /Users/John/Documents/workspace/getting-data/target/getting-data-0.1.0-SNAPSHOT.jar

Created /Users/John/Documents/workspace/getting-data/target/getting-data-0.1.0-SNAPSHOT-standalone.jar

It has created two files:

getting-data-0.1.0-SNAPSHOT.jar

getting-data-0.1.0-SNAPSHOT-standalone.jar

8. Run the file

java -jar ./target/getting-data-0.1.0-SNAPSHOT-standalone.jar 

How to find what is in the Clojure libraries with REPL

First, load the clojure.repl lib

First you need to load the clojure.repl library. You should be able to do this by typing:
(use 'clojure.repl)
If this fails then see this post http://fernmac.blogspot.co.uk/2013/09/cant-get-use-clojurerepl-to-work-in.html

What functions are available in a library

For example, what functions are in the repo library:
Type:
(dir clojure.repl)

Lookup documentation for a function

If you want to lookup the documentation for "when" for example, type:
(doc when)

Find definition by matching it's name

If we want to find all the definitions that contain the word "map" in their names then use apropos.
Type:
(apropos "map")

Find all documentation that contain a word

If you want to look up all definitions that have the word "stream" in it's documentation then type:
(find-doc "stream")

Look at the source code of a function

If you want to see the code for a function, yours or the Clojure functions.
To find the source code of the defn function, type:
(source defn)

Cheat-sheet

you can visit the Clojure cheat-sheet at http://clojure.org/cheatsheet or you can query via the REPL.

Can't get (use 'clojure.repl) to work in Clojure

Clojure 1.5.1
In the REPL you might not be able to use the clojure.repl library.

Normally you would type:
(use 'clojure.repl)

However this causes a problem. You need to Require it first.

So type,
(require 'clojure.repl)

then,
(use 'clojure.repl)

Propositional logic, First order logic and Higher order logic

Propositional logic

In Propositional logic symbols represent Facts (Propositions).
for example:

W - the person is a woman.

Syntactic elements

AND, OR, IMPLIES, EQUIVALENCE, NOT

First order logic (First order predicate calculus)

First-order logic represents the world in terms of objects and predicates. It uses the relations between objects and their properties. It also uses quantifiers and connectives.

syntactic examples

Objects - door, house, dog, ...
Relation - employer, sister, inside, ...
Properties - black, free, empty, hot, ...
Functions - first in, father of, friend of, ...

Syntactic elements

terms - objects
Quantifiers - UNIVERSAL (for all),  and EXISTENTIAL (there exists)
EQUALITY symbol (=)

Syntax

In propositional logic every expression is a sentence and every sentence expresses a fact. In addition to sentences First order logic also has terms that represent objects.

Higher-order logic

First order logic is so called because it works with the first order objects of the world. Higher-order logic enables us to quantify not only objects, but relations as well. What that can mean in practice is that two objects can be thought of as equal if and only if all of their properties are equivalent . We could say that two functions are equal when their arguments have the same value.

I'm not too sure of the value of HOL and it seems a bit contentious. 

Processor time in perfmon for SQL Server diagnostics

It's along time since I used Perfmon for SQL Server problems so I looked up the basic starting counters to use.

Processor = % Processor Time 

This can spike and probably should not be above 70%. This may be system wide problems and not SQL Server. If high it needs investigating because CPUS should not, are not generally the bottleneck in modern systems.

Processor - % Privileged Time

The amount of time the processor on Kernel system processing. Is there a problem with the IO OS system?

Process (sqlservr) - % Processor Time 

See how these relate to the previous Processor times. Is it SQL Server using the CPU?

Process (sqlservr) - % Privileged Time

System - Processor Queue Length

This value should be zero. It counts the number of threads waiting for CPU

Memory - Available Mbytes

> 300Mbytes is ok.
> 500Mbytes is better.

If memory is disappearing then it may be that you have not set a maximum for SQL Server. SQL Server will grab it all if it can.

What SQL Server Perfmon Counters to look at

There are a ton of Perfmon counters to look at when investigating SQL Server problems. This pdf is a great resource for starting out.

http://www.quest.com/backstage/images/promotions/SQLServer-Perfmonance-Poster.pdf

There is a cool spreadsheet too

http://blogs.msdn.com/b/jimmymay/archive/2008/10/15/perfmon-objects-counters-thresholds-utilities-for-sql-server.aspx

SQl Server Diagnostics scripts

Glenn Berry's SQL Server Performance

http://sqlserverperformance.wordpress.com/2013/07/09/sql-server-diagnostic-information-queries-for-july-2013/

Download and go. Pretty darned good.

SQL Server 2005 Waits and Queues

Really, really useful. Everywhere I go the same problems exist. SQL Server slow and no one knows why. It's a useful topic and this is a good part of it.

SQL Server 2005 Waits and Queues - SQL Server Best Practices Article, by Tom Davidson, 2005

It can be downloaded from http://bit.ly/aUh6S


"wait time" - the total time taken for a thread to go from Running, to being "put on the queue", or to being "queued, and then back to Running again.

"resource wait time" - the time a thread is waiting on the Waiter List with the state of Suspended.

"signal wait time" - time waiting for the processor once the resource has become available.

So wait time = resource wait time + signal wait time

When a thread has server time, and then becomes blocked, it is set as Suspended and then placed on the Waiter list.

When it receives a signal to say the resource is available, it is set to Runnable and it is place on the Runnable Queue. It will wait here until it is given to the Processor.

sys.dm_os_waiting_tasksDMV
Will show all threads that are currently suspended.

Machine Learning in Python

I was looking for info on Stocastic Gradient Decent algorithm and saw this:

Scikit

http://scikit-learn.org/stable/index.html

There is a good 45 min video http://scikit-learn.org/stable/presentations.html (the skit-learn video by Jake Vanderplas) where the url he talks about is wrong I think he means http://www.astroml.org/sklearn_tutorial/

I've since been searching around and found the following that look interesting too:

PyBrain

http://www.pybrain.org

mlpy

http://mlpy.sourceforge.net
Seems pretty extensive

milk

http://pythonhosted.org/milk/

Still early days

Calling Prolog from Python

This is my shopping list of things to investigate. It's a notes page of where I am at the moment in the process of connecting Python and Prolog. I hope it might help someone who is trying to do the same thing, or perhaps thinking about trying to do the same thing. If you have any ideas then it would be great to hear from you. At the moment I am thinking that programming in C++ and using SWI-Prolog's C++ interface might be the answer.

I've been looking into this lately and have collected these  things to look at.
I can program in Prolog and like it. I'm not great at it but I love the way of thinking. I have been using Python also, not that I particularly like Python, but it's so useful and appears in so many places. (yes I've been plating with the Raspberry Pi). So, can I get the Pi to use prolog as it's decision making engine?

pyswip

https://code.google.com/p/pyswip/
This is a bridge between SWI-Prolog and Python and it seems to be the first place to visit. It works on Linux and Win32 but not sure about OSX

I looked at this first and thought that it had stopped being developed. I'm not sure. I think there was an update in December 2012.

There seems to be a nice post on someone's expereince from 2011 here: http://ryepdx.com/2011/09/prolog-in-python-pt-1/

Picstus

This is an interface between Sicstus Prolog and Python. I probably wont look at this as I am using SWI-Prolog for the time being. Sicstus costs about 165 Euros and I'm not that professional.
http://www.biolab.si/picstus/picstus.html

A Prolog Interpreter in Python

This is very interesting stuff. Excelent stuff. http://wwwold.stups.uni-duesseldorf.de/thesis/Bolz2007-Bachelorarbeit.pdf but probably it's not going to help me.
[UPDATE 2016] The above link no longer works. Try this http://stups.hhu.de/w/A_Prolog_Interpreter_in_Python

There is also some work by Chris Meyers http://www.openbookproject.net/py4fun/ a few links in this page. In summary though it's too slow for a real world solution.

SWIG

"SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages." So We could wrap Prolog with Swig.

http://www.swig.org/

Probably not what I want to do as a bit complex.

Pwig

This is a swig extension for Python.

http://pwig.sourceforge.net
The last activity was in 2004.
Can download from here:
http://sourceforge.net/projects/pwig/files/pwig/
SWIG is at version 2.0 but PWIG required SWIG 1.3.23 which I couldn't see. It might be there somewhere.

PyKE

Now this is interesting. http://pyke.sourceforge.net/ 

It introduces a form of logic programming into Python. There is an interesting paper at http://pyke.sourceforge.net/PyCon2008-paper.html 

It has an inference engine but the syntax is new. I suppose it would do but it's a shame not to improve my skills in Prolog. That said though, the skill in prolog is the thinking not the syntax.

PyProlog

A Python extension embedding SWI-Prolog. This was last updated in 2001 so is a tad out of date with very little or no information.

http://pyprolog.sourceforge.net/

Pyrolog

I'm not sure about this one. A prolog interpreter written in RPython.
https://bitbucket.org/cfbolz/pyrolog

PyLog

http://wiki.python.org/moin/PyLog

Well I'm not sure about this. It's a first order logic library. It also

Knowrob

KnowRob is a knowledge processing system that combines knowledge representation and reasoning methods with techniques for acquiring knowledge and for grounding the knowledge in a physical system and can serve as a common semantic framework for integrating information from different sources.

Too big for my needs here but really cool.

http://www.knowrob.org/knowrob

So now what do I think?

Well I'm beginning to think I should look at coding in C or C++.
My task is for the Raspberry Pi and it's a shame to slow it up by getting into C again after so many years.  Python wold be such a fast and rapid development. Maybe I create a C interface to my Prolog. Then use Python for all other motor controlling, sensor controlling. Then, when it's a success and become a millionaire, I can refactor all the code from Python to C for the fun of it.
This is a disappointment. I'll do some more digging and write my thoughts here.

Can I instal prolog on the Raspberry Pi

Yep.

sudo apt-get install swi-prolog

Is numpy included on the Raspberry Pi

Yes it is.
My Raspberry came with Raspbian Wheezy installed.

At the command line I started python,

then entered:

>>> from numpy import *
>>> a = arange(15).reshape(3, 5)
>>> a

 Which gave:

array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14]])
>>> 

How to select the Python.3 interpreter in Eclipse on a Mac

This foxed me for a bit.
I have Pydev installed, and Python 2.n and all is well. But I needed Python 3.3

Install 3.3

So go to Python: http://www.python.org/download/ and download the installer.

Then install as usual on a Mac.

Now check that it's in place.
There should be a folder in Applications/Python 3.3

There should also
Macintosh HD - Library/Frameworks/Python.framework/Versions/3.3

Select the interpreter.

So now, select the interpreter in Pydev.

1. You go to Eclipse > Preferences > PyDev > Interpreter - Python
2. From here you, Select New, 
3. in Interpreter Name you enter python3 and in Interpreter Executable you enter /usr/local/bin/python3 or /usr/local/Cellar/python3/3.3.0/Frameworks/Python.framework/Versions/3.3/bin/python3.3.
4.  OK

Lego Nxt plug connector for Raspberry Pi

I haven't done this yet but I'm getting ready to. That is, connect the Raspberry Pi to the Lego NXT Mindstorm sensors.


I've a circuit schematic, but there's also some good info on how to adapt a regular telephone plug to use a NXT socket.
http://www.philohome.com/nxtplug/nxtplug.htm

You'll need to cut and file a bit.

How to connect the socket on the Pi for a NXT cable.

I haven't actually tested this yet. Maybe next weekend :)
My advice would be to use this as a starting off point and experiment.

How and why to document Business rules (and Use cases)

Business rules are, I think, not given enough consideration in the projects I come up against. I'm a bit of an itinerant member of smallish teams and regularly see teams short on resource and time. Often I hear people speak of "business rules",  and nearly every project I've been on I've not seen anyone documents them.
Perhaps they don't need documenting. I think they do.

In my most recent contract one project had an incredible amount of Business rules. The task of the project was to replace an old system and set of accompanying processes with a new set. Most of the Business rules were to be in an "expert system" to do with qualifications, grades and occupations but there were many outside of that. The expert system was techie and geekie and obtuse but it documented the rules via the application's UI so they were taken care of, almost by accident. The rest were stumbled upon, discussed in emails by people who didn't know for sure, and either forgotten or if we were lucky, a code change was made to support it. Either way the rule was lost either in code or the fog of the project. And this continued for years.

The project team was under-strength and spread over three locations, business analysis was inexperienced and absent for some months, and the project was far more complex than anyone really expected. For my sins I came into the project after two years and immediately struggled. After my first eight months the first major phase was working. Which is ok I suppose. I'm only saying this as an example. It's an example in the general sense and to a lesser or greater degree it is common.

The general picture of the development and project process in this example would be to say that some of the developers wanted a Scrum approach but other developers distrusted this and management certainly distrusted this. So Scrum was replaced by a version of nothing in particular. This is pretty common too. So what could have happened at the beginning to have made it all smoother?
The objective for the process at the beginning of the project was to let the Business knew what they were letting themselves in for as the foundations of the project were being laid and details being discovered.
What is it that we are really going to be building?
What changes is this thing going to make to the business processes, to the people, to the productivity, to the service? Oh and how many people will be needed to get this all done and how long will we have to wait?

The way not to do this is to get inexperienced people to document the requirements and blame them a few years later if it goes titsup. What not to do after this is to get all excited and give it to the developers, stand back, and wait for the first delivery.

The Scrum thing will not work here. What we should do is think about the business processes. What is it that the Business is doing and how will we change that. There are ways to document this and be all professional (BPMN see bpmn.org). But if you are without a BA a quick google can help. Something is better than nothing. A flow chart, a list of processes in MS Word is a great way to frame the problem domain. The Business Domain.

But this post is about BusinessRules. So where do they come in?

For me they come in alongside the Use Cases, or User Stories. Quite often a User Case or Story can be a Business Rule. For example, A Business Rule could be that we send a summary of trades to the Ombudsman within 20 days. This could also be a Story, As Compliance Manager, I must send the summary of trades to the Ombudsman within 20 days of the request.

However, it may be slightly different. Perhaps the story / requirement will be for a configurable date or alerting system.

Also, Business rules may define the data being held on a customer, and when it must be deleted for data-protection regulation purposes.

For me, the huge benefit of use cases is the fact that they describle, in easy terms, how a lump of work is done. A process. A small process. That is the business and that is what out work is going to support. Starting with declarative technical requirements such a the "the Save button must confirm the user is saved in the database before saving the data." is so abstract and difficult to communicate to a non-technical business user in a project meeting. They must be good for the development team a bit later on, maybe in a work unit, a task detail, but not in a 100 page requirements document.

How to start with Business rules

I always create a document with a few sentences explaining why the document exists, and then I put a list of Business Rules, and then Use cases.

The only important rules I apply to the writing of business rules is:

The rule is not dependent on nor refers to any technology or system. The rule should be equally valid if the business was being run using a pencil and paper.

Rules should be grouped into easily understandable and relevant groups. And each Business Rule should have three fields:

1. An ID (can be a number or a letter and a number). The letter can relate to the title of the group of the rules. If  a rule is in a group entitled Refund Business Rules, then number the rules R1, R2, R3...
2. Business rule name, one to 4 words in length
3. A short description of the rule. 

Rules can be things like:

• what bits of data are mandatory, 
• what bits of data are to be removed.
• Trading days,
• Compliance issues.
• Regulatory points.
• Welfare issues
• ...

Another very simple way of looking at Bayes theorem

In an earlier post (back in 2011) I mentioned that it would be interesting to visit Eliezer Yudkowsky's site and in particular his explanation of Bayes' theorem. It still is. In the post I posed the question that Eliezer uses on his site and gave the answer but I suggested you go and visit his site to find out how to calculate it. I didn't realise how common that question was back then. I was thinking about it recently and thought it might be nice to show another, more graphical way of getting to grips with the calculation.

Here is the question:

1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

Remember that in tests, only 15% of doctors got the correct answer. The same is true in courts where probability of guilt is calculated. There are lots of examples and it's pretty serious. Bayes' Theorem really needs to be better understood.

Step 1

Right, we can draw the following tree.

1% of women (0.01) have cancer. 

Therefore, 99% do not (0.99). 

We mark the values on the branches as shown
            
                
        0.01    
      --------- C [cancer]
     |          
     |          
     |          
     |           
     |
     |
start
     |
     | 
     |           
     |          
     |          
     |  0.99    
      --------- NC [no cancer]
                 

Step 2

What else do we know?

80% (0.8) of women with breast cancer will get positive mammographies. 9.6% (0.096) of women without breast cancer will also get positive mammographies.


                     0.8
                 ---------- P [positive]
                | 
                |
        0.01    |
      --------- C [cancer]
     |          |
     |          |
     |          |
     |           ---------- N [negative]
     |
     |
start
     |
     |              0.096
     |           ---------- P [positive]
     |          |
     |          |
     |  0.99    |
      --------- NC [no cancer]
                |
                |
                |
                 ---------- N  [negative]  


So the question is, given the above tree, a woman gets a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

Step 3

I want to fill in the missing branches just to make it complete. So before we get into calculating our answer we'll tidy up.

                    0.8
                 ---------- P [positive]
                | 
                |
        0.01    |
      --------- C [cancer]
     |          |
     |          |
     |          |   0.2 
     |           ---------- N [negative]
     |
     |
start
     |
     |              0.096
     |           ---------- P [positive]
     |          |
     |          |
     |  0.99    |
      --------- NC [no cancer]
                |
                |
                |   0.904
                 ---------- N  [negative]  


The thing to do here, is to make the related branches add up to 1.0. 
So 0.8 + 0.2 = 1.0
And from the other node, 0.096 + 0.904 = 1.0
That is the probability of going down one of those two routes at each node is 100% (1.0)

Step 4

We calculate the final nodes, (they are intersections actually but that terminology might be confusing in this diagram. The intersections between C and P, C and N etc)
But 

                    0.8
                 ---------- P  C ∩ P = 0.01x0.8 = 0.008 
                | 
                |
        0.01    |
      --------- C [cancer]
     |          |
     |          |
     |          |   0.2 
     |           ---------- N  C ∩ N = 0.01x0.2 = 0.002
     |
     |
start
     |
     |              0.096
     |           ---------- P  NC ∩ P = 0.99x0.096 = 0.095 
     |          |
     |          |
     |  0.99    |
      --------- NC [no cancer]
                |
                |
                |   0.904
                 ---------- N  NC ∩ N = 0.99x0.904 = 0.895  

Step 5

Now answer the question.
A woman gets a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

So we write this as,

P( P | C)   
      
which means Probability( Positive | Cancer), 
or, the probability of getting a Positive, given that the woman has Cancer.

This is conditional probability.
 P( P | C ) can be converted to:

 C ∩ P          Note that  C ∩ P = P ∩ C  
   P

So we plug in the numbers:



 C ∩ P     =      0.008             
   P          0.008 + 0.095   <--(all of the posibilities of being positive)

           =  0.008 
              0.103

           = 0.77 or 7.7%  

most of the doctors when tested, estimated the figure to be around 70%!

The equation

It doesn't look too much like Bayes' theorem though does it. It might be good to work toward the equation now though we don't need to know it to do conditional probablity.

P(A|B) 

is read as the probability of A given B. So we know B, or B has happened, so what is the probability of A.


    P(A|B) = P(A ∩ B)

               P(B)

we can rearranged this as

    P(A ∩ B) = P(A|B).P(B)

It can be noted that 

    P(A ∩ B) = P(B ∩ A)

   

Also,

    P(A ∩ B) = P(A|B).P(B)

    P(A ∩ B) = P(B|A).P(A)

So,

          P(A|B).P(B) = P(B|A).P(A)

which can be arranged to give us the usual equation:

    P(A|B) = P(B|A).P(A)

                P(B)