test objects design

I made it to Dallas on Friday night. I was scheduled to fly to Houston but the Houston airport was shut down I believe and my flight from Amsterdam got diverted to Newark. The entire trip took about 23 hours. I left Germany at 6am Friday morning their time. Then Amsterdam flight at 1:30pm to Newark. Got there at 5pm and a flight to Dallas at 7pm (about 2 hours delay) and got to Dallas just before 11pm. Saturday was getting together with family in Dallas day after being in Germany for 4 months. This morning back to Job Hunt. Already 3 solid opportunities. So back to writing about Software on this blog.

Technorati Tags: marekj

Some quick loosely joined ideas:

As a Software Tester participating in building support structures of cooperative games of invention and communication of Software Development I need to know the history of a particular invention I am going to test.

Each business process, each requirement flowing from a business model of a company is an invention that supports business in growing revenue and earning profit. We need to know the history of an invention and then the invention’s usage in the context of a business model.

Let’s say I am testing a toll free number billing system. If I see that the toll free number calling was an invention then I would need to understand it to know how users are practicing this invention every day and how businesses are using this invention to create services and bill for them.

It’s not enought to just call this 800 number and test that it rings and a person picks up on the other end. There is a whole business process just for that one phone call and I would need to reason for a while to come up with a whole suite if test scenarios. To think of scenarios I would need to keep in mind the historical nature of an invention I am dealing with here even as it’s being implemented in a different context. - This further calls for separation of design from implementation. One tests design to verify that what is going to be developed is that was intended. One tests implementation to validate the assumptions of design.

Technorati Tags: agiletesting

Rational Robot is a capture/playback tool. On playback sometimes It has a problem with focusing on a web browser because it records the setting of playback context using literal strings, usually a string literal title of an html document and it puts it in it’s Caption parameter. Here is a a short way one can use custom keyword command for setting a focus and waiting an a web page to load before running next command. It’s a mini automation framework. You can read more about automation frameworks by looking at Carl Nagle’s RRASF which is really but to implement it on a project you need management support. Well worth it.

But for now let’s make your own mini framewor. Here is a simple example (really really simple). We create a keyword command ‘SetFocus’ with no parameters to use in your scripts to focus on a webpage and wait for page to load before proceeding.

Set up a ‘library.sbl’ where you define a procedure called ’setFocus’ like this:

‘****************************************************

Sub setFocus

’set focus to main app browser frame and wait for page to load

‘REQUIRE: use constants to define a TAG and FRAME targets

‘****************************************************

Window SetContext, WINDOW_TAG, “”

Browser SetFrame, WINDOW_FRAME,”"

Browser NewPage, “”, “”

End sub

Then define your global constants (in the same sbl file)

‘Used in StartBrowser command (You can call your AUT any name you need.

global const WINDOW_TAG$ = “WindowTag=AUT”

‘in this calse ‘main’ is the id we want to work on.

global const WINDOW_FRAME$ = “Type=HTMLFrame;HTMLId=main”

And now in your script you can use the command ‘SetFocus’ before running other commands against your target window. In your script you must include the library with ‘$Include “libraryname.sbl” directive.

I think Rational Robot is cumbersome procedural hog machine. But for people who have learned how to use .sbl and .sbh files and modularize subs (and never ever ever use the DataPool utility but instead hook up a database directly to your scritps) then Rational Robot is a nice test automation tool.

Technorati Tags: Rational Robot

I downloaded MaxQ from Tigris projects. Trying it out. (sidenote: Tigris projects are hosted by CollabNet, a company founded by Brian Behlendor, a co-founder of Apache) to run on Mac OS X and Firefox browser.

Install: just download and run ‘bin/maxq’

Sep 13, 2005 4:44:46 PM com.bitmechanic.maxq.Main startGUI

INFO: MaxQ Proxy Recorder v0.98

Sep 13, 2005 4:44:46 PM com.bitmechanic.maxq.Main startGUI

INFO: Proxy server started on port: 8090

and a test editor window pops up.

Now we need to setup Firefox: Open Firefox>Preferences window. Undr General tab click on ‘Connection Settings…’ button to set up a proxy. Check radio button ‘Manual Proxy Configuration’ and type ‘localhost’ in HTTP Proxy edit box and type ‘8090′ in Port edit box. We are done.

Go to MaxQ Editor and select File>New>Standard Script You should get this template: Now we are ready to record. Select Test>Start Recording.

In a browser I type “http://marekj.com/” and hit ‘enter’ key. I wait for the page to load and then click on a link to go to this blog. I wait for the entire page to finish downloading and return to MaxQ editor and select Test>Stop Recording. MaxQ asks to save this script. I call it ‘blindspot.py’ So now we should have a script that looks like this. So now what?

That’s a pretty long script. Not very readable. Will it run?

Let’s try. Select Test>Run. Here is the output. Done. I got a 404 on favicon.ico because it wasn’t there. It was recorded because it’s part of HTTP protocol.Then I got 301 on a js file embedded by technorati script at the end of the document.

The script itself is quite wordy I might say. It would be so useful to have some framework to make it keyword driven testing ruather than pure python code. OK. that’s it for now. We’ll get back to MaxQ some other time.

Technorati Tags: Agile Testing, MaxQ, test script

In Learning By Breaking Barry Warsaw writes:

“One of the best ways to learn how a system works is by breaking it.” and “Python’s pdb module and print statements are the indispensable tools of the professional software monkey wrencher, at least in the Python world. They may not be terribly high tech, but they can be pretty effective when exploring”.

I downloaded a paper by Jeff Jewell of ProtoTest presented at StarEast 2005 conference. PDF found at stickyminds (powerpass required) but I uploaded it to my server because when you google it it’s not found anywhere. It’s also too bad that the pdf is password protected and you can’t copy any content out. I uploaded it to my server as a reference (will not be indexed by robots)

“This paper will provide some of the information needed to successfully navigate the minefield of open source test tools. Some of the benefits and common misconceptions of open source software will be presented, along with information on what it takes to successfully use open source tools, how to find an appropriate tool, and what challenges may be faced in its implementation”

Jeff writes about the current state of Open Source tools for testing. Points to OpenSourceTesting and Tejas Open Testware Reviews.

So, what’s available? Mostly tools where testers would need to know some scripting language (like Python or Ruby). Most tools are for unit testing, performance testing and test Management: like FitNesse. I personally think Wiki is a great tool for collaborative requirements gathering.

For Functional Web testing there is Canoo but it’s for back end and it does not exercise GUI which is what we want to do to emulate the real user scenarios. There is also selenium which uses JavaScript and iFrame.

Lots of tools for performance testing (but not my specialty). I am all about functional testing, requirements and such… I did once use JMeter for some simple performance tests. Another one is OpenSTA and uses Script Control Language. Here are some general Rules of SCL usage.

As for defect tracking there is a slew to choose from but I think I would select FogBugz from Joel’s company. Not open source but worth it. For bug tracking I would definitely use FitNesse or some other Wiki. I think because the minute you find a bug it screams to have a test written against it and added to regression testing.

I think tracking bugs is for management mailny. Sometimes I think it’s a waste of time to manage a bug as a defect, I would rather manage it as a Test Case which either passes or fails. The real issue is that there is a bug and if we can reproduce it then this is a Test so I would put it in FitNesse, write a story in Wiki and track it that way.

I think using bug tracking is better if you have geographically dispersed team with customers and other shareholders who can enter bugs. I also found most bugs are misconceptions and/or non-captured requirements that are coming out as software progresses. Bug tracking tools are more about eliciting conversations for improving sotware. Any Bug Id is really a repository for the history of a conversation. The actual bugs should be in Tests like FitNesse or some other executable form.

It is true that “The landscape of software testing tools is dominated by a few high-end vendors with suites of tools that are very rich in features and relatively easy to use” but the problem with those tools is that they all use Vendorscript languages and many do not have any Frameworks defined. Out of the box they all mostly capture/playback tools. (XDE tester uses Java which I think is an over-kill and it’s very time consuming to write tests, at least my experience from the pilot project I did).

Technorati Tags: Agile Testing, test tools

Reading Brian Marick’ s paper (PDF version) “Methodology Work is Ontology Work” and I love the way he starts it “In this paper, I’m going to abuse the word “ontology”. A Very good 9 pages of reading.

What Brian is postulating about Agile ontology, as “distinct from the standard “software engineering” ontology, which is much more dour. In it, entropy is a fundamental thing inthe world. Software development is a struggle against entropy that will inevitably fail as the product devolves into Foote and Yoder’s “big ball of mud”", is a possibility of “basic nature of a program is to be incomplete and flux, to solicit then beings. Given the right practices, social organizations, workspace arrangement, and tools, the software will be fundamentally supportive of human striving.Software has it in its nature to be soft, if only we know the right way to accommodatethat nature.”

Software is written by human beings striving to implement in time and space their gestures of commitment to a world they want to live in. A more common way this can be expressed that software implements the business requirements which begs a quesions: aren’t business requirements somewhat partial expressions of a world we strive for?

Technorati Tags: Agile Testing

I understand the wiki. It’s a nice example of a front end to FIT. Of course someone has to be writing Fixtures, right? I am not clear on who writes them but I think it would be developers hooking up tests to invoke their methods. So far I am going over examples provided. What I understand is that I write Tests against the Fixture code which is piping it to my App code. So if Tests are in Wiki then where is Fixture code then? Plus will Fixture code be written first? or do I first write tests, then create Fixture code?

There are 3 types of Fixture tables

1. Column = If I give you this and that then will I get that other thing back?

2. Action = calling methods and getting stuff back? (need to get clear on that)

3. Row = run queries?

As a sidenote: I love this from How to Kill a Software Project Fitnesse page

Problems!

At some point, usually late in the project, the team discovers that among other problems, they are finding one or more of the following problems with the features being delivered:

• They are not exactly what the customers/analysts/product managers think they asked for.

• They are not exactly what those folks wanted or needed.

• They are not useable by the system’s eventual users.

• Subsystems cannot be integrated with one another, because their interfaces are incompatible.

I think when Management steps in it makes very little difference because Functional Design needs to be recreated at this point. I used to say to my manager “This will need to be designed, we can do it now or 5 minutes before going live at 2 in the morning where everybody is tired. So when do you want to do it?”

I am trying to connect the dots with Fitnesses; I just don’t get how fixture can be a living doc for testers and customer; and what do developers get out of it? I guess it’s time to join fitnesses mailing list and snoop around the archived conversations. Some 1000 members and almost 4000 messages since Feb 2003. Looks like going strong.

Technorati Tags: Fitnesse

Downloaded FitNesse. I am on Mac OS X. Here are some of my quick reference notes on installing it and shutting down (See: StartingAndStoppingFitNesses)

Since I am running a local Apache on port 80 I opened run.sh file and put made it run on port 8080 like so:

java -cp fitnesse.jar fitnesse.FitNesse -p 8080 $1 $2 $3 $4 $5

and then I ran ’sh run.sh’ in terminal (I was in a folder were fitnesse was copied)

Terminal gaves me a message;

FitNesse (20050731) Started...
port:              8080
root page:         fitnesse.wiki.FileSystemPage at ./FitNesseRoot
logger:            none
authenticator:     fitnesse.authentication.PromiscuousAuthenticator
html page factory: fitnesse.html.HtmlPageFactory
page version expiration set to 14 days.

… and FitNesse server was runing on http://localhost:8080/

OK, now in terminal window let’s run ^Z to end. I am not sure if this was a desirable action. Perhaps a better way is to open another Terminal window nad run “java fitnesse.Shutdown” command. Or as I just found out a url “http://localhost:8080/?responder=shutdown”

Tried starting FitNesse again and instead got this message:

FitNesse cannot be started…

Port 8080 is already in use.

Use the -p <port#> command line argument to use a different port.

So perhaps I shouldn’t terminate the session like that

I then ran a url “http://localhost:8080/?responder=shutdown”. It looks from the tests there is a 100 delay (I guess it’s 100 seconds so let’s wait)

After few mintutes I ran ’sh run.sh’ command and got the same error message. I then ran “java fitnesses.Shutdown -p 8080″ and I got this error:

Exception in thread “main” java.lang.NoClassDefFoundError: fitnesse/Shutdown

This is strange. H.. Ah Yeah. I used wrong command: Here is the correct one.

java -cp fitnesse.jar fitnesse.Shutdown -p 8080

Now I am waiting and waiting and waiting and I never got the prompt back. System.exit didn’t happen. Not sure why. So I terminated the session window. Opened a new one and restarted the server. It went ok.

Is the trick to close the terminal window to end the session then?

Technorati Tags: Fitnesse

I was reading Johana Rothman’s slides from “Hiring Yesterday’s Testers” (PDF file) from PNSQC 2004. I shall quote the Slide 11 titled “Need for Frist-Class Testers”

- First-class testers provide information about the product underdevelopment and test

- Testers have at least three sets of customers

1. Developers

Testers write tests before code is written

Testers find problems early

Developers are more interested in creating fewer defects

Developers have more flexibility about when and how to fix problems

2. Product users

3. Company managementTesters discuss risks, not stamps of approval

There is a whole world of assumptions in this slide. I think one could write a page book just based on this slide alone. To define a tester is to see someone whose job it is to bridge the gaps of disparate worlds, to switch vocabularies, languages; from taking code to talking design to talking measures; communicating with different worlds in their own native vocabulary.

At the same time a tester would need to be well versed in a body of knowledge of a particular business domain software is building. For example If I don’t know how ’stop’ or ’stop limit’ stock market orders are executed using an online brokerage I would be a useless tester of that feature even if I know how to do boundary value analysis. I would need to know a bit about being an ‘Investor’ who is actually using the brokerage trade tool to set up ‘limit’ or ’stop limit’ orders on a system and I would need do know about Ask and Bid spreads and how they trigger orders on NASDAQ and NYSE or AMEX. Domain knowledge is the first thing I would jump on as a tester. Only after that my skills as a tester are useful. Domain knowledge is tacit knowledge for most but testers need to externalize it to build test scenarios.

If my customers are Developers then I need to know how to read the code with them. As a tester my job is to create Use Cases, scenarios, Stories, whatever it takes before any code is ever written to find out ambibuities in system design. I can then go over those stories with developers who are going to implement them. A developer can’t just read few lines of requirements and start coding. This will lead to disaster.

Yes, my customers are product users. Then I need to be versed in Usability Interface Design because this is all black-box approach for them. This is where User Stories is a common agreement for Acceptance Testing. I need to stand in two worlds. One of Developers looking at the code and one of Customers looking at UI only. On top of that Management wants to know risks, measures, results, costs, Return on Investment. Discussions on approach to testing and sunken costs as well.

On top of this I need to be a toolsmith. Well versed in OS interoperability, run SQL queries, discuss missing data with DBAs, run regression tests to verify the last snapshot of scenarios passes, document changes, retest changes etc… etc… That’s a lot.

It’s a complex job to be swiftly moving from domain to domain, jumping language barriers, speaking different vocabularies given different worlds a tester communicates with every day. - It’s an exciting intellectual complex work. And if this Sotftware Development is a cooperative game of invention and communication then it’s pretty exciting as a tester to support that game with my skills.

Technorati Tags: Agile Testing