And, we’re back for another semester of poking and prodding the Microsoft Source Server for use with the Mozilla debug symbols.

First order of business is to clean up as much of the symbolstore.py patch as possible and then the next step would be to start writing a pdbstr.exe replacement that can be distributed. Also, there’s the issue of Mozilla 2 and mercurial…

Looks like there was an involved process in choosing the next generation VCS for Mozilla 2 and so now I am wondering whether I should be continuing to work on my 0.3 or start working on something that will make source server work with mercurial.

This week I’ll be trying to pull together a solid list of what I hope to accomplish this term for my 1.0 release, i’ll post more soon.

Today was the first class of my BTP600 Design Patterns class. The teacher talked about how there are 3 ways to go about learning to write larger, non-trivial programs.

1. Try + Fail + Fix
2. Read the code of other (superstar) coders to see how they did it well
3. Design Patterns

I like this class. It’s very hard to see how I will ever write something large and non-trivial but I know that’s what we will accomplish by the end of BTS and this course as well. When I look at the mozilla code base, I can’t see how something like this came to be. What do you write first?

The teacher mentioned that Kernighan (sp?) and Richie write really great code and so I’d like to read some. Anyone know of a good example? I welcome tips on code that you may have found exciting to read.

Something that I have come to appreciate about Mozilla is that its chat channels on IRC are incredibly positive, helpful and relevant. As there are so many areas in developing and using Mozilla projects, several channels exist that allow you to join a channel when it is specific to your area of discussion.

Recently I went to the Joomla channel because I am working on a project for a friend in Joomla. Now, this is my first time working with Joomla. I have attended a brief “Hands on” workshop at the Ontario Linux Conference and I’ve read some of Dan Rahmel’s “Professional Joomla”. I was having trouble understanding the way that Joomla uses plug-ins. Basically there’s a plug-in that can recognize a snippet of custom code in an article – ie: {images view|gallery} or something like that. What I wanted to do was to be able to put that code in a module so that I could position it on the page in the way that modules can be positioned. I can’t find a way to position articles.

So I go to the Joomla channel in the hopes that talking with real person will help – much as going to the mozilla channels has helped in the past. I know that the #seneca channel is special because it is targeted at students who are learning, but the Joomla channel said at the top “Be patient, as your question – if no answer, try giving more info” (paraphrasing). I assumed that I could ask my question and get at least one person to talk it out with me.

Wrong.

What I got instead was a big fat nothing. NO response. NO acknowledgement. I asked politely if I should provide more information, and someone (who had been talking away about someone being gay) said “Oh, i didn’t see the question”. So I rephrase it and ask again – then nothing again. What?!

In frustration, I ask “is anyone here knowledgeable about plugins?” and a half hour later, at which point I had moved on to other tasks since there was no reply yet again, I get a private message from someone who says “ya, what you want?”. I didn’t bother responding. I created an html workaround to what I wanted to accomplish and moved on.

Needless to say, this experience was disheartening for a moment. But then I saw the positive side. I am a small part of a channel or two on the mozilla irc that is friendly, helpful and welcoming. So that’s nice to notice and appreciate.

Today was the first day back in the hallowed halls of S@Y. I managed to clear up the conflict signing up for DPS911 and am now officially registered.

I’ve gotten some great feedback on the bug I filed for the source server so there’s lots to get started on. First rewriting the patch better and then the grand task of writing a new pdbstr.exe that we can distribute because the Microsoft version cannot do so.

Anyway, that’s the short, glossy version.

I’m looking forward to getting back in the game. As the last few days of the holiday whip by I’m also working on some websites for friends, training to be a tutor in the Learning Center, helping some friends paint and trying to finish another pair of socks.

Yesterday at a New Year’s brunch I spent some time with a friend who has recently lost a large amount of her vision. She is almost (but not quite) legally blind and her vision will continue to deteriorate at an unknown rate. She’s a visual artist as well as a teacher and we had a lively discussion about software and adaptive technology. I’m determined to find a way to develop simple, well designed, as open source as possible programs for the Mac. This is a challenge since a) I have been informed that there is no money in writing adaptive software and b) apple development is a mystery to me and c) I’m not sure if it’s even possible to write open source programs for Apple.

Learning about this will be the sidebar on the to-do list for the winter you know, in my spare time

Well, the big ball ‘O duct tape that is my hacks on symbolstore.py are complete for now – here is the patch. This project will be carried on in the next term at which point I hope to address some of the current issues, namely – manipulating the directory structure, cleaning up the python code and assembling a unified documentation page.

For now – you may test out the fruits of my labour by downloading this installer of Firefox 3.0b2pre which will install Minefield in your Program Files/Minefield directory (you have no choice for installation dir.) and then copying these symbols into the folder where your Visual Studio (or Windbg) is looking for debug symbols.

The last thing you have to do in order for this to work is to put a path to cvs.exe in your PATH – I personally pointed it to the mozilla-build/mysys/bin/cvs.exe but if you like you can download cvs.exe and put it wherever you want.

Now that you have all you need, open up your debugger (VS or Windbg), attach it to the Minefield process and go!

It will ask you about the cvs calls that gather up the source code, say “OK” or “Run” or whatever is positive. The file should download into the proper /mozilla/whatever structure – on my computer it goes into Program Files/Microsoft Visual Studio 8/Common7/IDE/mozilla/…

That will be a focus in the next round, is manipulating the cvs command so that the source files go to a c:/temp or something. The debugger will automatically want to look for the path that the source file originally came from (in this case, my c:/ff_clean/mozilla/…) – you will have to point your debugger to the correct file the first time and then after that it knows to look in the mozilla folder you selected.

It works for now like what it is; a ball of duct tape that allows you to get the right source code for where you are in the browser without having a build on your computer.

I look forward to feedback about how this works for you, if you use it and if you have any problems arise.

After many nights of tweaking this space, that path, this logic – the symbolstore.py script is now playing nice with pdbstr and the pdb files are getting indexed. So begins the second battle – how to get Visual Studio to pull the code out of CVS and locate that code on the local drive. Right now, when you have the symbols loaded up and you attach to firefox as a process, a window comes up confirming the call that will be made.

If I copy and paste this command into my command line (I have CVS in my path from mozilla-build\mysys\bin) then I can pull the file and it goes into the directory structure as it is in the command starting from my current working directory.

When I click on “Run” in Visual Studio however, the code may or may not be getting pulled from CVS – I have no feeback and if I then try to look at something on the call stack, I get a dialog asking me to find the source code file. BUT WHERE IS IT?

Visual Studio is trying to find it in the hierarchy that my build was based on…that is not going to exist on the remote user’s computer.

I’ve made a few tweaks to symbolstore.py to try and create a %CVS_WORKINGDIR% in the srcsrv data block where the code would then be pulled to. This does not appear to be working, as the directory I am instructing it to create is not getting created.

This is the current symbolstore.py. More tweaking on the horizon.

Many huge thanks to Ted for figuring out the minute details that fixed the call to pdbstr.

The most up to date set of symbols can be found here: http://avnerd.tv/symbols/crashreporter-symbols-firefox-3.0b2pre-WINNT-2007120422.zip

The installer for this indexed version is here: http://avnerd.tv/sharedFiles/ff_SourceServer.exe.zip

I’m not sure where things are going wrong right now so I am hoping that you – faithful readers – will be able to help me out. I have run the “make buildsymbols” and checked with srctool.exe to confirm that the pdb files are source indexed. So following that I made an installer to see if I could attach it in Visual Studio and pull some code. However I am not getting the symbols to load.

Here’s what I have:

1. A modified symbolstore.py: http://pastebin.mozilla.org/255918
2. Symbols and source indexed .pdb files here: http://avnerd.tv/symbols/test/crashreporter-symbols-firefox-3.0b2pre-WINNT-2007120422
3. An installer for the build I was working on when I did this: http://avnerd.tv/sharedFiles/ff_SourceServer.exe.zip

When I point Visual Studio to the symbols I can’t get anything to load up. So this means that I am unable to test if the source indexed code’s cvs commands actually work.

There weren’t any gotchas the last time I uploaded symbols – so I’m not sure where to begin in figuring out why Visual Studio isn’t grabbing them properly.

I have to work all day tomorrow – and I have a big assignment still to work on for my BTS system design class, so anyone who can take a minute to poke at this and give me any feedback would be a hero!

Thanks!

Here is the code I have so far in symbolstore.py
http://pastebin.mozilla.org/254868

My portion is specifically lines 36 – 71 where I am now gathering up a string of source files along with the revision number, and generating a (almost) properly formatted stream file in the pdb’s directory.

Here are the barriers:

1. I would prefer to have a template where I have a variable %SOURCEFILES% that can be replaced and output to a new file for each pdb, instead of writing out each line of the stream file in symbolstore.py. If I can’t get that happening for this release – oh well, Dave said it can be a bit ugly at the 0.3 stage. At least what I have right now, is creating the stream file properly.

2. On lines 38 and 39 – what I am making here is a string that looks like this: c:\mozilla/xpcom/glue/nsTraceRefcnt.h and in fact, this has to be c:\USERS_SOURCEFILE_DIR\mozilla\xpcom\glue\nsTraceRecnt.h — so two things need to happen here, first, I need the / to be \ (except for the one after c:, cause it’s right already) and second, for now I can probably name the dir “ff” and the user can make sure that VStudio is pointing to that path (c:\ff) when looking for source files…but in the long run? I don’t know yet.

3. This is along the same lines as the template issue. The reason I haven’t successfully implemented the template is because I don’t know where to put, or how to create a path to, the right folders when symbolstore.py is being called. Unfortunately, it is not as easy as I had hoped in that putting the template in the same folder as symbolstore.py would allow me to open it with its name only. So as well as not being able to open the template, I don’t know how I can call upon pdbstr from inside of symbolstore.py

That’s going to be the last part, the first two are a little more pressing.

Huge changes since last night’s post.

I can insert a .stream file (see previous post for what a .stream file data block contains) into an unindexed .pdb file via pdbstr -w as long as it has certain variables.

I still need to test what the minimum info needed is, right now I just copied and pasted from one that works. I will slowly delete variables from it until it doesn’t work anymore.

Using cvs to pull a specific revision of a file looks like this: cvs export -r 1.23 filename

So – in my test.pdb.stream file I changed the CVS_EXTRACT_CMD to :

CVS_EXTRACT_CMD=%fnchdir%(%CVS_WORKINGDIR%)cvs.exe -d %fnvar%(%var2%) export -r %var4% -d %cvsdatetarg% %var3%

(p.s. %var#% are based on * delimited text in the source files section)

In the source files, i add a source file that contains cvs in the symbol file
like so:

SRCSRV: source files —————————————
c:\ff\mozilla\accessible\public\msaa\AccessibleMarshal.def*MYSERVER*mozilla/accessible/public/msaa/AccessibleMarshal.def*1.3

Then, when I write that to the accessiblemarshal.pdb file followed by a call to srctool -f on that .pdb file, I am greeted with a source indexed file that contains the cvs command that the source server will use to extract the source code when the time comes.

The result looks like this:

[c:\ff\mozilla\accessible\public\msaa\AccessibleMarshal.def] cmd: cvs.exe -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot export -r 1.3 -d 12-01-07 mozilla/accessible/public/msaa/AccessibleMarshal.def

c:\ff\mozilla\objdir_debugInfo\dist\crashreporter-symbols\2007120119\accessiblemarshal.pdb\8EEE4D7316254E22AE3D99EB4082744C3\accessiblemarshal.pdb: 1 source files are indexed – 199 are not

So now I need to learn how best to add this into the symbolstore.py file.

My first instinct is to add something in the ProcessFile function that keeps track of all the files that start with cvs, they look like this:

FILE 2 cvs:cvs-mirror.mozilla.org/cvsroot:mozilla/accessible/public/msaa/AccessibleMarshal.def:1.3
FILE 3 c:/program files/microsoft visual studio 8/vc/platformsdk/include/RpcNsi.h

I already did a little playing around with the python script and was able to insert some code here where I wrote if filename.startswith(“cvs”) then some debug statement to see that I could in fact catch the right lines (which I did).

So let’s break it down:

1. I need to track all lines that start with “cvs” and then turn these into proper file paths
2. I need a (minimal) template .pdb.stream file that I can write the source file paths to in source files section
3. Then I will call pdbstr -w and write that .pdb.stream (each named for the pdb it will write to) to the appropriate pdb file

The final hurdle – learn to write good python code to make this all happen.

Working on 0.3 release of source server is all about getting the pdb files gathered up when symbolstore.py is made and then feeding them to pdbstr.exe. I’ve copied the binary into the mozilla/toolkit/crashreporter/tools folder where symbolstore.py is called and I’ve spent the last 4 hours or so playing around with python.

So far I have learned this:

1. THIS IS WRONG, PLEASE IGNORE That I can call pdbstr.exe on an unindexed pdb file and it will in fact read and generate a data block that has all the information needed to set up the cvs cmd calls – see this output from calling pdbstr -r -p:thepdbfile -s:srcsrv > thepdbfile.stream ** for corrections see next post **

2. The next part would be to write this block back to the pdb file i think – so that it’s available to the source server

This is what calling pdbstr -r gets me:

SRCSRV: ini ------------------------------------------------
VERSION=2
INDEXVERSION=2
VERCTRL=http
DATETIME=Tue Nov 27 03:56:53 2007
SRCSRV: variables ------------------------------------------
CVSDATE=11/25/07
CVSDATETARG=11-25-07
CVS_WORKINGDIR=%targ%\%var2%\%fnbksl%(%var3%)
SRCSRVVERCTRL=http
MYSERVER=:pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot
HTTP_ALIAS=http://avnerd.tv/source
HTTP_ALIAS=http://avnerd.tv/source
HTTP_EXTRACT_TARGET=%HTTP_ALIAS%/%var2%/%var3%/%var4%/%fnfile%(%var1%)
SRCSRVTRG=%http_extract_target%
SRCSRVCMD=
SRCSRV: source files ---------------------------------------
c:\ffDebug\mozilla\accessible\public\msaa\AccessibleMarshal.def*MYSERVER*mozilla/accessible/public/msaa/AccessibleMarshal.def*1.3
SRCSRV: end ------------------------------------------------


So the next goal is to successfully keep a list in symbolstore.py of all the pdb so that pdbstr can be called on them.