I’ve been reading a bit about how the effectiveness of a search is measured and have thinking how we can apply it to our sourcing/internet research efforts. I kept coming across several terms which you all may have heard of, terms like precision, recall, and fall-out, which are the most common measurements of search effectiveness. After reading extensively on the subject all I got was a headache. : )

Here is what I took from this migraine generating study. As best as I understood it, Precision deals with accuracy or how well a search engine prevents those unwanted results or how big a portion of the results are actually meaningful to your query. Now, Recall on the other hand deals with how many of all the available relevant documents have been retrieved through your specific query, in other words how well it finds what you want. As far as Fall-Out, it refers to the proportion of non-relevant documents that are retrieved, out of all non-relevant documents available.

Even though I couldn’t find a good way for me to measure my search query effectiveness through recall, because it seems impossible to estimate the number of existing resumes on the web, even worse is trying to determine how many of those resumes are actually relevant to the particular skill set that I may find myself searching for. But Precision showed some potential, as I see it. For instance suppose that I write a query for a 3D piping design engineer as follows:

resume ( P&ID | PFD) (“Piping & instrumentation diagrams” | “process flow diagrams” | “Block flow diagrams”) (CAD | 3D Modeling)(“process control” | “Process flow” | “Control and shutdown | Flow directions)(HVAC | Chemical | metal)

Initially this query seems impressive, especially since I used four (4) sets of synonymous concepts to string together ideas that went over my head. When I ran this query in google.com though, it returned 83 results out of which 12 were resumes for Piping Engineers. Which means: my precision on this query was that of 14.46%. Considering that Piping designer are hard to find that may not be too bad but for comparison let’s run a different query:

intitle:resume inurl:resume (CAD | 3D Modeling)(Process-flow | Control shutdown | Flow directions)(HVAC|Chemical|metal) -site:miscojobs.com -~openings -jobs -careers -career

The results of this query were drastically different. At first glance it looks great. Out of the 21 results 21 were resumes that apparently match the desired skills which would be a 100% accuracy of course this is not accounting for all qualifications. But by improving me precision and recall I will review more resumes that are in the ballpark of what I need without wasting time going thru irrelevant results.

I guess what it comes down to is this; it is a good practice to monitor the results or each search string studiously, because they can show us valuable information as to how to increase our queries performance. As we see how accurate or inaccurate our queries are, we can then learn how to better target our search.

If you have any thoughts on this issue let me feel free to comment as I am still trying to figure out a way to use recall and fall-out metrics. It would be good to hear from you.