AVMA Economics Division head Michael Dicks and his team recently conducted a survey and concluded, "... the AVMA Report on Veterinary Employment published in March reported that weighted unemployment within the veterinary profession was 3.19% in 2014. To produce this number we obtained a sample of 1881 veterinarians. Since our membership consists of 72,174 veterinarians (excluding students) we are 95% confident in a .8% margin of error on both sides of the mean unemployment rate."

While we take issue with the unemployment number specifically, we take much greater issue with the failed methodology used to achieve the number, and the scientific failure it represents.  Particularly coming from an association that claims to value and promote science and the scientific method.

In an attempt to bring this to greater light, we wrote a Letter to the Editor of JAVMA.  Below is the subsequent exchange, as there was much go-round in an attempt to edit our claims and assertions.

Our side of the conversation is left-justfied and colored black.

The JAVMA side of the conversation has been right-justified and colored dark blue.

From: Eden Myers

Date: Saturday, June 13, 2015

Dr. Kurt J. Matushek, Editor-in-Chief

June 10, 2015

Dear Dr. Matushek,

In the May 15 2015 issue of JAVMA, the article titled “Validity of Survey Findings” states: "... the AVMA Report on Veterinary Employment published in March reported that weighted unemployment within the veterinary profession was 3.19% in 2014. To produce this number we obtained a sample of 1881 veterinarians. Since our membership consists of 72,174 veterinarians (excluding students) we are 95% confident in a .8% margin of error on both sides of the mean unemployment rate."

The findings of this survey were invalidated by an incorrect assumption before these calculations were ever performed: the assumption that AVMA membership is representative of the whole profession.  

Unemployed vets, especially chronically unemployed vets, are under-represented among AVMA members precisely because they are unemployed: they cannot pay their membership dues. While AVMA offers a hardship dues waiver, membership is no longer tied to health insurance¹, and professional liability alternatives abound², offering no incentive to renew AVMA membership upon expiration.  

Unemployed vets stop being members- resulting in a higher unemployment rate for the whole profession than can possibly be calculated from any sampling of the AVMA membership.

Unemployed veterinarians whose AVMA membership has not yet lapsed are less likely to respond to employment surveys due to social desirability bias³- the “tendency of survey respondents to answer questions in a manner that will be viewed favorably by others”.  Weighting of responses, designed to ensure accurate representation of various groups, amplifies the overly positive responses received. This distorts the results even if applied to only the  AVMA membership.

Last, vets who have voluntarily chosen to disengage from the profession are not accounted for. Those vets are still vets; they must be counted as unemployed for the purposes of stating overall unemployment among veterinarians. Ignoring the anecdotally significant and growing number of graduates outside the field further invalidates the survey’s findings, and misleads those considering the career.

Veterinarians who are not AVMA members differ significantly from AVMA members.  Those differences result in significantly different unemployment rates for the two groups. A survey to assay risk factors for suicide in veterinarians⁴ was deployed with AVMA participation to the entire US veterinary population using broadcast methodology; the authors documented an unemployment rate of nearly 11%⁵.  The AVMA Econ Division’s methodology preferentially samples ‘successful’ veterinarians to yield the much lower rate of 3.19%. While the Econ Division may have calculated their unemployment rate correctly, they have not calculated us correctly at all.

Sincerely, 

Eden Myers DVM MS

Harold Jones DVM

Daniel Skirvin, DVM

Michelle Harvath DVM

Suzanne Cosentino DVM

Lauren Bowling DVM RYT

Ryan G Gates DVM

Sidney D Lehr DVM

James A Taylor DVM

Robin T Stamey DVM

Robert J Nix DVM

Amber L Oteri DVM

Nikki Levine MSU c/o 2017

Bonnie Mader MS

1. AVMA website. Important News about AVMA Medical Insurance. Available at http://atwork.avma.org/2013/01/30/important-news-about-avma-medical-insurance-2/ Accessed June 10 2015.

2. Justvetdata.com. Professional Liability Alternatives Directory. Available at http://bit.ly/1zsBGEQ. Accessed June 10 2015.

3. Wikipedia. Social Desirability Bias. Available at http://bit.ly/1GsdI4Z. Accessed June 10 2015.

4. Nett RJ, Witte TK, et al. Notes From the Field: Prevalence of Risk Factors for Suicide Among Veterinarians- United States, 2014.  CDC Morbidity and Mortality Weekly Report Feb 13 2015; 64(05): 131-132.

5. personal communication, Randall Nett MD. March 15 2015.

On 6/15/15 at 3:53 PM, Merri Benney wrote:

Hi Dr. Myers et al:

Attached is your edited letter and copyright forms. Please print out all of the attachments, hand-write any changes to your letter on the hard copy, have everyone sign the copyright form and fax or email them all back to me as soon as possible to 847-925-9329. Please provide me with the highest academia degree for Nikki Levine and what the city and state information would be as well. Please also make sure that the signature block for Dr. Lauren Bowling is correct as well. I did not receive any city or state information for her military address.

If you should have any other questions, please let me know.

Thanks,

Merri Benney

Senior Assistant to the Editor-in-Chief

American Veterinary Medical Association

Attachment:

The recently released 2015 AVMA Report on Veterinary Employment states that the overall unemployment rate for the veterinary profession, according to results of a 2014 survey of 1,881 veterinarians in the United States, was 3.19%.¹ In a followup blog post regarding the report, Dicks et al² explain that based on the current AVMA membership of 72,174 veterinarians, we can be 95% confident that the true unemployment rate for the veterinary profession is between 2.39% and 3.99%. 

We contend, however, that the reported rate is not a true reflection of unemployment in the veterinary profession as a whole, because only AVMA members were surveyed and the AVMA membership is not necessarily representative of the entire profession in the United States. For example, unemployed veterinarians, especially chronically unemployed veterinarians, are likely under-represented among AVMA members because they are less likely to be able to afford to pay membership dues. In addition, unemployed veterinarians who are AVMA members may be less likely to respond to employment surveys than employed veterinarians. Finally, veterinarians who have taken a position outside the field of veterinary medicine would likely have been considered employed for purposes of the AVMA survey, but for purposes of providing information to students considering various career options, veterinarians who leave the profession should, we contend, be classified as unemployed.  

Overall, while the 2015 AVMA Report on Veterinary Employment paints an optimistic picture of unemployment in the veterinary profession, we worry that the results may not be telling the whole story. 

1. Nolen RS. Report sheds new light on veterinary employment. Findings range from wage implications of internships to veterinarians’ wish to work fewer hours. J Am Vet Med Assoc 2015;246:816–819.

2. Dicks M, Bain B, Knippenberg R. Validity of survey findings. Available at: https://www.avma.org/PracticeManagement/BusinessIssues/economics/Pages/Validity-of-Survey-Findings.aspx. Accessed June 15, 2015.

On 6/17/15 1:47 PM, Eden Myers wrote:

Ms Benney,

While I can accept the copy changes you have made to our letter, I cannot sign off on a reference to which I did not, in fact, refer.

If you wish to suggest changes that refer only to those references we did consult and cite, we would be happy to review them. If I have been unclear or incorrect in citing our sources please let me know and I would be happy to clarify.

Thank you,

Eden Myers

On 6/1715 at 2:10 PM, Dr. Kurt Matushek wrote:

Hi Dr. Myers,

Can you tell me which reference you object to? Your original letter refers to an article titled “Validity of Survey Findings” published in the May 15, 2015, issue of JAVMA, but of course, this article was not published in JAVMA and is only a blog entry posted on the AVMA website. Therefore, the second reference was added to direct readers to this blog post.

As for the first reference, authors are expected to reference statements of fact, such as the statement that the AVMA Report on Veterinary Employment provides an overall unemployment rate for the veterinary profession of 3.19%. I elected to reference the recent JAVMA News story on the report, rather than the report itself, because readers have to purchase the report to read it, but the news story is available in the journal.

Sincerely,

Kurt

On 6/18/15 11:35 AM, Eden Myers wrote:

Thanks for you for contacting me.  I am out of town and will have intermittent access to email for the next two months. Dr. Ryan Gates will be responding on my behalf.

Eden Myers DVM

On 6/19/15 9:08 AM, Ryan Gates wrote:

Dr. Matushek and Ms. Benney,

As Eden is biking half-way across the country with family under the cover of Tropical Storm Bill, I appreciate the opportunity to step in and speak for my colleagues.

I can appreciate your changing of the reference from the May 15 JAVMA to the May 15 Blog Post.

I cannot, however, appreciate the changes to the content of our letter.  We chose the terms and phrases of our letter with great care.  We don't believe a letter *to* the editor ought to be rewritten *by* the editor.

Therefore, we don't accept the modification of our letter as it has been returned.  We are not pleased with the way our profession has been represented recently by the AVMA Economics Division, and we have itemized these areas of displeasure.  If you would like to discuss running our letter in a form that resembles our original, I (we) would like that.

Sincerely,

Ryan

On 6/19/15 12:52 PM, Dr. Kurt Matushek wrote:

Dear Dr. Gates,

Thank you for following up on this. As stated in our instructions for authors, all letters are subject to editing. In this instances, changes were made to focus on what I perceived to be your major argument in an attempt to make the letter more understandable to readers (remember that most readers are not as conversant in this issue as you are). Additional changes were made to clarify when the authors were stating their opinions, as opposed to established fact, and to remove material that detracts from your argument (unemployed veterinarians have little use for professional liability insurance; therefore, veterinarians who drop their AVMA membership because they can obtain liability insurance through other sources increase the percentage of non-AVMA members who are employed).

If there are changes to the letter you object to or if your meaning has been altered, then I am happy to discuss additional changes to the letter. However, it would be most constructive at this point to work from the edited version.

Sincerely,

Kurt J. Matushek, DVM, MS, DACVS

Editor-in-Chief,

Journal of the American Veterinary Medical Association

On 6/19/15 4:04 PM, Ryan Gates wrote:

Dr. Matushek,

I recall our exchange on the topic of editing letters to the editor the last time Eden and I submitted a piece.  I appreciate editing for the sake of ease of reading and punctuation/grammar.  I hear your position and arguments for making the edits that you did.  I, however, disagree that our piece ought be whittled to the degree it currently stands.  Specifically, you've removed the "why" behind our assertions.  In the interest of providing clarity to readers who are not as conversant as my co-authors and I, why not include the foundation for our claims?

Additionally, the concluding paragraph as it has been edited does not illustrate our intent, and we cannot sign off on it.  We don't worry that the results *may not* be telling the whole story.  We are bothered because the results *do not* tell the whole story *because of the design flaw of the survey.*  The fact that this design flaw seems to be of little consequence or concern to the AVMA Economics Division specifically, and the AVMA leadership in general, is what prompted us to write, and the edited version of the letter does not sufficiently communicate our position. 

Again, we would like for this issue to remain in the spotlight, as it is of vital importance to our profession.  We are not opposed to some degree of editing of the piece.  As it stands, though, your current version of our letter is not acceptable to us.

Sincerely,

Ryan

On 6/23/15 12:51 PM, Dr. Kurt Matushek wrote:

Hi Ryan, 

Hanging on the wall in my office is the following sentence from an article called “The science of scientific writing,” which was published in the American Scientist in 1990: “We cannot succeed in making even a single sentence mean one and only one thing; we can only increase the odds that a large majority of readers will tend to interpret the discourse according to our intentions.” As I read your original letter, I interpreted it to mean that you believe the estimated 3.19% unemployment rate for US veterinarians reported in the 2015 AVMA Report on Veterinary Employment is likely an underestimate, because (1) non-member veterinarians were not represented in the survey population and non-member veterinarians are likely to have a higher unemployment rate than member veterinarians, (2) unemployed member veterinarians are potentially less likely than employed member veterinarians to respond to employment surveys, and (3) veterinarians who have left the field of veterinary medicine should be considered “unemployed,” even though they have taken a position in another field.

In editing your letter, I removed information that appeared to me to be extraneous or distracting and that I believed would lead readers to misinterpret your intent. If there are specific statements that you believe should be returned to the letter, let me know, and we can discuss them further. It wasn’t my intent to remove any information that was foundational to your claims. 

As for the final sentence, if you would rather that this read “Overall, while the 2015 AVMA Report on Veterinary Employment paints an optimistic picture of unemployment in the veterinary profession, we believe that the results do not tell the whole story,” that would be fine. 

Kurt

On 6/23/15 6:38 PM, Ryan Gates wrote:

Ken,

I hear your reasoning.  In response to an article about survey result validity, we pointed out that the survey's methodology invalidated its results.  To support this, we gave evidence, generated by the same organization using a different methodology, that a single assertion in the survey, in this case unemployment, was highly inaccurate.  I encourage you to re-read our original piece, and in particular the concluding sentence, where we make clear that our criticism is not with the unemployment number, unrepresentative of the entire population as it is, but rather with the methodology used by the AVMA Economics division to arrive at their conclusion.  For us, this comes down to science and our faithfulness to the scientific method.

Respectfully, we feel our colleagues are both more perceptive and more interested in diverse opinions than you appear to believe.  You have been unsatisfied with our writing, and we continue to be unsatisfied with the nature of editing that has been proposed.  We will not sign our names to anything that doesn't include the references to the broadcast methodology and to the Nett/Witte paper.  While we urge you to reconsider your stance, we will seek another venue for our letter if you cannot.

Sincerely,

Ryan

On 6/24/15 12:42 PM, Dr. Kurt Matushek wrote:

Dear Dr. Gates,

Unfortunately, your letter as submitted was confusing and unclear in several places. In editing, I attempted to clarify your intended meaning, although I freely admit that (because of the unclear writing), I may have misinterpreted your thoughts in some places. Several times, I have asked you to provide suggested changes to the edited version that would better convey your thoughts, but you have refused to do so.

I agree that readers are interested in diverse opinions and over the years have consistently worked to publish a wide variety of viewpoints in the JAVMA. However, there are limits to how much time I can spend on any particular letter without taking time away from other equally important letters and articles. Given that you have refused to work with me to get your letter published, spending additional time on it at this point does not seem useful. Therefore, I am afraid we will not be able to publish your letter.

Sincerely, 

Kurt J. Matushek, DVM, MS, DACVS

Editor-in-Chief,

Journal of the American Veterinary Medical Association

On 6/25/15 8:25 AM, Ryan Gates wrote:

Kurt,

On every occasion that Eden and I have written, from the original submission to my last email, we've explained to you that our observation and complaint is against the design methodology of the particular survey distributed by the AVMA Economics Division and the poor way they have represented the profession.

I understand how this is difficult to acknowledge.  And I'm sorry that you've interpreted my lack of submission of a revised letter as being unwilling to work with you.

Sincerely,
Ryan

---Additional exchange since original post.---

On 6/25/15 10:14 AM, Kurt Matushek wrote:

Hi Ryan,

One final follow up. The survey was sent to 8,278 veterinarians who had graduated from one of the 28 U.S. veterinary colleges in 2012, 2008, 2003, or 1988 (ie, 1, 5, 10, or 25 years previously), with usable responses obtained from 1,881. According to Mike Dicks, the survey was sent to all veterinarians in the AVMA database who met those criteria, and because the AVMA database includes information on individuals who had dropped their membership, the survey would have incorporated both members and nonmembers. 

kurt  

On 6/26/15 10:45 AM, Ryan Gates wrote:

Kurt,

Thank you for keeping the dialogue going.  And please accept my apology for mis-typing your name in an earlier e-mail.

You've mentioned "unclear writing" in several instances over the past week with regard to our observation and letter submission.

What then is clear about the Dicks article (Validity of Survey Findings, May 15, 2015) that would lead a reader to conclude that non-members were included in the survey?  Statements from the piece include...

• As we’ve learned, at a 15% response rate we need to survey at least 10,060 members to get a valid representative sample of 1509 veterinarians.
• Applying the formula to the AVMA population of approximately 85,000 members…
• To produce this number we obtained a sample of 1881 veterinarians. Since our membership consists of 72,174 veterinarians…
• Another unique attribute of the AVMA is that the members’ demographic characteristics are disproportionately distributed.

And the table itemizes various classifications of AVMA members.  All of these lead me/us to conclude that the survey sample was limited to AVMA members.

I hear what you claim, that former members would be included in the sample, but there is nothing in the piece that suggests non-members were a part of the sample size.  At the end of the day, given the degree of unknowns we're left with, we stand beside our observations as originally written.

Perhaps the lack of clarity itemized above is something that could be addressed in future AVMA writings on the topic.

Sincerely,
Ryan

On 6/30/15 9:51 AM, Kurt Matushek wrote:

Hi Ryan,

Unfortunately, I can’t provide answers to your question about the Validity of Survey Findings blog post, as these posts are prepared by our Economics Division. Dr. Dicks has said in the past that members and non-members were included in the employment survey.

Sincerely,

Kurt

And here we are.

This exchange is insightful on a number of levels.  First, there appears to be some degree of willful ignorance regarding the poor science behind the AVMA Economics Division survey design.

Second, after making clear (again) that our issue was with the survey design flaw more than the number the survey result based upon the design flaw, Dr. Matushek continued to misrepresent our conclusion.

Third, we're not sure the JAVMA staff thinks very highly of JAVMA readers, based upon his assessment of what they're able to comprehend and internalize.

Fourth, at the end of the day, our writing is "unclear" and we have "refused" to work with the editor, according to the editor.

Fifth, why is the AVMA Economics Division not held to the same standard of writing clarity that we small fries are held?

What do you think?  Were we unclear in our original letter and subsequent communication?