https://survey.usask.ca/survey.php?sid=17684
:)
Thursday, November 12, 2009
Wednesday, November 11, 2009
Feedback on the survey
Of the six people that I contacted to provide feedback, four have responded - lucky me :). I will go through the feedback I received and comment on whether or not I changed the survey, how and why section by section.
Page 1 - demographics
Feedback indicated confusion because the groupings had overlapping numbers (age 16-18 or 18-25 for example). I thought I had watched for this but obviously not so - easy fix :)
I accepted the suggestion to change question 7 to, "What branch do you live in?" I had deliberate phrased it initially as "belong to" because I wanted the participants to identify the branch where they participate (and sometimes that isn't connected to where they live) but I then realized that the information I was looking for will be captured in another section where they indicated their level of activity in their "local branch". Changing this question clarifies it and will give richer information when considered with the level of activity questions.
While none of the test subjects :) commented, I am going to change question 5. the information I am trying to gather is regarding whether or not participants in the society tend to have memberships or not and why. Changing question 5 to, "Do you let your membership lapse?" will provide much more meaningful information, especially if they address it in the open ended question that immediately follows.
Page 2 - activities
I was expecting to have a few additional activies suggested. I was suprised there were no comments.
Page 3 - frequency of activity
One test subject suggested that using the term "weekly" in the first sentence may be confusing if the local branch doesn't have weekly activities and suggested merely dropping it. I have decided to leave it in because my understanding that every branch does have some kind of activity weekly (it isn't always the same activity though) and because I am trying capture whether or not the survey respondants are participating on a weekly basis rather than trying to determine if they are participating with their local group. What i will do to clarify is is change the answers to 'weekly, monthy, quarterly, and annually' then add a couple more questions asking about the frequency of participating in other groups. I think that is more likely to provide the information that I was hoping for - though it does make the survey a little longer, I don't think it will be significant.
There were no comments from my test subjects :), but I am going to add one open ended question here. These questions will capture some of the frequency and distances involved in participation, but will not provide any insight into why people participate the way they do. I will add the questions, "What would entice you to travel to participate in the activities of another branch?" and "What would entice you to travel outside your current comfort zone to attend an event?"
Page 4 - opinions
This section prompted the most feedback and I anticipated this. As I mentioned before, these are hot button items for many people and I deliberate presented them in a closed format to force participants to consider the basic level of answer, without conditions. Based on their feedback, I have decided to alter the wording of question 14 to "Avacal should become it's own kingdom." This removes the indication of immediacy which was, I think the basis of the need for conditions and theneed to comment on that particular question. I have also decided to add a box at the bottom of the page allowing participants to make any comments they wish - not connected to a particular question. Responses might be interesting and will only be made by those who feel the strongest. One of my commentors felt that allowing justifications might increase the response rate).
Page 5 - awards
Based on specific feedback, I will change the structure of the questions to, "Identify the level of recognitions that you have received in the ... Check all that apply. peerage, kingdom, principality, baronial, other, none yet" followed by, "If you wish, please describe the recoginition you received". The first question is clearer, I think. while the second may be a little more abiguous, it may provide richer information.
Based on conversations that ensued when reviewing these questions, I have decided that there are more questions to ask. This information will provide a factual snapshot of where people are at, but there is an underlying emotional issue that I would like to get at. I will add an opinion statement to the previous page, "The award system in the SCA effectively recognizes individual efforts." I will also add the open ended question, "Please describe if and how recognition affects your enjoyment of participating in the SCA."
Page 6 - recruitment
There was no feedback specific to this section and I can see no reason to make changes :)
I will post the link to the survey when I am finished making all the changes so you can see the final product. I will also be sending the link and an invitation to fill out the survey to as many mailing lists within the principality that I can find. The reading material indicates that I should open the survey for three weeks with reminders sent out at one week intervals - I will do that and see how many responses I get. There are roughly 360 members but I have no idea how many are participating.
Page 1 - demographics
Feedback indicated confusion because the groupings had overlapping numbers (age 16-18 or 18-25 for example). I thought I had watched for this but obviously not so - easy fix :)
I accepted the suggestion to change question 7 to, "What branch do you live in?" I had deliberate phrased it initially as "belong to" because I wanted the participants to identify the branch where they participate (and sometimes that isn't connected to where they live) but I then realized that the information I was looking for will be captured in another section where they indicated their level of activity in their "local branch". Changing this question clarifies it and will give richer information when considered with the level of activity questions.
While none of the test subjects :) commented, I am going to change question 5. the information I am trying to gather is regarding whether or not participants in the society tend to have memberships or not and why. Changing question 5 to, "Do you let your membership lapse?" will provide much more meaningful information, especially if they address it in the open ended question that immediately follows.
Page 2 - activities
I was expecting to have a few additional activies suggested. I was suprised there were no comments.
Page 3 - frequency of activity
One test subject suggested that using the term "weekly" in the first sentence may be confusing if the local branch doesn't have weekly activities and suggested merely dropping it. I have decided to leave it in because my understanding that every branch does have some kind of activity weekly (it isn't always the same activity though) and because I am trying capture whether or not the survey respondants are participating on a weekly basis rather than trying to determine if they are participating with their local group. What i will do to clarify is is change the answers to 'weekly, monthy, quarterly, and annually' then add a couple more questions asking about the frequency of participating in other groups. I think that is more likely to provide the information that I was hoping for - though it does make the survey a little longer, I don't think it will be significant.
There were no comments from my test subjects :), but I am going to add one open ended question here. These questions will capture some of the frequency and distances involved in participation, but will not provide any insight into why people participate the way they do. I will add the questions, "What would entice you to travel to participate in the activities of another branch?" and "What would entice you to travel outside your current comfort zone to attend an event?"
Page 4 - opinions
This section prompted the most feedback and I anticipated this. As I mentioned before, these are hot button items for many people and I deliberate presented them in a closed format to force participants to consider the basic level of answer, without conditions. Based on their feedback, I have decided to alter the wording of question 14 to "Avacal should become it's own kingdom." This removes the indication of immediacy which was, I think the basis of the need for conditions and theneed to comment on that particular question. I have also decided to add a box at the bottom of the page allowing participants to make any comments they wish - not connected to a particular question. Responses might be interesting and will only be made by those who feel the strongest. One of my commentors felt that allowing justifications might increase the response rate).
Page 5 - awards
Based on specific feedback, I will change the structure of the questions to, "Identify the level of recognitions that you have received in the ... Check all that apply. peerage, kingdom, principality, baronial, other, none yet" followed by, "If you wish, please describe the recoginition you received". The first question is clearer, I think. while the second may be a little more abiguous, it may provide richer information.
Based on conversations that ensued when reviewing these questions, I have decided that there are more questions to ask. This information will provide a factual snapshot of where people are at, but there is an underlying emotional issue that I would like to get at. I will add an opinion statement to the previous page, "The award system in the SCA effectively recognizes individual efforts." I will also add the open ended question, "Please describe if and how recognition affects your enjoyment of participating in the SCA."
Page 6 - recruitment
There was no feedback specific to this section and I can see no reason to make changes :)
I will post the link to the survey when I am finished making all the changes so you can see the final product. I will also be sending the link and an invitation to fill out the survey to as many mailing lists within the principality that I can find. The reading material indicates that I should open the survey for three weeks with reminders sent out at one week intervals - I will do that and see how many responses I get. There are roughly 360 members but I have no idea how many are participating.
Assignment 3 - practising survey creation
I think it would be helpful if I provided some background for the survey I'm about to share with you. I am involved in a historical re-creation group called the Society for Creative Anachronism. It is a group that studies and then attempts to re-create the middle ages (generously considering that to be 600ish -1600ish). I chose to use this venue to practise survey taking because it would allow me access to different people for testing the survey and an easy place to put the survey into practise (plus I'm genuinely interested in the results :). Just to clarify a couple of terms within the survey (that won't need to be clarified to the participants), like most international organizations we are organized as branches, my local branch is called a Barony in includes Saskatoon with surrounding area (most of the northern half of SK). The next level is called a Principality and encompasses SK, AB, plus the northern and western parts of BC. The Kingdom to wish we belong includes the rest of BC, Washington and Oregon with the northern tip of Idaho. If you are curious about the organization please feel free to ask me, or visit http://www.sca.org/ or to view my personal pictures of activities, visit http://picasaweb.google.com/raouldelaroche.
The purpose of the survey is mostly to gather information about the opinions of those who are participating in the activities of the society within the principality. There are several topics within this that could be considered for a program eveluation type of situation. As with most administrations of large groups of people, there have been many attempts (few of them organized programs) to do a multitude of things ranging from education of the membership, develop specific activities, recruit and retain members etc. As an involved member, I have been involved in many of these "programs" and am curious about their effectiveness - thus the connection to program evaluation :). These thoughts were the primary consideration of questions and topics for the survey. In addition, however, I also contacted a couple of other people in the leadership of the principality to see if there were questions and/or topics that they would like to see included in the survey.
Detailed purposes of each section:
PG1 - demographics & membership
This section is pretty self explanitory. It's purpose is to gather demographic information and determine if the participant is a member or not. There is interest in membership because in our society there are only a few activiieis that require membership to participate. There is a debate as to whether or not this should change. Many people say that 75% of our active participants do not have memberships. I do not believe that to be the case in our principality and am hoping to gather some information about both the numbers of members and the reasons why the participants have or do not have a membership. There have been some previous surveys done and it would be of interest to compare data and any activity (campaigning) done to convince people on either side of the debate.
PG2 - activities
There is an extremely wide range of activity in the SCA. I find it interesting to see what activities people are involved in and to see f there are any trends or themes that stand out.
PG3 - travel
In the last 5-10 years there seems to have been a change in the travel patterns of people in the principality. During many disucssions there has not been an explanation offered that seems reasonable to very many people. I would like to try and discover what some of the current patterns are and some insight into why.
PG4 - opinions
A few of these statements are 'hot button' items during discussion. There are some previous surveys asking similar questions and I would like to gauge changes in the general opinions of the participants. Questions regarding the transparency of decision making was the topic requested by someone sharing the leadership role at the principality level. Most of these ideas are quite complex and would be difficult to answer in such a straightforward way, but I choose to present them this way deliberately - almost like taking a temperature - I'm hoping to gain a general feel about the issue, not all of the context (that would be a different survey :).
PG5 - awards
I am mostly curious about this section cross references with other questions.
PG6 - recruitment and retention
This is a topic of conversation that has become more and more frequent of late. I am hoping that this survey will be provide information from wide section of participants and will provide some good insight that will be useful.
Once I designed the initial questions I sent them to six people for feedback and comments. I selected a mixture of geographic locations (Saskatoon, Calgary, Edmonton, & Medicine Hat), both genders, several age ranges. All of these people have had several years of experience in the society, would be interested in seeing the survey collect useful information (I think) and have at some point been involved in administsrative matters to varying degrees and in various ways. In hindsight, all of these people are professionals (one is a student) and probably should have attended to that. As well, I should have asked at least one person who is fairly new to the society to review the questions.
The purpose of the survey is mostly to gather information about the opinions of those who are participating in the activities of the society within the principality. There are several topics within this that could be considered for a program eveluation type of situation. As with most administrations of large groups of people, there have been many attempts (few of them organized programs) to do a multitude of things ranging from education of the membership, develop specific activities, recruit and retain members etc. As an involved member, I have been involved in many of these "programs" and am curious about their effectiveness - thus the connection to program evaluation :). These thoughts were the primary consideration of questions and topics for the survey. In addition, however, I also contacted a couple of other people in the leadership of the principality to see if there were questions and/or topics that they would like to see included in the survey.
Detailed purposes of each section:
PG1 - demographics & membership
This section is pretty self explanitory. It's purpose is to gather demographic information and determine if the participant is a member or not. There is interest in membership because in our society there are only a few activiieis that require membership to participate. There is a debate as to whether or not this should change. Many people say that 75% of our active participants do not have memberships. I do not believe that to be the case in our principality and am hoping to gather some information about both the numbers of members and the reasons why the participants have or do not have a membership. There have been some previous surveys done and it would be of interest to compare data and any activity (campaigning) done to convince people on either side of the debate.
PG2 - activities
There is an extremely wide range of activity in the SCA. I find it interesting to see what activities people are involved in and to see f there are any trends or themes that stand out.
PG3 - travel
In the last 5-10 years there seems to have been a change in the travel patterns of people in the principality. During many disucssions there has not been an explanation offered that seems reasonable to very many people. I would like to try and discover what some of the current patterns are and some insight into why.
PG4 - opinions
A few of these statements are 'hot button' items during discussion. There are some previous surveys asking similar questions and I would like to gauge changes in the general opinions of the participants. Questions regarding the transparency of decision making was the topic requested by someone sharing the leadership role at the principality level. Most of these ideas are quite complex and would be difficult to answer in such a straightforward way, but I choose to present them this way deliberately - almost like taking a temperature - I'm hoping to gain a general feel about the issue, not all of the context (that would be a different survey :).
PG5 - awards
I am mostly curious about this section cross references with other questions.
PG6 - recruitment and retention
This is a topic of conversation that has become more and more frequent of late. I am hoping that this survey will be provide information from wide section of participants and will provide some good insight that will be useful.
Once I designed the initial questions I sent them to six people for feedback and comments. I selected a mixture of geographic locations (Saskatoon, Calgary, Edmonton, & Medicine Hat), both genders, several age ranges. All of these people have had several years of experience in the society, would be interested in seeing the survey collect useful information (I think) and have at some point been involved in administsrative matters to varying degrees and in various ways. In hindsight, all of these people are professionals (one is a student) and probably should have attended to that. As well, I should have asked at least one person who is fairly new to the society to review the questions.
Friday, October 16, 2009
Thursday, October 15, 2009
Wednesday, October 14, 2009
Comments on Logic Model
I will keep trying (and also watching to see if anyone has advice for me) to upload the actual document/chart of my logic model or a readable image of it or something, but in the mean time, here are some comments and explanations.
This is a snapshot of the CLS summer school as I see it. I have been involved in the design and implementation since the beginning. The bulk of the logic model comes from my own perceptions of what we have been trying to accomplish, though I have also talked with two of the directors that have also been involved since the beginning (to varying degrees) to see if I am on the right path - more so for input into the evaluation plan that will be posted shortly (in text form!).
What is primarily missing from the chart is the inputs since that made the most sense to me (considering we are immersed in a facility wide strategic planning panic right now and thus - in my brain). So...to list inputs is easy: staff, money, venue for meetings/lectures, beamlines for practical sessions, social venues, time.
To be a little more specific, there is staff input regarding the logistics of event planning (room bookings, hospitality, registrations, budgeting etc); staff that develops the scientific program in collaboration with the SR community; and collaboration between aforementioned staff and user community for promotion. some of the class documents mention that the list of activities should be specific in listing who does each activity. I chose not to do that because most would be listed as, "a collaboration of administrative and scientific staff and user community". Most of the other inputs are pretty self explanatory.
So, in a nutshell, I am using this snapshot of the program to design an evaluation. There was no needs assessment at the beginning so I have only documents, notes, and correspondence to formulate a concept of what was initially intended (to compare to what we 'remember' we intended to do :) and there has been much formative evaluation in the form of feedback surveys and debriefing meetings so it seems time to do a formal, summative evaluation :), but that will be the next blog ...
This is a snapshot of the CLS summer school as I see it. I have been involved in the design and implementation since the beginning. The bulk of the logic model comes from my own perceptions of what we have been trying to accomplish, though I have also talked with two of the directors that have also been involved since the beginning (to varying degrees) to see if I am on the right path - more so for input into the evaluation plan that will be posted shortly (in text form!).
What is primarily missing from the chart is the inputs since that made the most sense to me (considering we are immersed in a facility wide strategic planning panic right now and thus - in my brain). So...to list inputs is easy: staff, money, venue for meetings/lectures, beamlines for practical sessions, social venues, time.
To be a little more specific, there is staff input regarding the logistics of event planning (room bookings, hospitality, registrations, budgeting etc); staff that develops the scientific program in collaboration with the SR community; and collaboration between aforementioned staff and user community for promotion. some of the class documents mention that the list of activities should be specific in listing who does each activity. I chose not to do that because most would be listed as, "a collaboration of administrative and scientific staff and user community". Most of the other inputs are pretty self explanatory.
So, in a nutshell, I am using this snapshot of the program to design an evaluation. There was no needs assessment at the beginning so I have only documents, notes, and correspondence to formulate a concept of what was initially intended (to compare to what we 'remember' we intended to do :) and there has been much formative evaluation in the form of feedback surveys and debriefing meetings so it seems time to do a formal, summative evaluation :), but that will be the next blog ...
Subscribe to:
Posts (Atom)