As in the past, I am sharing my PASS Summit 2011 evaluation results. As also in the past, I explain that I do this because I want to improve my serve. If this kind of post makes you uncomfortable for any reason, please stop now and go read Brent’s awesome post on improving your evaluation numbers. It has awesome advice (as usual).
I encourage feedback when I present – especially ways people believe I can improve. I know that this is begging for dings to my score. I’m ok with that; this is not a race. Presenting is about serving the community and I simply want to do the very best job I can.
Presentation Levels Do Not Help
I have a suggestion for any organization organizing events with presentations: Please discard numeric levels. Why? Because they are an engine of loss and I believe they set up the presenters for some magnitude of failure.
A Question For Presenters:
Have you ever delivered a presentation that contained only material from one level? Me neither. Why is this? For me, it’s because the attendees are never merely seeking one level of material. How can I tell? By the questions. I get questions above and below any level of material I deliver. In a multi-track, multi-session format event, this is because some people attend my session because they seek to learn everything about a certain topic while others attend because very little being presented in this particular timeslot interests them and my session abstract sounds slightly more interesting than the others. This is not to say my presentation won’t help these attendees in some way.
When events provide pre-attendance scheduling capabilities, it would be interesting to note those who intended to attend my presentation – in advance – by adding my presentation to a timeslot. I am not suggesting adjusting the score for these attendees. I simply think that would make an interesting metric to use as a filter when analyzing the evaluation scores and comments.
When delivering a Level 500 session, I set the stage with lower-level material. I may ramp up quickly, spending only 5-10 minutes on introductory topics. This generates feedback that the session wasn’t at the correct level.
When delivering a Level 100 session, I poll attendees and quickly realize some already know the main points I am going to deliver. For example, each time I deliver “I See A Control Flow. Now What?” I ask for a show of hands of how many attendees have built an SSIS package that is in Production. The abstract clearly states this is for beginners with SSIS building their very first package. I am convinced, however, that some of the more experienced attendees learn things from this presentation. Why? They tell me personally or provide feedback via evaluations or email.. Apparently, I am including material to which they have not been exposed. Or I am presenting my understanding of a beginner topic in a way they have not considered. I am not sure. I rarely get complaints about the level of the Level 100 presentations I deliver, but I often – nearly always – get complaints about the level of any Level 400 or higher presentation I deliver.
I have a theory: People attending Level 100 sessions are open to learn more. Since they are just beginning, their context is unbiased and they are open to learn as much as possible. They don’t care if I wander into a Level 200 or Level 300 topic for a few minutes. They are learning stuff they don’t know. I believe people who attend Level 400+ sessions expect me to share stuff they don’t already know. If someone attends and my presentation turns out to be a review of things they’ve already learned, they feel it wasn’t a Level 400+ session because they didn’t learn something new. In my opinion, this is a symptom of Engineer’s Disease: We undervalue what we know and overvalue what we don’t know. My response to this is different: I celebrate the fact that many people already know anything I can show them. This means that my peers and I have succeeded in getting the word out. Yay us.
A Question For Attendees:
Think about the absolute best presentations you have ever attended. Was consistency in the level of material presented the best metric for you to describe those sessions? Me neither. I love Dr. DeWitt’s PASS Summit keynote presentations because he takes complicated material and breaks it down so even I can understand it. He breaks presentation rules in his slides, and yet continues to Wow! those in attendance. How does he do that? The best word I can use to describe it: Connecting. Dr. DeWitt connects with us. To connect, you have to understand the context attendees bring to a session. He does. Second, I think you have to know what the attendees want. What do all geeks want? We want to learn. Learn about what? Anything that will help us do our jobs better. For a few, this is probably an ignoble motive. But for the rest of us, we are in it for the brain exercise. We want to be more efficient, we want to learn stuff so we can help our companies, our communities, and ourselves do more with less. We simply want to know.
Dr. DeWitt uses what we already know to teach us something we want to know. That is connecting.
Conclusion
The last reason I think events should abandon levels is: they’re painfully distracting. It’s painful for presenters to decide which levels to assign; painful for the organizers to fiddle with scheduling a balanced smattering of 100s, 200s, 300s, and 400s in each timeslot; painful for attendees who bring their own concepts of which material should be assigned which level. I believe all complaints about level are a mask for some other – more valid – complaint. Remove this metric, and I believe better feedback will result. As a presenter, I can tell you I always get complaints about the level assigned to my presentation. I have delivered hundreds, perhaps thousands, of hours of material over the past decades of training and education. For me, complaints about level and level alone are evaluation noise. I simply ignore it at this point.
If event organizers insist on including levels, I respectfully submit that the levels themselves be as fuzzy as possible. What do I mean by fuzzy? Non-numeric. Perhaps Beginner and Advanced. With maybe a dash of Intermediate thrown in. If that occurs, you will see a spike in Intermediate sessions submitted due to the effects of the bell curve. That’s normal.
And that is another fantastic argument for eliminating levels altogether.
Andy
Great blog post.
I had to deliver an MS BI training once and my manager said I needed to give a level 100 one. (students were total beginners)
I had a level 200-300 training lying around, and lazy as I am :), I gave that one with a bit more focus on the harder parts. Result: students were very pleased with the training and I got great feedback. Conclusion: they don’t care about the level, as long as they learn.
I think the session level numbers are needed as a guide, for both the presenter and the attendees. It’s hard to tell just by reading the title (and sometimes even the abstract). I don’t see how the current system is different from using non-numeric levels – Beginner (100), Intermediate (200-300), Advanced (400), Expert (400-500).
"I don’t see how the current system is different from using non-numeric levels…"
Hi Matt,
I agree. That’s why using non-numeric levels is a compromise and not truly a solution. If someone is developing a curriculum, levels make sense. For a collection of presentations from different presenters on different topics? No. It’s an unnecessary distraction for both presenters and attendees, in my opinion.
Andy
Does any presenter ever tell someone in a 400 level course that they won’t cover material because they should already know it, or they shouldn’t have come?
Hi Jon,
I am not sure about other presenters. I have never said someone should not attend my Level 400 session if they do not already know A, B, and C.
I’m sure some people get more out of some presentations than others, but I believe anyone can take something away from every presentation.
:{>
Jon: Kind of. If I have 75 minutes to cover a 400 or 500 level topic, and someone asks a 200 level question, unless I’m doing VERY well on time I’m going to (politely!) tell them that we can talk later. And privately I’m going to be thinking, "why did they come to this session?" But no, I probably wouldn’t come out and say anything.
As for session levels, they are very much needed. Consider the following session title:
"Transaction Logging: How it Works"
Is this a beginner talk? Sounds like it. It might cover basic concepts around transaction logging. But wait! … It could also be an expert talk. It could be hardcore internals. Which is it?
Well, you could read the abstract, but oh wait, the abstracts are huge and we can’t possibly print them all in the little pamphlet that is put into the conference’s badge holder. We need a way to quickly convey who should attend. We need… Levels.
Hi Adam,
That makes sense. When I get questions that are beyond the scope of a given talk, I also politely defer them until later.
I like your example about Transaction Logging. I believe a beginner could learn a lot from an advanced presentation on the topic, and an experienced database professional could glean something from a beginner-level session.
As someone who reads the abstracts provided online and in the bigger booklet containing them, I may just be projecting. I still think complaints about level are misdirected and the actual complaint lies elsewhere.
:{>
I’ve recently been to sql rally nordic, where there were no level indicators on the sessions in the online agenda or physical schedule. Some of the levels could be guessed from the abstracts, but on most of them this wasn’t the case either. Partly due to this i went to 4 sessions (out of 10 time slots) that i wouldn’t have gone to if i would have known the levels, which appeared first when filling in the session evaluations.
Personally i would never complain about the level put on a session (as long as one doesn’t mix up 100 with 500), but when there are multiple interesting sessions in the same time slot i find the level indicator helpful to pick which session to go to. Though it is never the primary reason.
Hi Toine,
Your experience has me rethinking. Do you think clearer abstracts would have helped? The PASS Summit requests a list of take-aways the attendee should expect to learn. Did they do that at Rally Nordic?
Andy
Hi Andy,
Clearer abstracts would definitly have helped in 1 or 2 cases, since the abstract that was available wasn’t really related to the content of the session. There weren’t any take aways in the abstracts either.
The main problem i see with sessions/levels is that the 100’s and 500’s are quite easy to recognise/classify. And the expected content of sessions containing one of those levels is generally matching.
The real problem exists with the 200’s, 300’s and 400’s (or Intermediate/Advanced) because the level is based on the reference window of both the presenter and the attendees. For attendees the indication about their own level is only related to the knowledge they have about what they know and don’t know. And since most of them don’t know which part they don’t know they’ll put their personal level higher than what an expert would put them on (ofcourse there is also the group of attendees that think they are intermediate while they are near expert). Presenters will generally be subject experts, but when they put a level on their sessions they’ll have to take into account how an attendee would rate the session as well.
I totally agree that level indicators shouldn’t be the leading argument for choosing a session (for some attendees it is totally unrelated because they are only interested in 1 of the tracks anyway), but for the ones that are interested in content from 2 or more different tracks it should be the abstracts. Howevere when multiple sessions look interesting a level indicator will definitly help to choose the right session to go to.
One example i had at the Nordic rally was 2 sessions were going at the same time. Both of their abstracts stood out and i choose the one which i expected to be the session that would be the "right" one. However the session i picked pointed out to be a 200 and the other one was a 400. I can say that i got maybe 1 or 2 things out of the 200 session, but i know for sure that i would have gotten a lot more out of the 400 one. However if both of the sessions would have had the same level i would have gotten most out of the session i attended.
Toine
I’m with ya – I abhor the numeric level system, and I’d rather see categories like Concept Introductions, Demo-Fest, Performance Tuning, Troubleshooting, etc.
Hi Toine and Brent,
Toine: Excellent description of the difficulty with 200, 300, and 400 sessions – and I completely agree with your points about the context attendees bring to every session.
Brent: I like this suggestion a lot. One presentation may cover more than one category and your list would account for that. Cool!
:{>
I never mind going to a Level 100 session on SSIS. Yes I have packages in Production systems I have writen Script tasks to pull data from https sites as xml and used XSD to convert the data to load into the databases. But there is always something I don’t know.
If there is a choice between a 100 and a 400 level session in SSIS I would probaly go for the 400 but If there was a 200 level and Andy or Matt were presenting I wouldn’t mind doing that session over the 400. I like their presentations and always get more out of the session the the level they said the talk was.