Microsoft Expands Performance-Based Testing

Microsoft is getting ready to introduce simulation questions, or performance-based testing, to more of its exams.

The company currently has five exams with a performance-based testing (PBT) component, with the sixth expected to hit the market within the next three months. Following that, many more PBT exams are expected to be introduced during the next 15 to 18 months. “We see [performance-based testing] as a major part of our program and reinforced in the exams that we release to the market,” said Rob Linsky, group product manager for IT pro and developer certification at Microsoft.

“In a performance-based testing environment, you actually have to use the technology to demonstrate your skills; you can’t get by with reading and memorizing something,” Linsky said. “It actually discovers and validates real skills.”

PBTs mean an improvement in exam security, as well. Traditional exam items such as multiple-choice, matching, drag and drop, true/false and listing often are stolen and pirated to people who resell that information. But a performance-based testing environment makes it almost impossible to do that. “You might be able to pirate a question, but at the end of the day, in order to get the question right on a performance-based exam, you still have to show that you can perform the task by using the technology,” Linsky said.

Linsky feels that forcing candidates to actually perform tasks is a much more relevant way to measure skills.

Reliability is one of the variables that Microsoft considers when analyzing the performance of its exams. “Microsoft has an in-house analyst who’s constantly reviewing our performance. And our traditional exams, in terms of reliability, have always been very high,” Linsky said. “Add in performance-based items and the reliability goes up even more; [it] makes the whole exam more relevant.”

That spells huge benefits for IT companies, as it makes candidates more marketable. “If you know that someone can maneuver a performance-based testing experience on their way to a certification, then you as a hiring manager have a much more reliable indicator of their skill level,” Linsky said. “We’re going to spend a lot of time and energy [indicating to] hiring managers that this is something they need to start looking for and have to be searching out and recruiting against,” he said.

Just because performance-based testing is more realistic doesn’t mean it’s designed to trick people. “Microsoft Learning’s role is to get an accurate measurement of a candidate’s true skill level to a hiring manager [or] an IT manager,” Linsky said. “If you know how to manipulate the technology, you should do well; if you don’t, you’re going to struggle. We think of our exams as testing someone’s ability to go do a job.”

Feedback from the usability study indicates that test-takers seem excited about performance-based testing. “It mirrors real-life situations that they’re going to run into rather than traditional multiple-choice answers, so they definitely prefer much more of a performance-based, lab-based, hands-on experience because it’s extremely relevant to what they’re going to be doing from day-to-day,” said Bill Wall, director of certification at Microsoft. “[Test-takers] also like the fact that after they pass, they feel like they’ve achieved something, set a higher bar, and perhaps over time it does a better job of eliminating those people who shouldn’t have the certification — who don’t have the skills.”

While performance-based testing is a more realistic gauge of a candidate’s skills, the scoring doesn’t differ from the traditional section, except that instead of having a choice of four or five answers, they are left to complete an open-ended scenario to find the correct answer using the technology. “If you’re using the [user interface] — using the command line — you have to go to the command line and enter in the proper information, the proper string of commands,” Linsky said. “So rather than having the answers displayed for you, you’re given a scenario and you’re given access to the technology and you’re asked to work through the technology. And to find the correct commands and components of the technology will allow you to get to a right answer.”

While Microsoft is not allowed to reveal passing scores, it can reveal some changes that will be made to the score report candidates receive. Each Microsoft exam typically consists of five to seven functional groups, or topic areas. When Microsoft first released simulation questions, the limitations of the technology made it difficult to give candidates a traditional exam-section score by functional group. The new score reports are laid out by functional group, thereby allowing candidates to assess their specific areas of weakness.

“We’re going to be involving a lot of our current [Microsoft Certified Professionals] who will be testing and validating the new performance-based testing technology we’re working on that incorporates Microsoft’s own virtualization technology,” Linsky said. “A lot of people think we’ve been giving it lip service, but it’s a complex thing to integrate into your program. We’re a leading certifier of people in the IT state, [and] we couldn’t wade into it haphazardly. We’ve had to really think about how to incorporate it correctly into our program.”

Oracle DBA Certification Introduction

The Oracle Certification Program continues to be the most popular certification among all database vendors. This is no surprise considering Oracle has dominated the market for large scale relational databases worldwide. Oracle has completely revamped the DBA Certification track with the release of the Oralce9i database. To best understand the modifications that the DBA track has undergone, it's best to first review the Oracle8i DBA track.

To become an Oracle8i Certified Professional (OCP) DBA, the certification candidate needed to pass five exams:

• Introduction to Oracle: SQL and PL/SQL
• Oracle8i: Architecture and Administration
• Oracle8i: Backup and Recovery
• Oracle8i: Performance Tuning
• Oracle8i: Network Administration

Passing all five exams was a very challenging task, to say the very least. The first two exams in the track were often considered to be among the toughest...as mastery of their content was essential in passing exams 3 and 4. The Network Administration exam was an entity of its own, containing content much different than the others...although extremely important for the DBA to perform his or her duties.

With Oracle9i, the DBA Certification track is now divided into 2 primary tiers, as well as a third Master-level tier:

• Oracle9i Certified Associate (OCA)
• Oracle9i Certified Professional (OCP)
• Oracle9i Certified Master (OCM)

The core Oracle9i DBA track consists of 4 exams: 2 for OCA, and an additional 2 for OCP.

Oracle9i Certified Associate (OCA)

The OCA tier was designed to provide for an entry-level DBA certification designation for the Oracle9i platform. Information Technology professionals with little or no Oracle experience can obtain the knowledge and skills necessary to break into the highly lucrative and challenging Oracle profession. The skills obtained from completing this certification track will allow the candidate to participate on Oracle-related projects acting as junior-level DBAs. Typically, a senior-level DBA will work closely to assign appropriate DBA tasks to strengthen existing skills, and challenge them to learn slightly more advanced procedures.

The Oracle9i Certified Associate (OCA) requires passing two exams:

• Introduction to Oracle9i: SQL
• Oracle9i Database: Fundamentals I

Oracle9i Certified Professional (OCP)

The OCP tier was designed for mid to senior-level DBAs with one or more years of focused Oracle DBA experience. The knowledge and skills obtained from completing this tier of the certification program prepare the DBA to provide the full assortment of DBA-related tasks...from Database Design and Development to production tasks such as Performance Tuning and Backup & Recovery.

The Oracle9i Certified Professional (OCP) requires passing the two exams in the OCA Tier, plus two additional exams:

• Oracle9i Database: Fundamentals II
• Oracle9i Database: Performance Tuning

Note: It is also possible to become an Oracle9i Certified Professional (OCP) by taking one exam: Oracle9i: New Features for Administrators. To qualify to become an OCP by taking this one "Upgrade" exam, you must first be an Oracle8i Certified Professional (OCP).

Oracle9i Certified Master (OCM)

There is a third tier to the Oracle9i DBA Certification Track, the OCM. This track is targeted at very senior level DBAs and Oracle consultants. If you have less than 5 years of serious Oracle work experience...and have not spent many hours working with and testing advanced Oracle9i features...don't even consider this Tier as an option. I don't mean to sound negative about this, but let me continue with more on the requirements...and hopefully you'll see what I mean.

The OCM designation first requires that you obtain your Oracle9i OCP credential. Once this is secured, Oracle requires that you take two advanced Oracle courses from Oracle University. These are Instructor-led courses that will cost on average $2,000 each. There are about a dozen different courses to choose from:

• Oracle9i: Program with PL/SQL
• Oracle9i: Advanced PL/SQL
• Oracle9i: SQL Tuning Workshop
• Oracle9i: Database: Real Application Clusters
• Oracle9i: Database: Data Guard Administration
• Oracle9i: Database: Implement Partitioning
• Oracle9i: Data Warehouse Administration
• Oracle Net Services: Advanced Administration
• Oracle9i: Advanced Replication
• Oracle9i: Enterprise Manager

Once you've completed the course requirements, you can then register for and take the "practicum" exam. The practicum is a two-day, hands-on performance exam that requires the candidate to perform a number of tasks using Oracle9i on the Linux platform. Some of the skills that need to be proven include: database configuration, replication, partitioning, parallel operations, diagnostics, troubleshooting, performance tuning, and backup and recovery...all with "real-world" scenarios.

So, what's the problem with that you may ask? First, the skills you'll need to know and be able to perform are covered in many more than just two of the advanced courses you'll be required to take from Oracle University. So even if you do take the two required courses, you'll need to learn and be prepared for much more. This could require many months of additional self-study or many thousands of dollars in additional Oracle University courses. Second, the cost of the practicum exam is an outrageous $2,000. You won't want to fail and re-take (re-pay) for this exam!!

Don't get me wrong, I love the idea of a Master-level Oracle certification, but the course requirements and exam cost make this Tier cost-prohibitive for most.