PJM Stakeholders Self-Reflect

On July 25, PJM hosted a “Stakeholder Process Super Forum” to discuss stakeholder input on PJM’s governance process. From my perspective, it is terrific that PJM and its members are looking into stakeholder process improvements, given there are multiple opportunities for beneficial reform.

One of the items reviewed during the meeting was the results of a Super Forum Survey PJM sent to members, asking for input on a variety of questions. 

About 20% (204) of PJM members participated in the survey, consisting mostly of affiliates (141) – meaning responses from a large parent company with multiple affiliates were counted several times. There were 49 unique respondents and 53 voting (i.e. non-affiliated) members participating.

The bulk of the survey asked stakeholders to respond by choosing up to six options ranging from “strongly disagree” to “strongly agree” (i.e. a Likert scale), with some options to add written responses.

It seems, PJM used the same survey questions from the 2009 Governance Assessment Special Team exercise, perhaps to compare 2018 responses to those from a decade ago. However, the 2009 baseline comparison data was not provided.

A Likert scale is a good tool to understand the intensity of a respondent’s feelings towards a statement. Understanding stakeholder feelings upfront is a useful starting point, making it an appropriate tool. However, as with any method, there are limits.

From a design standpoint, the stakeholders overwhelming tended to agree with the statements made in the survey (see graphic to the right). This shouldn’t suggest wrongdoing by PJM, rather reinforces a common problem with the Likert format – this format is prone to leading statements that result in acquiescence bias (i.e. agreement). 

But again, every survey method has shortcomings.

Another shortcoming of the Likert format is use of calculating arithmetic means (which PJM provided) from Likert responses, because assigning numeric values to qualifiers like “strongly agree” or “disagree” is imprecise since these responses have no numeric relationship (i.e. they are ordinal, not interval).  

So, now that you understand some of the typical limitations of the format, what does the survey reveal?  I put together a document (download below) with visualizations to better understand survey responses. I believe the survey most usefully indicates how the majority (i.e. over or under 50%) of respondents collectively feel (i.e. generally agree or disagree) about statements. But, recall, there is a clear bias towards agreement in the design of the survey. 

DOWNLOAD THE RESPONSE VISUALIZATION DOCUMENT

The survey begins with six questions (13 parts) about “support for the stakeholder process.” On only four questions did over 50% of the respondents’ lean towards disagreement (i.e. over 50% of the stakeholders chose “strongly disagree, disagree, or neutral-leaning disagree). In general, respondents believed the stakeholder process is not working well to

• Reach agreement on solutions,
• Efficiently resolve issues related to market design, or
• Reasonably balance competing interests.

Moreover, over 50% of respondents were dissatisfied with PJM’s stakeholder process.

On goals, respondents most strongly agreed that the essential goal of the stakeholder process is to ensure PJM meets its mission (i.e. reliability, competitive markets, etc.), followed by informing the board on member perspectives, then finally reaching agreement among members.

The next set of questions pertained to workload. Here, there was overwhelming agreement on all questions, especially regarding the need for prioritization. The lowest level of agreement pertained to the need to set firm timetables for resolution of issues. Respondents overwhelmingly believe that currently, there are too many stakeholder meetings needed to accomplish PJM’s workload.

Related to issues at the lower level committees – where the Consent Based Issue Resolution (CBIR) process is used – stakeholders disagreed that greater participation by senior level executives from member organizations would be beneficial. Almost 100% of stakeholder agreed that even when members can’t reach agreement on a solution, it is still beneficial for issues to be fully vetted through the stakeholder process.

Respondents overwhelmingly agreed PJM should more actively work with state regulators to understand their views and issues during the stakeholder deliberation process.

Pertaining to PJM staff, respondents disagreed that PJM staff or management should be brokering agreements between members. However, they agreed that PJM should advocate for competitive market solutions that are technically sound. Respondents were generally satisfied with how PJM staff facilitate/chair meetings and provide technical expertise. However, the overwhelming majority of respondents believe PJM needs to assign two staff members to an issue when the RTO has strong opinions about how an issue should be resolved – one to facilitate/chair and the other to advocate.

One of the most interesting outputs of the survey related to PJM’s board.  In general, stakeholders believe the board does not receive sufficiently clear and detailed information on diverse member perspectives when stakeholders fail to reach supermajority (i.e. 2/3rds) agreement on significant matters.

During the Super Forum, it was stressed that this survey was just used to identify issues member would like to consider addressing, not to form solutions. The intent is to distill issues for consideration into a problem statement and issue charge, to be considered for endorsement later in the year.

Going forward, it would be useful to see how these data compare to the 2009 survey results. It would also be useful to develop a survey with questions more relevant to contemporary issues and observation with the stakeholder process, as an important input to construction of the forthcoming problem statement.