Begin entering notes:

From the first part of this session with Steve and Lisa:

What is the goal of performance metrics? Is there a magic number we’re trying to achieve? From a PSAP perspective, wanted to ensure we’re all talking about the same unit of measurement; having standard definitions.

3 natural points of measurement? Maybe 4 points. Initiation, transition to next domain, ready to present to call-taker, time that call-taker makes that call live. Consistency is what we’re talking about. Brian thinks we need to solve this in SLAs.

RFC 6076 deals with some standards. We need more. RTP, voice quality, RCP(?), metrics for how good the media is. Alarms. Misrouting. A definition of misrouting.

CP Performance and human from PSAP. Authority is interested in network and core elements. Public point-of-view – how long for ring to occur – how long it takes to talk to a telecommunicator. Should we require full delivery of SIP? No stripped down? Repeatable, verifiable data every call.

Carrier means origination network. LNG or BCF forward concentration for this session? State of Maine - 5 9’s system availability from BCF to call delivery. Packet delivery 3 9’s per calendar month. Single point of failure redundancy. Emergency call delivery accuracy. Latency and jitter. Monthly measurement? “MOS” scoring test?

Packet loss, jitter, and delay are metrics that need to be measured. A spec for each can be tight but aggregate scores can cause ineffective results. “MOS” is subjective. Definition of terms is important. Not really voice quality that’s being measured – it’s packet quality. Mean opinion score = “MOS”. It’s opinion. Even though it’s subjective, it’s “stood the test of time”. Numbers are different for video.

What does call processing mean? Network call processing time – or ESInet call processing time – and PSAP call processing time. Two separate measurements that, together, make the total call processing time?

Call originated, ESInet, presentation, call is answered – should those 4 points be “named” so they can be measured?

Dan Mongrain kicks off the second half

Performance Measures & Metrics in a NG9-1-1 World8:30AM – 11:15AM | North BallroomFacilitators: Lisa Dodson, ENP; Steve O’Conor, ENP; Michael Smith; Rick Blackwell, ENP

What we’re looking for is people. This is a new WG. Mondays 1 PM ET is the meeting time.

Dan talks about the first version covering call processing. Now have the ability to measure when the call hits the first BCF. Second version will cover incidents. Not intending to touch reports – not defining what reports MIS should generate. Reports is a user interface.

Definition in STA-010 talks about when events should be generated.

First one is StartCall – this is talked about in i3. This is when a device sees a call from a caller – hasn’t reached a PSAP yet. Could the testcall mechanism be used to measure the time it takes a call to get to an agency? Yes. Video and text are SIP sessions – there will be an invite.

MESSAGE is a NHI call in SIP. You get StartCall and EndCall when you see a MESSAGE. This could be an alarm, for instance, or a fire detector.

StartMedia will only start when a call is established – voice, video, or text. Performance measurements can be done against this. Media can be added and/or removed from a call.

Slide above talks about logging events.

Dan is moving fast!

AdditionalAgency can be measured if the other agency allows access to data.

CallProcess - Some ESRPs won’t give you when call started and when call ended. They are not “call stateful”? You will know when they saw the call but you can’t compute when it determined that call ended.

Route – makes a decision on current state of affairs. This is reported in a route event. Time can be computed between how long it took for an intermediate ESInet to make a decision.

LoSTquery/LoSTresponse – in ESRP, when ECRF is queried for where it’s supposed to go.

At 10:29, these folks are logged in to TurboBridge:

This above is how to compute a time a request sent to LIS and comes back.

Dan describes the above. The data is available if we’re interested in it.

Dan talks about the various CallStateChange events above. With SIP, we’re lucky to have a lot of data that can be made useful.

There is some material to use to determine industry-standard. IETF RFC 6076 talks about how to compute certain times between invite and correlating events. We can borrow that information. These documents are in Kavi for the workgroup to use for reference material.

This above are metrics we know we want to capture. NFPA 1221 may have some info that helps. Don’t make the assumption there has to be a flame involved. Start and end points may be different for different vendors. Law and Fire may have different call processing standards.

The first version of the standard the WG is working on is for calls. Second version is for incidents.

Early media – ACD is not handled by an agent but, as far as the ESInet is concerned, the call has been answered. How do we handle this? We have to be careful what “callanswer” means. Maybe a separate metric for “answered by a machine”. There may be a need to talk about this in i3. Maybe the device will be identified so we can tell a human didn’t answer it? “CallAnswered” may mean a telecommunicator answered it.

PSAPs that have automatic greeting in the telecommunicator’s own voice. Those have been taken into account because the call has not been processed? On current system, they may not be able to talk to the caller until it’s done playing. It’s a very long message – statutorily required drivel. Maybe an NG PSAP requirement any automated response message must be interruptible. Maybe an event logged when “zip tones” are played.

Wrap-up Time (After Call Work) – how is wrap-up time measured? We also have agent state change. Coming up with a metric for this is dangerous. A potential for wrap-up time to be goof-off time. We need to make sure our agentevent makes this distinction. This may be a hole. We need to examine agentstates.

That’s all Dan has. Asks for volunteers. Go to NENA.ORG – click on “committees” – NENA development committees – Agency Systems – Agency Metrics. Do it. We could use help.

What’s Next / Action Items:

If any new WGs are formed provide the name of the WG and contact information for those wanting to volunteer.

Ensure any presentations/documents displayed or handed out are included on the USB stick.