SIP trace and reading SIP events
When a call fails, a trunk will not register or a phone behaves oddly, the quickest way to the cause is to look at the SIP messages themselves. MonoPbx gives your company two tools for that, both in the Logs section of the admin console:
- SIP trace (Admin > Monitoring and reports > Logs > SIP trace) captures the messages of one trunk, extension or number for a short time, shows the calls live as a ladder and gives you a pcap file for Wireshark.
- SIP events (Admin > Monitoring and reports > Logs > SIP events) keeps every request of the last 7 days with its responses and, when the PBX itself refused a call, the PBX cause that says why.
Use SIP events to find what went wrong after the fact. Use SIP trace when you want to reproduce the problem and watch it.
Who can use it
| You hold | You can |
|---|---|
diagnostics:read | Open SIP trace, read traces, open ladders, watch the live stream and download pcap files. |
diagnostics:write | Also start, stop and delete traces, and switch Keep message bodies on the SIP events page. |
audit:read | Open SIP events for your company. |
The built-in Owner role has both diagnostics permissions. For any other role, add them under Admin > Roles. A company only ever sees its own traffic: a trace captures a message only when it belongs to your company's domains or comes from or goes to one of your trunks, and another company's call that happens to share a Call-ID is never captured.
Trace a trunk or a phone
- Open Admin > Monitoring and reports > Logs > SIP trace and choose What to trace:
- Trunk: every call and registration message of that trunk.
- Extension: calls to and from the extension, from every one of its phones. Type the extension number.
- Device: one registered phone.
- Number: calls to or from a number of 3 to 20 digits, with an optional leading
+. - Filter: a caller, a called number or a peer IP address (at least one of the three). Use it for a scanner or an address you do not recognise.
- Optionally give the trace a name, and set the Duration with the slider. The page shows the limits that apply (below). Adjust the limits lowers the call, message or size caps.
- Choose Start trace. The page says the trace is running.
- Place the call, or restart the phone or trunk registration, that you want to see. Calls appear in the Calls list as they happen.
- Choose Stop when you have what you need, or let the trace end on its own. What it captured is kept.
Only one trace per target can run at a time, and your company can run up to five traces at once. If you try to start a second trace of the same target, the page tells you and points to the running one.
Transports and devices covered
A trace captures your company's messages on every transport the PBX accepts, with the same target and filter rules on each:
| Transport | Typical senders |
|---|---|
| UDP | Desk phones, most trunks |
| TCP | Phones and trunks set to TCP |
| TLS | Phones and trunks with encrypted signalling (SIP over TLS, usually port 5061) |
| WS and WSS | The MonoPhone mobile and desktop apps and the web phone, including when the PBX sits behind a reverse proxy that ends the TLS connection |
Calls from the apps therefore show up like any other call: the INVITE and its answers, the ACK, re-INVITEs and the BYE, and the REGISTER messages of the app's sign-in when the trace target is the extension or the device. Each message in a call shows the transport it used next to its addresses (udp, tcp, tls, ws or wss), and a call that went from the app to a trunk shows both legs, each with its own transport.
This holds on a single-server install and when SIP runs in separate processes or on separate servers from the console: the server that handles the SIP traffic does the capturing, whichever server you started the trace on.
Limits
A trace can never fill the disk. Every trace stops by itself at the first limit it reaches, and the list shows why it stopped.
| Limit | Default | Most you can set |
|---|---|---|
| Duration | 15 minutes | 4 hours |
| Calls | 200 | 2000 |
| Messages | 20,000 | 200,000 |
| Size | 50 MB | 1 GB |
| Traces running at once, per company | 5 |
On top of that, one budget covers every trace on the server: 2 GB, or 5% of the free disk of the data directory, whichever is lower. The Capture space used on this server line shows how full it is.
When a trace stops, the list shows the reason:
| Reason shown | Meaning |
|---|---|
| Its time ran out | The duration ended. |
| Call limit reached | The trace had captured its calls. Calls already in progress finish within the other limits. |
| Message limit reached, Size limit reached | Nothing more is stored. |
| The server's capture space is full | The server-wide budget was used up. |
| Stopped by hand | Someone chose Stop. |
Captures are never deleted early to make room. They are kept for 7 days after the trace ends and then expire. While the server-wide budget is full, a new trace is refused with the message that the capture space is full; delete old traces you no longer need, or wait for them to expire. The alert centre also raises an alert when the budget is full or a trace hits a cap.
Message bodies, including SDP, are captured so that you can read codecs and media addresses. Passwords and keys are removed, but telephone numbers and addresses are kept, so treat a trace and its pcap as confidential.
Record the call audio (RTP)
By default a trace holds the SIP messages only. To also capture the audio of the traced calls, tick Also record the call audio (RTP) when you start the trace. The RTP and RTCP packets are added to the trace's pcap, in time order among the SIP messages, with the real addresses and ports.
- It is call audio, so it is a separate choice: only a company administrator with
diagnostics:writecan ask for it, and the audit log records the start with its limits. Treat the downloaded file as confidential. - It has its own, tighter limits: 120 seconds of each call by default, counted from the answer (at most 600), and 20 MB of audio per trace (at most 100). When a limit is reached the first packets are kept and the rest is dropped.
- The audio of a call is stored when the call ends, counts in the server-wide capture budget and expires with the trace after 7 days.
- Calls whose audio does not pass the media relay, such as a call answered by voicemail, an IVR menu or a conference, have no RTP in the trace.
- If the option is not shown, RTP capture is not set up on this server. The operator enables it with the optional
compose.trace-rtp.yamlfile of the Docker install, which shares a recording directory between the media relay and the PBX. Starting a trace with audio on a server without it is refused withMPBX-SIPTRACE-RTP-UNAVAILABLE.
Watch the calls live
While a trace runs, the page connects to the server and adds every captured message as it arrives. The state next to the list reads Live, Connecting or Reconnecting; after a dropped connection it resumes where it left off, so no message is shown twice and none is skipped. Above the list the meters show Time left, Calls, Messages and Size against their limits.
With several PBX nodes, the live stream shows the messages of every node, whichever node your browser is connected to.
Read the ladder
Choose Ladder on a call to open its Call ladder. It reads like the diagrams in the SIP standards and in sngrep:
- A column for each address that took part: phones, the PBX and the trunk's provider. The time runs from top to bottom.
- An arrow for each message, pointing from the sender to the receiver, with the request line or the response code on it. The arrow colours follow the legend under Arrow colours: requests by method (INVITE, ACK, BYE, REGISTER and so on) and responses by class, with 1xx provisional, 2xx success, 3xx redirect, 4xx client error and 5xx or 6xx failure each in their own colour.
- A Time column with the time since the message before and since the first message of the call. Use it to spot a slow answer, a retransmission (the same message again after 500 ms, 1 s, 2 s) or a call that waited the full 32 seconds for a response.
Select an arrow to open the whole message: the start line and every header as a list, the body (the SDP) collapsed under SDP, and a Raw toggle for the exact text. Copy puts it on the clipboard. Search the messages dims the messages that do not match and highlights the matches, with Previous and Next to move between them.
What to look for
- A request with no response. The other side never answered: a firewall, a wrong address or port, or a device that is off. The same request repeating at growing intervals is the sender retransmitting.
- 401 or 407, then the same request again with credentials. This is normal: the PBX or the trunk asks for credentials once and the sender retries. If the retry gets another 401, 403 or 407, the username or password is wrong.
- A 4xx or 5xx after the retry. The credentials were accepted and the call itself was refused. Open the final response and read its
ReasonandWarningheaders, or the PBX cause of the call on the SIP events page (below). - 488 Not Acceptable Here, or 606. The two sides share no codec or no encryption method. Compare the
m=lines anda=cryptoora=fingerprintlines of the offer and the answer. - A call that connects with no audio. The signalling is fine. Look at the
c=andm=lines of the SDP in both directions: a private address in the SDP of a phone behind NAT points to a firewall or NAT problem, not to the PBX. - BYE from the PBX a few seconds after 200 OK. The far end never sent an ACK, or the session timer was not refreshed.
Download a pcap and open it in Wireshark
On a call, Download pcap saves that call. Download pcap of the whole trace saves every call of the trace, all PBX nodes merged and in time order. The file is a standard pcap (sip-trace.pcap for a call, sip-trace-<id>.pcap for a session).
To read it:
- Open the file in Wireshark. SIP messages are shown in the packet list.
- For a ladder, choose Telephony > VoIP Calls, select the call and choose Flow Sequence.
- If some packets show as plain UDP or TCP instead of SIP, the port is not the standard one: choose Analyze > Decode As, set the UDP, TCP or TLS port to SIP.
Notes on what is in the file:
- Each packet carries the real addresses and ports the PBX saw, and the original time.
- Messages that arrived over TLS or WSS are written as the decrypted SIP the PBX read, in a plain packet. You do not need keys to read them in Wireshark, and the file does not show the encryption.
- SIP packets are rebuilt from the captured messages. They are not a copy of the wire, so there are no TCP handshakes, checksums or sequence numbers to analyse. RTP is included only when the trace recorded the audio (above); in Wireshark, use Decode As and choose RTP for those UDP ports. For a full packet-level capture, use
tcpdumpon the host. - Credentials are removed. Telephone numbers are kept.
SIP events: reading a request and its responses
Admin > Monitoring and reports > Logs > SIP events lists each REGISTER, INVITE, MESSAGE and BYE with its final response, as described under Logs. Each row now opens up. Use the expand button at the start of a row to see Request and responses:
- the request the PBX received or sent, Received by the PBX or Sent by the PBX with the time, as a list of headers;
- every distinct response of that transaction, in order, provisional (such as 100 Trying and 180 Ringing) and final, each marked as received or sent;
- a Raw toggle and Copy on each message, the same viewer as in the ladder.
What you will find in the headers:
| Header | What it tells you |
|---|---|
| Via, Contact | The path the request took and where to reach the sender. Compare with the address the connection came from to find NAT problems. |
| From, To, Call-ID, CSeq | Who is calling whom. The Call-ID and CSeq tie the request to its responses and to the ladder. |
| User-Agent, Server | The phone or provider software and version. |
| Reason | Why a call ended or was refused (Reason: SIP;cause=486;text="Busy Here", or Q.850 causes from trunks). |
| Warning | A free-text hint added by the sender. |
| Retry-After | How long to wait before trying again (on 503 and 480 answers). |
X-Mpbx-Reject | The PBX's own cause code, sent to the phone of a signed-in user whose call it refused (see below). |
One line for a challenge and its retry
A 401 or 407 answer followed by the same request with credentials is shown as one line, marked after 407 challenge (or 401). Open it to see the Challenge with its messages above the Authenticated retry. The sign-in column keeps its meaning: Challenged, OK, Failed.
What is masked
Everything the PBX stores for a message is cleaned first: digest response and cnonce values, bearer and basic credentials, headers or parameters that carry a secret, passwords written in a SIP address (user:password@host), SRTP keys (a=crypto inline:) and ICE passwords. A message is stored in full up to 16 KB per event, and the final response is always kept. Events are kept for 7 days.
Message bodies (the SDP) are not kept by default. Someone with diagnostics:write can turn on Keep message bodies (SDP) with the SIP events on the SIP events page. From then on the bodies of your company's messages are stored, with keys and passwords masked as above; switching it off stops it again. Existing events keep what they have.
Jump to the trace or the call
Under an open event, Trace extension N or Trace address opens SIP trace with that target already filled in, so you can start a trace and reproduce the problem. Open the call opens the call in the call history, for events that belong to a call.
PBX cause codes
When the PBX refuses a call itself, it writes a stable cause code on the SIP event. The console shows it as a PBX cause badge on the row and explains it in words in the opened row, next to the Reason header. It tells you why the response was generated, which a status such as 503 cannot. The API can filter the events by cause (mpbx_cause).
A signed-in user's own device (a desk phone, the MonoPhone app or the web phone) is also told: the refusal carries Reason: SIP;cause=..., a Warning: 399 mpbx text and an X-Mpbx-Reject header with the code, and the apps translate it. A trunk, or an unauthenticated peer such as a scanner, gets no internal cause on the wire; the code stays in your SIP events and the call history only.
| Cause code | What it means and what to check |
|---|---|
no_rule | No dial plan rule matches the number. Check Outbound rules and the Rule tester. |
no_route | The matching rule has no usable route. Check its routes and trunks. |
all_routes_failed | Every route of the matching rule was tried and failed. Look at the trunk attempts of the call. |
no_route_reachable | No route could be reached, for example every trunk is down or unregistered. Check Trunks. |
trunk_full | The trunk is busy or its channel limit is full. |
not_found, no_such_extension, unknown_extension | The number or extension does not exist. |
extension_disabled | The extension is disabled. |
forbidden | The caller may not make this call. |
not_permitted | The extension's class of service does not allow this call. Check its Calling profile. |
blocked | Calls to this number are blocked by the Blocklist. |
cli_not_permitted | The caller ID chosen is not one of the caller's numbers. |
velocity_exceeded, cost_cap_exceeded | The toll fraud guard's call rate limit or spending cap was reached. See Toll fraud and CLIR. |
call_limit | A concurrent-call limit of the company or the trunk was reached. |
simultaneous_call_limit | The licence's limit of simultaneous calls was reached. See the licence page. |
voice_bundle_exhausted, voice_user_cap_reached, voice_balance_insufficient | A call over the MonoPbx Voice trunk: the company's pack minutes for this destination or the user's minutes are used up (there is no pay as you go). voice_balance_insufficient appears only in calls from before 2026-11-21. |
voice_destination_not_in_plan, voice_service_suspended, voice_unavailable | A call over the MonoPbx Voice trunk: no pack of the company covers the destination, the service is off or paused, or the plan could not be checked (try again). |
forward_loop, forward_refused | Call forwarding loops back or could not be followed. |
emergency_unroutable, emergency_ceiling | No route can carry an emergency call, or too many emergency calls are in progress. |
no_answer | Nobody answered before the ring time ended. |
callee_busy, callee_unavailable | The called party is busy or not registered. |
callee_rejected_<code> | The called party refused the call with that SIP status. |
media_unavailable | The media server was not available. |
pbx_overloaded, pbx_restarting | The PBX was at its call maximum or restarting for an upgrade. Try again. |
invalid_request | The request was not valid. |
internal_error | The PBX hit an internal error. Check the System log. |
A refusal of a REGISTER or of a MESSAGE by the registrar does not carry a cause code yet. For those, read the sign-in result (Failed, Challenged) and the response headers.
Example: a 503 after the 407 retry
An INVITE from extension 109 to an outside number first gets 407, then the retry gets 503. Open the line: the cause reads no_route_reachable, so the number matched a rule but none of its trunks could be reached. Open Trunks and check whether the trunk is registered. Then start a trace of that trunk and place the call again to see whether the PBX sends the INVITE to the provider at all and what comes back.
Tips
- Trace for as short a time as you can and stop as soon as you have the problem on screen. Short traces are easier to read and use less of the shared capture space.
- Start the trace first and place the call after, so that the first message is captured.
- To trace a phone that cannot register, use a Filter with the phone's IP address.
- Before sending a pcap to someone outside your company, remember it contains your telephone numbers and addresses.
API
The company routes are under /api/v1/diagnostics/sip-trace (tag sip-trace): GET and POST .../sessions (the list also returns the limits and the capture space used), GET and DELETE .../sessions/{id}, POST .../sessions/{id}/stop, GET .../sessions/{id}/calls, the ladder and pcap of a call, GET .../sessions/{id}/pcap, and the WebSocket GET .../sessions/{id}/live. The SIP events detail is GET /api/v1/logs/sip/{id} and the settings GET and PUT /api/v1/logs/sip/settings. The interactive reference lists every field and error code, for example MPBX-SIPTRACE-TARGET-BUSY, MPBX-SIPTRACE-TOO-MANY and MPBX-SIPTRACE-BUDGET-FULL.